Threading vs Parallelism ?

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Threading vs Parallelism ?

fredvs
Hello.

Some developers treat me as dinosaur because I use threads in place of doing parallelism.

Huh, ok, but why parallelism is better and how to do it with fpc ?

Thanks.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Free Pascal - General mailing list
Hi,

On 29/3/2017 9:57 μμ, fredvs wrote:
> Hello.
>
> Some developers treat me as dinosaur because I use threads in place of doing
> parallelism.
>
> Huh, ok, but why parallelism is better and how to do it with fpc ?

   a nice article I've found long ago when I was researching the same
topic "Threading/Concurrency vs. Parallelism" is the following :

http://tutorials.jenkov.com/java-concurrency/concurrency-vs-parallelism.html

  and I tried the multithreadprocslaz package in Lazarus :

http://wiki.freepascal.org/Parallel_procedures

Happy coding ....

Regards,

--
Dimitrios Chr. Ioannidis
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Michael Van Canneyt


On Wed, 29 Mar 2017, Dimitrios Chr. Ioannidis via fpc-pascal wrote:

> Hi,
>
> On 29/3/2017 9:57 μμ, fredvs wrote:
>> Hello.
>>
>> Some developers treat me as dinosaur because I use threads in place of
> doing
>> parallelism.
>>
>> Huh, ok, but why parallelism is better and how to do it with fpc ?
>
>   a nice article I've found long ago when I was researching the same
> topic "Threading/Concurrency vs. Parallelism" is the following :
>
> http://tutorials.jenkov.com/java-concurrency/concurrency-vs-parallelism.html
>
>  and I tried the multithreadprocslaz package in Lazarus :
>
> http://wiki.freepascal.org/Parallel_procedures
Showing nicely that parallelism in a single process implies threads.
I would not pay too much attention to what these developers are saying.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Free Pascal - General mailing list
Hi,

On 29/3/2017 11:15 μμ, Michael Van Canneyt wrote:

< snip >

> Showing nicely that parallelism in a single process implies threads.
> I would not pay too much attention to what these developers are saying.

besides hardware parallelism, how software parallelism ( in the sense of
true simultaneous execution ) in a single process without using ( OS
dependent ) threads can be achieved ?

regards,

--
Dimitrios Chr. Ioannidis
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

fredvs
> I would not pay too much attention to what these developers are saying.

Excellent idea, I will do the same ;-)

> how software parallelism ( in the sense of true simultaneous execution )
> in a single process without using ( OS dependent ) threads can be achieved ?

Huh, it is exactly was I do not understand...

Thanks.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

fredvs
In reply to this post by Free Pascal - General mailing list
> besides hardware parallelism,

Is it possible, with fpc, to assign one processor (if multi) for a thread and say to the system to use this one only for the thread ?
But maybe it will gives more problems than solutions.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Karoly Balogh (Charlie/SGR)
Hi,

On Wed, 29 Mar 2017, fredvs wrote:

> > besides hardware parallelism,
>
> Is it possible, with fpc, to assign one processor (if multi) for a
> thread and say to the system to use this one only for the thread ? But
> maybe it will gives more problems than solutions.

Yes, it's called thread/process affinity, and most operating systems have
a specific API for it. But one should treat such things extremely
carefully, otherwise with badly designed data access concurrency you end
up with potentially hard to identify structural problems like false
sharing (see Wikipedia on this) and others. This can be especially bad for
for example on NUMA architectures, or others, where data storage locality
to a given core is not obvious. (Just think about clusters, etc.)

In other words, as the OS knows best your hardware's layout and how it can
best provide performance, energy, bandwith, etc, it's usually very wise to
rely on the OS to solve the thread to CPU core assignments for you.

Bottom line: parallelism and threading is a very complex problem and very
diverse area, and anyone laughing at any approach is just shows how little
they know about true depth of the whole topic, IMO. Thinking about any
technical solution, or way of implementation as a silver bullet for
everything is a very "expert beginner" approach.

My 2 cents.

Charlie
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

fredvs
@Karoly Balogh (Charlie/SGR)

Perfect, I have now all the arguments to defend the "Dinosaur Threading" choice.

Thanks.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Ryan Gonzalez
In reply to this post by fredvs
Parallelism is what hipster Node programmers do, threads are everything else.

/jk

--
Ryan (ライアン)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://refi64.com

On Mar 29, 2017 1:57 PM, "fredvs" <[hidden email]> wrote:
Hello.

Some developers treat me as dinosaur because I use threads in place of doing
parallelism.

Huh, ok, but why parallelism is better and how to do it with fpc ?

Thanks.

Fre;D



-----
Many thanks ;-)
--
View this message in context: http://free-pascal-general.1045716.n5.nabble.com/Threading-vs-Parallelism-tp5728018.html
Sent from the Free Pascal - General mailing list archive at Nabble.com.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Ryan Joseph
In reply to this post by fredvs

> On Mar 30, 2017, at 4:25 AM, fredvs <[hidden email]> wrote:
>
>>
>> how software parallelism ( in the sense of true simultaneous execution )
>> in a single process without using ( OS dependent ) threads can be achieved
>> ?
>
> Huh, it is exactly was I do not understand...

Look at an OpenCL tutorial for more information. If you select a CPU device (instead of a GPU device with hundreds of compute units) then the API will automatically allocate the amount of threads available on the all cores available (which may be only 2). It’s still threaded but scales easily depending on the task. I don’t know why but I got the OpenCL CPU target to process multiple times faster than just using the CPU directly.

Parallelism is only helpful if you have a problem involving massive amounts of discrete tasks which are part of larger task and can be designed as such. If your problem is designed like this then you can offload the processing to the GPU and get some serious performance benefits. The catch is that reading/writing memory to the GPU is slow so unless you’re processing on large batches of data you’ll spend more time accessing memory than processing.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Mark Morgan Lloyd-5
In reply to this post by fredvs
On 29/03/17 22:30, fredvs wrote:
> @Karoly Balogh (Charlie/SGR)
> Perfect, I have now all the arguments to defend the "Dinosaur Threading"choice.
> Thanks.

I'd second Charlie's point, and add that a very small change to a
system's layout, e.g. a DIMM on a NUMA node going dodgy and being
excluded at boot, can have a drastic effect if process or interrupt
affinity has been locked down inadvisedly.

Another thing to take into account is that when many people mention
parallelisation they're thinking of something like OpenMP, which in
practice is built on top of threads and processes and obviously
introduces substantial overhead. It's good for large jobs carefully
designed, particularly on very large datasets.

Some languages were designed with at least some measure of
parallelisation in mind. I'd highlight in particular APL, which tried to
minimise control flow variation so that in principle at least a function
could be applied to all elements of an array simultaneously. Granted
that in those days CPUs were a scarce resource, but even then there were
understood to be hazards which could break the programming model.

Ryan mentions OpenCL etc., which is about as close as we've got to "a
vector processor on every desktop".

Finally, I suggest that you look at least briefly at
https://en.wikipedia.org/wiki/Vector_Pascal which appears to have some
quite good stuff in it.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Michael Schnell
In reply to this post by fredvs
On 29.03.2017 20:57, fredvs wrote:

> Huh, ok, but why parallelism is better and how to do it with fpc ?
>
Parallelism within a process always is based on threads.

AFAIK, fpc does not (yet) provide a more convenient abstraction for
parallelism (such as parallel loops) than TThread.

-Michael

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Mark Morgan Lloyd-5
In reply to this post by Mark Morgan Lloyd-5
On 30/03/17 08:00, Mark Morgan Lloyd wrote:

> Finally, I suggest that you look at least briefly at
> https://en.wikipedia.org/wiki/Vector_Pascal which appears to have some
> quite good stuff in it.

Quoting from manual section 5.4.

Future machines like the Larrabee will have considerably wider
SIMD registers, increasing the benefits of SIMD code. But newer chips
also have multiple cores. For these, the recent versions of the Glasgow
Pascal Compiler will parallelise across multiple cores if the arrays
being worked on are of rank 2. The Pascal source code of the program
remains the same independently of whether it is being targeted at a
simple sequential machine, a SIMD machine or a multi-core SIMD machine.
Targeting is done by flags passed to the compiler:

[...]

Two threads are dispatched to process the work using a fork - rejoin
paradigm. The run time library is built on top of pthreads. For a two
core machine, two server threads are initiated at program start up.
These wait on a semaphore until post_job passes them the address of a
procedure and a stack frame context within which the procedure is to be
executed.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Jon Foster
In reply to this post by Michael Van Canneyt
On 03/29/2017 01:15 PM, Michael Van Canneyt wrote:

>
>
> On Wed, 29 Mar 2017, Dimitrios Chr. Ioannidis via fpc-pascal wrote:
>
>> Hi,
>>
>> On 29/3/2017 9:57 μμ, fredvs wrote:
>>> Hello.
>>>
>>> Some developers treat me as dinosaur because I use threads in place of
>> doing
>>> parallelism.
>>>
>>> Huh, ok, but why parallelism is better and how to do it with fpc ?
>>
>>   a nice article I've found long ago when I was researching the same
>> topic "Threading/Concurrency vs. Parallelism" is the following :
>>
>> http://tutorials.jenkov.com/java-concurrency/concurrency-vs-parallelism.html 
>>
>>
>>  and I tried the multithreadprocslaz package in Lazarus :
>>
>> http://wiki.freepascal.org/Parallel_procedures
>
> Showing nicely that parallelism in a single process implies threads.
> I would not pay too much attention to what these developers are saying.
>
>
Either I'm dense, which is entirely possible, or the author of that article
didn't do a good job of making his point, even assuming he had one.

Thinking about this a bit more I would say that "threads" are a means
(vehicle) to achieve parallelism. I think that in the majority of
programming contexts they are synonymous. But I can think of other contexts
where they may not be: Say animating a film. You have 1000 computers
generating frames to be assembled into a final sequence. The process of
dispatching each frame render is most likely not going to be done with the
OSes thread call, but through a networked work queue of some sort (ala
Torque or something). So you have the "task" (rendering the movie) being
worked on in "parallel" via a networked job dispatch mechanism.

Or the case of using "make -j ..." which "forks" multiple jobs (compile,
link, ...) in parallel based on a dependency tree.

I say threading is parallelism, even if just one form of it. The other
methods of parallelism I mentioned here could also be done in FPC with the
appropriate code. But that is a rather nebulous topic to discuss. Concrete
answers for which are only available when you know how a specific job needs
to be broken up.

I'm still looking for a good easy to use job queue myself...

-Jon

--
Jon Foster
JF Possibilities, Inc.
[hidden email]

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Ryan Joseph
In reply to this post by Michael Schnell

> On Mar 30, 2017, at 3:06 PM, Michael Schnell <[hidden email]> wrote:
>
>>
>> Huh, ok, but why parallelism is better and how to do it with fpc ?
>>
> Parallelism within a process always is based on threads.
>
> AFAIK, fpc does not (yet) provide a more convenient abstraction for parallelism (such as parallel loops) than TThread.
>
> -Michael

It’s my understanding that for parallelism to make sense you need to have at least more than 1 separate compute unit, be that a CPU core or a GPU.

If you had a GPU with 250 compute units or a CPU with 250 cores you would need to design your task in a way so that it could be broken down into as many discrete portions as possible so that you could take advantage of the multiple cores running in parallel. Even if you didn’t have a single thread and the execution blocked until finished you wouldn’t see any performance increases unless you designed your program to scale for parallelism. Running 250 threads on a single core isn’t going to be 250x faster but running 250 threads on 250 cores may be.


Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Tony Whyman
The problem I have with this thread (no pun intended) is that it is not
comparing like with like. As demonstrated by many of the replies,
Parallelism and Threads are not the same thing.

I would offer the following definitions:

- Parallelism is a (design) concept for expressing collateral actions in
which the processing order of the actions is unspecified. They may take
place serially or contemporaneously in real time, or a mixture of the two.

- Threads are an implementation mechanism for realising collateral
actions within a single processing environment.

Neither of the above implies multiple CPUs or processing units.


On 31/03/17 07:43, Ryan Joseph wrote:

>> On Mar 30, 2017, at 3:06 PM, Michael Schnell <[hidden email]> wrote:
>>
>>> Huh, ok, but why parallelism is better and how to do it with fpc ?
>>>
>> Parallelism within a process always is based on threads.
>>
>> AFAIK, fpc does not (yet) provide a more convenient abstraction for parallelism (such as parallel loops) than TThread.
>>
>> -Michael
> It’s my understanding that for parallelism to make sense you need to have at least more than 1 separate compute unit, be that a CPU core or a GPU.
>
> If you had a GPU with 250 compute units or a CPU with 250 cores you would need to design your task in a way so that it could be broken down into as many discrete portions as possible so that you could take advantage of the multiple cores running in parallel. Even if you didn’t have a single thread and the execution blocked until finished you wouldn’t see any performance increases unless you designed your program to scale for parallelism. Running 250 threads on a single core isn’t going to be 250x faster but running 250 threads on 250 cores may be.
>
>
> Regards,
> Ryan Joseph
>
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Michael Schnell
In reply to this post by Jon Foster
On 30.03.2017 18:29, Jon Foster wrote:
>
> I say threading is parallelism, even if just one form of it. The other
> methods of parallelism I mentioned here could also be done in FPC with
> the appropriate code.
Threading is the way parallelism can be achieved using a "standard"
programming language and running the result as a "standard" program on a
"standard" OS with "standard" computer hardware.

Of course there are lots of other options such as Multiple Data CPU
instructions, starting multiple programs. and even using FPGAs (to be
programmed in languages like "VHDL" (where parallelism is the "simple"
syntax, while sequential execution needs do be dedicatedly coded).

The only thing that seems to be a viable option for a future "normal" PC
compiler could be stuff like "parallel loops" (appropriate Pascal syntax
is already done in Oxygene), as a syntax candy on a library allowing for
parallelism by using a thread pool.

-Michael

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Gary Doades
In reply to this post by Tony Whyman
> I would offer the following definitions:

> - Parallelism is a (design) concept for expressing collateral actions in which the processing order of the actions is unspecified. They may take place serially or
> contemporaneously in real time, or a mixture of the two.

> - Threads are an implementation mechanism for realising collateral actions within a single processing environment.

> Neither of the above implies multiple CPUs or processing units.

I would agree wholeheartedly with most of that. Parallelism is purely a concept of multiple tasks running at the same time

Threads or processes are just implementations of that concept. Threads tend to be used for related tasks in a single process. Separate processes tend to be used for unrelated or independent tasks. Those are not hard and fast rules.

However, multiple independent compute units must be required for *true* parallelism. On a single processor any tasks running at the same time is just an illusion, normally created by the OS in time slicing between tasks based on certain criteria (priority, I/O, cpu usage etc.). That applies equally to threads or processes

Various languages assist, or purely exist, to make creating multi-tasking easier, but it ultimately all boils down to the same thing.

Regards,
Gary


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Ryan Joseph

> On Mar 31, 2017, at 3:55 PM, Gary Doades <[hidden email]> wrote:
>
> However, multiple independent compute units must be required for *true* parallelism. On a single processor any tasks running at the same time is just an illusion, normally created by the OS in time slicing between tasks based on certain criteria (priority, I/O, cpu usage etc.). That applies equally to threads or processes

Yeah exactly. Even if those are nodes on a network you need more than one. Unless you're making a bot-net or software designed for specific hardward which is known to have multiple cores, parallelism probably means using the GPU via an API like OpenCL, which is far cry from threading some tasks to run async.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Threading vs Parallelism ?

Tony Whyman
In reply to this post by Gary Doades

On 31/03/17 09:55, Gary Doades wrote:
However, multiple independent compute units must be required for *true* parallelism. On a single processor any tasks running at the same time is just an illusion, normally created by the OS in time slicing between tasks based on certain criteria (priority, I/O, cpu usage etc.). That applies equally to threads or processes
I think that what are referring to here is not so much *true* parallelism but that when parallelism is designed into an application, it enables real time parallel computing when the application is deployed.

For example, this distinction is very important in matrix algorithms. When operating on two matrices to produce another, the operations on each cell can be identified as n x m parallel actions at design time. At deployment time, it is often desirable to have a scalable implementation that can use anything from 1 to n x m processors to do the job. Thus you can have a design that identifies parallelism leading to an implementation that can non-parallel, partly or wholly parallel (in real time) depending on the size of the matrices and the number of processors available.

At what point does this become *true* parallelism?

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
123