Happy tickets benchmark

classic Classic list List threaded Threaded
75 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark [OT]

el_es
On 18/02/16 11:15, Serguei TARASSOV wrote:

> On 18/02/2016 12:00, [hidden email] wrote:
>> Date: Wed, 17 Feb 2016 18:55:29 +0100
>> From: Adrian Veith<[hidden email]>
>> To: FPC-Pascal users discussions<[hidden email]>
>> Subject: Re: [fpc-pascal] Happy tickets benchmark
>>
>> I don't want to insist on this, but: if you measure the runtime of your
>> program you result = runtime + error. If you measure a series against
>> MIN you measure MIN(result) = runtime + MIN(error) which delivers the
>> best value for runtime.
> Not at all, any series against MIN delivers the best value for MIN(result) not for runtime!
>
Std deviation also matters:
> Ex.
> Prog 1
> Run 1: 130 = 100 + 30
> Run 2: 160 = 100 + 60
> Run 3: 170 = 100 + 70

> Min = 130, Avg = 153
>
std dev = 17

> Prog 2
> Run 1: 120 = 110 + 10
> Run 2: 150 = 110 + 40
> Run 3: 200 = 110 + 90
> Min = 120, Avg = 157

std dev = 33

> MIN shows that Prog 2 is faster that is wrong.
> AVG shows that Prog 1 is faster.
>
Std dev shows that runs of Prog2 have more error
so its measurements can't be trusted /more/ than Prog1.

The sample also is too small for meaningful statistics ;)

> Regards,
> Serguei

-L.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Serguei TARASSOV
In reply to this post by Serguei TARASSOV
On 17/02/2016 12:00, [hidden email] wrote:

> Date: Tue, 16 Feb 2016 12:48:31 +0100 (CET)
> From: Michael Van Canneyt<[hidden email]>
> To: FPC-Pascal users discussions<[hidden email]>
> Subject: Re: [fpc-pascal] Happy tickets benchmark
>
>
> I have asked Florian to integrate his patch anyway, he has agreed,
> so I imagine it will result in a new -OoNNN switch.
>
> Michael.
Good news.
Do you have any ideas why this kind of optimization is special?

For info, simple loop test like

   while i < 1000000000 do
     i := i + 1;

shows that the FPC code is 2 times slower than Delphi 7 and Borland C
5.5 and 4 times slower that C#.

Regards,
Serguei

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
--
Regards,
Serguei
Reply | Threaded
Open this post in threaded view
|

Re: fpc-pascal Digest, Vol 140, Issue 69

Serguei TARASSOV
In reply to this post by el_es
On 19/02/2016 12:00, [hidden email] wrote:
Message: 2
Date: Fri, 19 Feb 2016 09:16:15 +0000
From: Lukasz Sokol [hidden email]
To: [hidden email]
Subject: Re: [fpc-pascal] Happy tickets benchmark [OT]

Std deviation also matters:
Std dev shows that runs of Prog2 have more error 
so its measurements can't be trusted /more/ than Prog1.

The sample also is too small for meaningful statistics ;)
Exactly, the variance too. Many things matter.
My example shows only that AVG estimation is better than MIN in the simple technical test with 10 runs because of average the runtime "noise".
I suppose it could be approved more strictly.

Regards,
Serguei

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
--
Regards,
Serguei
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Jonas Maebe-2
In reply to this post by Serguei TARASSOV
Serguei TARASSOV wrote:
> For info, simple loop test like
>
>    while i < 1000000000 do
>      i := i + 1;
>
> shows that the FPC code is 2 times slower than Delphi 7 and Borland C
> 5.5 and 4 times slower that C#.

If that's really all there is in your program, then the C# compiler
probably replaces that with "i+=1000000000;" (gcc and clang definitely
will do that, I don't know about Delph 7/Borland C 5.5). This is another
kind of optimisation that is seldom useful in real world programs
(normally the loop will also do something useful, in which case you
can't do that).

FPC indeed does not implement many of these optimisations, because we
prefer to spend our time on other things. If you, or anyone else, wants
to implement transformations for such idioms, you're welcome and we'll
happily apply your patches if they are implemented as generic node tree
optimisation passes.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

leledumbo
Administrator
In reply to this post by Serguei TARASSOV
> Do you have any ideas why this kind of optimization is special?

Didn't Florian said that this kind of optimization has no benefit in real world programs and will only increase compilation time?

> For info, simple loop test like
>
>   while i < 1000000000 do
>     i := i + 1;
>
> shows that the FPC code is 2 times slower than Delphi 7 and Borland C
> 5.5 and 4 times slower that C#.

Just checked such a code with gcc 5.3.0, in -O2 the generated assembly is:

        xorl %eax, %eax
        ret

which literally does nothing. Without optimization the speed is equal.
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Serguei TARASSOV
In reply to this post by Serguei TARASSOV
On 19/02/2016 20:20, [hidden email] wrote:

> Date: Fri, 19 Feb 2016 14:01:16 +0100
> From: Jonas Maebe<[hidden email]>
> To: FPC-Pascal users discussions<[hidden email]>
> Subject: Re: [fpc-pascal] Happy tickets benchmark
>
> Serguei TARASSOV wrote:
>> >For info, simple loop test like
>> >
>> >    while i < 1000000000 do
>> >      i := i + 1;
>> >
>> >shows that the FPC code is 2 times slower than Delphi 7 and Borland C
>> >5.5 and 4 times slower that C#.
> If that's really all there is in your program, then the C# compiler
> probably replaces that with "i+=1000000000;" (gcc and clang definitely
> will do that, I don't know about Delph 7/Borland C 5.5). This is another
> kind of optimisation that is seldom useful in real world programs
> (normally the loop will also do something useful, in which case you
> can't do that).
MSVC does (hence I didn't include it) but all previously listed
compilers don't remove the loop.
The time ratio is about
FPC - 100, Delphi 7, Borland C - 50, C# - 25, MSVC - 0.

So my doubts was about extra-code generation for the loops.

Regards,
Serguei

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
--
Regards,
Serguei
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Serguei TARASSOV
In reply to this post by Serguei TARASSOV
On 19/02/2016 20:20, [hidden email] wrote:
> Date: Fri, 19 Feb 2016 11:49:57 -0700 (MST) From: leledumbo
> <[hidden email]> To: [hidden email]
> Subject: Re: [fpc-pascal] Happy tickets benchmark
>> >Do you have any ideas why this kind of optimization is special?
> Didn't Florian said that this kind of optimization has no benefit in real
> world programs and will only increase compilation time?
After more than 20 years in software engineering I'm avoiding to use the
phrases like "real world program" or I put in quotes the word "real" at
least :)

>
>> >For info, simple loop test like
>> >
>> >   while i < 1000000000 do
>> >     i := i + 1;
>> >
>> >shows that the FPC code is 2 times slower than Delphi 7 and Borland C
>> >5.5 and 4 times slower that C#.
> Just checked such a code with gcc 5.3.0, in -O2 the generated assembly is:
>
> xorl %eax, %eax
> ret
>
> which literally does nothing. Without optimization the speed is equal.
>
It shows that gcc do the same kind of optimization as MSVC (hence I
didn't include it), but other don't.
See my answer to Jonas for more details.

Regards,
Serguei
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
--
Regards,
Serguei
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Serguei TARASSOV
In reply to this post by Serguei TARASSOV
Hello,

Some updates of tests.

I added a simple assembler code as a reference of the "minimally poor code".

GCC has a good optimizer reducing the time in two.
C#/IL has about the same result.
Unfortunately, FPC (3.0.0 64 bits) is always under the "minimally poor".

In addition :
- changing types of n1-n8 variables to "longint" increases the time to 10%
- changing types to "integer" doubles the time!! In contrast, with Delphi 10 it decreases the time by 60%.
--
Regards,
Serguei
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

vfclists .
In reply to this post by Serguei TARASSOV


On 14 February 2016 at 10:06, Serguei TARASSOV <[hidden email]> wrote:
Hello,

thank all for assistance!

Sorry, I was not clear, the series should be ran with all tests _on the same computer_ regardless its hardware capacity and on the _same OS_. That's why I cannot compare with Delphi.

So if you have modern Delphi, FPC and maybe .NET on your Windows computer please compare them with about 5-10 runs to get average time.

Thank you for the Inc() hint, I added it to the comments, it gains about 10% for me.
However, in Delphi Inc() is faster until you turn on the checking overflow.

Another strange effect in FPC.
Only longint shows correct result. With the integer type the optimizer seems to replace type with shortint (16-bits) and I see 31902 instead of 4816030.



Is there a reason for this, or is it a bug?


--
Frank Church

=======================
http://devblog.brahmancreations.com

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Jeppe Johansen-3
On 03/02/2016 12:48 AM, vfclists . wrote:


On 14 February 2016 at 10:06, Serguei TARASSOV <[hidden email]> wrote:
Hello,

thank all for assistance!

Sorry, I was not clear, the series should be ran with all tests _on the same computer_ regardless its hardware capacity and on the _same OS_. That's why I cannot compare with Delphi.

So if you have modern Delphi, FPC and maybe .NET on your Windows computer please compare them with about 5-10 runs to get average time.

Thank you for the Inc() hint, I added it to the comments, it gains about 10% for me.
However, in Delphi Inc() is faster until you turn on the checking overflow.

Another strange effect in FPC.
Only longint shows correct result. With the integer type the optimizer seems to replace type with shortint (16-bits) and I see 31902 instead of 4816030.



Is there a reason for this, or is it a bug?
The Integer type depends on what compiler mode you are in, and what operating system. Sometimes it's 32bit and other times it's 16bit.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Serguei TARASSOV
This post was updated on .
Jeppe Johansen-3 wrote
On 03/02/2016 12:48 AM, vfclists . wrote:
>
>
> On 14 February 2016 at 10:06, Serguei TARASSOV <[hidden email] 
> <mailto:[hidden email]>> wrote:
>     Another strange effect in FPC.
>     Only longint shows correct result. With the integer type the
>     optimizer seems to replace type with shortint (16-bits) and I see
>     31902 instead of 4816030.
>
> Is there a reason for this, or is it a bug?
The Integer type depends on what compiler mode you are in, and what
operating system. Sometimes it's 32bit and other times it's 16bit.
It was FPC 2.6.4 64 bits in FPC mode on Linux. See the sources to reproduce.
Change:
TicketsCount: longint;
to
TicketsCount: integer;

Compile: fpc -O2 -Cr- HappyTickets.pas

Result: Found 31902 tickets. Elapsed time, msec: 203

I don't see any reason to use 16-bits integer in this code much less in 64-bits mode.

I don't test it yet in FPC 3.0 but 3.0 has other problems with my test:
- changing the type of n1-n8 from 0..9 to longint increases the time by 10%
- changing the type of n1-n8 from 0..9 to integer doubles the time!

On the contrary, in Delphi 10 changing the type of n1-n8 from 0..9 to integer decreases the time by 60%
--
Regards,
Serguei
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Jonas Maebe-2

Serguei TARASSOV wrote on Thu, 03 Mar 2016:

> Jeppe Johansen-3 wrote
>>>
>> The Integer type depends on what compiler mode you are in, and what
>> operating system. Sometimes it's 32bit and other times it's 16bit.

AFAIK it only depends on the compiler mode.

> It was FPC 2.6.4 64 bits in FPC mode on Linux. See the sources to reproduce.
> Change:
> TicketsCount: longint;
> to
> TicketsCount: integer;
>
> Compile: fpc -O2 -Cr- HappyTickets.pas
>
> Result: Found 31902 tickets. Elapsed time, msec: 203
>
> I don't see any reason to use 16-bits integer in this code much less in
> 64-bits mode.

The reason is, as always, compatibility. FPC mode started as an  
extension of the Turbo Pascal compatibility mode. In Turbo Pascal,  
integer is 16 bits. In Delphi, it's 32 bits, so both in Delphi and in  
ObjFPC modes, integer is 32 bits. Code that was written with integer =  
16 bits may no longer work the same if the size of the integer type is  
changed to 32 bits (especially if overflow checking is off).

> I don't test it yet in FPC 3.0 bit 3.0 has other problems with my test:
> - changing the type of n1-n8 from 0..9 to longint increases the time by 10%
> - changing the type of n1-n8 from 0..9 to integer doubles the time!
>
> On the contrary, in Delphi 10 changing the type of n1-n8 from 0..9 to
> integer decreases the time by 60%

If you want to compare with Delphi, it's better to compile the program  
in Delphi mode. You should see the same 10% increase in FPC that you  
see when you change it to longint.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Serguei TARASSOV
Jonas Maebe-2 wrote
The reason is, as always, compatibility. FPC mode started as an  
extension of the Turbo Pascal compatibility mode. In Turbo Pascal,  
integer is 16 bits. In Delphi, it's 32 bits, so both in Delphi and in  
ObjFPC modes, integer is 32 bits. Code that was written with integer =  
16 bits may no longer work the same if the size of the integer type is  
changed to 32 bits (especially if overflow checking is off).
...
If you want to compare with Delphi, it's better to compile the program  
in Delphi mode. You should see the same 10% increase in FPC that you  
see when you change it to longint.

Jonas
If I understood correctly, the FPC programmer in 2016 always should include {$MODE DELPHI} or {$MODE ObjFPC} to avoid TurboPascal legacy that I use in earlies 1990?
Ok, it's not so important, only a rhetorical question.

I retested in Delphi mode, the time is the same.
Changing type to integer with Delphi7 decrease the time from 215 to 155 msec in my environment. FPC shows about 189 msec and the same 155 in delphi mode with integer type.

So the default FPC mode seems to be significantly more slow than Delphi-mode and ObjFPC-mode.
--
Regards,
Serguei
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Jonas Maebe-2

Serguei TARASSOV wrote on Thu, 03 Mar 2016:

> If I understood correctly, the FPC programmer in 2016 always should include
> {$MODE DELPHI} or {$MODE ObjFPC} to avoid TurboPascal legacy that I use in
> earlies 1990?

FPC mode started in those same early 1990s. The first code that was  
written for this mode dates from that period as well.

> Ok, it's not so important, only a rhetorical question.

On the contrary, it is fundamental. Changing defaults in existing  
language modes, or changing the default language mode that is used,  
would break existing code with as only reason that we wouldn't want to  
look bad when people use the compiler for the first time and don't  
realise what the default mode is/entails. We're not about projecting a  
particular image, we're about trying not to break things that already  
work as far as the language is concerned.

Look at how Inprise/CodeGear/... alienated many developers by (a.o.)  
breaking backwards compatibility all the time and pushing what was  
supposed to be the next greatest thing instead. It's just not fair to  
existing users, who mainly don't want their code to break (just like  
you with the with/property problem earlier on). Any improvements on  
top of that is gravy, but it's not the most important.

> So the default FPC mode seems to be significantly more slow than Delphi-mode
> and ObjFPC-mode.

It's unrelated to the compiler mode. However, performing 16 bit  
integer arithmetical instructions on a processor that is very  
inefficient at doing so (such as 32 and 64 bit Intel x86 processors  
running in 32 or 64 bit mode -- I don't know about AMD) can indeed be  
quite slow. Ideally, the compiler would internally "upgrade" those  
operations to 32/64 bit where possible, but it doesn't do so. The  
amount of 16 bit integer code that should run as fast as possible on  
32/64 bit x86 Intel processors is probably not that big either.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Happy tickets benchmark

Serguei TARASSOV
This post was updated on .
Jonas Maebe-2 wrote
On the contrary, it is fundamental. Changing defaults in existing  
language modes, or changing the default language mode that is used,  
would break existing code with as only reason that we wouldn't want to  
look bad when people use the compiler for the first time and don't  
realise what the default mode is/entails. We're not about projecting a  
particular image, we're about trying not to break things that already  
work as far as the language is concerned.
I agree your point but with some limitations.
IMHO, the period of 10 years as a maximum is very sufficient for migration or we stay with the old version of compiler and common libraries.
On other hand, there are different types of compatibility breaking.
The TP mode can be applied globally by compiler switch and it solve the problem of default mode.
But adding the functions/methods/properties to existing classes of common libraries breaks the old code that should be rewritten (if the error is detected by compiler).
The second case is really important because there is no workaround.
--
Regards,
Serguei
1234