Feature announcement: Dynamic array extensions

classic Classic list List threaded Threaded
94 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Mark Morgan Lloyd-5
On 07/06/18 10:45, Martok wrote:
>> What actually happens is that the memory is released back to the heap > (but *not* to the OS, at least on Linux), with the result that > concatenating elements will introduce a substantial hit particularly if > space for a new element allocated from the heap isn't contiguous.Writing a preallocating wrapper *where needed* for heavily grown arrays isfairly simple.
> The thing is: it's rarely worth it ;-)I recently did that for a wavefront mesh loader that appends vertices one-by-oneto an array, and for a scene with some 400k vertices, the difference was justsome milliseconds out of several seconds overall.Turns out the allocator usually finds a spot where the array doesn't need toactually be copied around for a while, and the pure bookkeeping of realloc isvery cheap.

The bit of code I was looking at was a mainframe emulator reading a tape
to find the block positions, I was appending elements individually using
a user-defined + and it wasn't exactly fast (resulting in a tape load
time probably slower than the hardware would have had 60 years ago)...
although obviously not as bad as Nitorami's suggestion that every
concatenation would result in a syscall.

I'd been expecting that SetLength() would do the same as it does with a
string, i.e. prune the valid range without releasing memory.

What I've done so far is only interim code and I've got lots of ways
round it, but the reality check was useful :-)

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Free Pascal - General mailing list
In reply to this post by Mark Morgan Lloyd-5
Mark Morgan Lloyd <[hidden email]> schrieb am Do., 7. Juni 2018, 11:46:
On 04/06/18 12:00, Nitorami wrote:
>>> It would be reasonable to assume that the predefined + might be>> substantially more efficient than a programmer-defined one could be.
>> Yes, that's one of the reasons I vote for keeping the new feature>and allow to overload the operator.
> I don't think that argument holds water. Concatenation of dynamic structuresis a slow function in the first place. I am not a core developer, but Iconsider that regardless how you do it, it will require a call to setlength,which in turn will call the operating system to allocate memory for the nowlarger structure. That takes much more time than a few assembly instructionsfor a normal pascal procedure call.

The obvious workaround is to preallocate a dynamic array to a nominal
size and then to trim it using SetLength(0), I'd assumed that it would
retain ownership of the memory but I've just checked.

What actually happens is that the memory is released back to the heap
(but *not* to the OS, at least on Linux), with the result that
concatenating elements will introduce a substantial hit particularly if
space for a new element allocated from the heap isn't contiguous.

Of course SetLength(0) releases the memory as the value of a dynamic array with length 0 is Nil, so nothing can be saved there. 

Regards, 
Sven

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Florian Klämpfl
In reply to this post by Martok
Am 07.06.2018 um 12:42 schrieb Martok:

>> What actually happens is that the memory is released back to the heap
>> (but *not* to the OS, at least on Linux), with the result that
>> concatenating elements will introduce a substantial hit particularly if
>> space for a new element allocated from the heap isn't contiguous.
> Writing a preallocating wrapper *where needed* for heavily grown arrays is
> fairly simple.
>
> The thing is: it's rarely worth it ;-)
> I recently did that for a wavefront mesh loader that appends vertices one-by-one
> to an array, and for a scene with some 400k vertices, the difference was just
> some milliseconds out of several seconds overall.
> Turns out the allocator usually finds a spot where the array doesn't need to
> actually be copied around for a while, and the pure bookkeeping of realloc is
> very cheap.
>

It would be worth though to think about extending the memory manager to make use of the realloc syscall (not sure about
its actual name) on some OSes as the OS can "copy" memory to a different virtual address by just re-mapping pages. I
think at least linux really does this.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Roger Rivero Jr. (SAMPA WebMaster)
In reply to this post by Free Pascal - General mailing list

Dear Sir:

I don´t like your way of answering. Period.

Mathematical operators on matrices (+, -, *, ~, etc.) have being defined centuries ago, and all the scientific community uses the same conventions on them. It´s not proper that somebody now would like to redefine the conventions on his own will, regarless of the user´s community opinions.

If you like an operator for concatenation, which is A WHOLE NEW FEATURE, please use a different or invented operator. You can use concatenator operarators borrowed from PHP ("."), Excel - VB ("&"), or invent your own one ( +.   +~   +*   etc)

I firmly oppose to any of your intentions (define the + operator for concatenation and allow user defined operators to take precedence). It doesn´t make sense. It´s not logical.

Thank you for hearing at me, and my apologies for my rough language.

Best regards,

Roger Rivero


El 02/06/2018 a las 8:10, Sven Barth via fpc-pascal escribió:
denisgolovan <[hidden email]> schrieb am Sa., 2. Juni 2018, 10:28:
Yes, I strongly support removing that functionality in favor of user operator overloads or vector-compatible way.

To clear something up: this new operator will definitely not be removed. Period. 

What might be done however (and what I had planned) is that existing overloads of the "+"-operator take precedence to the internal operator. 

Though I wouldn't mind introducing a syntax that can be used to force a element wise operation on a array. This way one wouldn't need to do the overload for the array, but the compiler would pick the operator of the element type instead. 

Regards, 
Sven 


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal




Avast logo

El software de antivirus Avast ha analizado este correo electrónico en busca de virus.
www.avast.com



_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

schuler
In reply to this post by Free Pascal - General mailing list
If I can, I would like to vote for "&" instead of "+". As I use Free Pascal with math (neural networks), I would vote for "&" with concatenation and "||" with "union" (in the case it's ever required).

On Mon, Jun 18, 2018 at 5:24 AM, Roger Rivero Jr. (SAMPA WebMaster) <[hidden email]> wrote:

Dear Sir:

I don´t like your way of answering. Period.

Mathematical operators on matrices (+, -, *, ~, etc.) have being defined centuries ago, and all the scientific community uses the same conventions on them. It´s not proper that somebody now would like to redefine the conventions on his own will, regarless of the user´s community opinions.

If you like an operator for concatenation, which is A WHOLE NEW FEATURE, please use a different or invented operator. You can use concatenator operarators borrowed from PHP ("."), Excel - VB ("&"), or invent your own one ( +.   +~   +*   etc)

I firmly oppose to any of your intentions (define the + operator for concatenation and allow user defined operators to take precedence). It doesn´t make sense. It´s not logical.

Thank you for hearing at me, and my apologies for my rough language.

Best regards,

Roger Rivero


El 02/06/2018 a las 8:10, Sven Barth via fpc-pascal escribió:
denisgolovan <[hidden email]> schrieb am Sa., 2. Juni 2018, 10:28:
Yes, I strongly support removing that functionality in favor of user operator overloads or vector-compatible way.

To clear something up: this new operator will definitely not be removed. Period. 

What might be done however (and what I had planned) is that existing overloads of the "+"-operator take precedence to the internal operator. 

Though I wouldn't mind introducing a syntax that can be used to force a element wise operation on a array. This way one wouldn't need to do the overload for the array, but the compiler would pick the operator of the element type instead. 

Regards, 
Sven 


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal





_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Free Pascal - General mailing list
Am 19.06.2018 um 00:57 schrieb Joao Schuler:
> If I can, I would like to vote for "&" instead of "+". As I use Free
> Pascal with math (neural networks), I would vote for "&" with
> concatenation and "||" with "union" (in the case it's ever required).
There won't be any voting as the "+" operator was chosen for Delphi
compatibility.

However I've worked out a solution using modeswitches that should
probably satisfy most people, now I only need to find the time to commit it.

Regards,
Sven
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Free Pascal - General mailing list
In reply to this post by Free Pascal - General mailing list
Am 20.05.2018 um 14:23 schrieb Sven Barth:

> Hello together!
>
> I'm pleased to announce that after nearly a year various extensions
> for dynamic arrays have been finished. This includes the following
> features:
>
> - support for array constructors using "[...]" syntax
> - support for Insert(), Delete() and Concat()
> - support for "+" operator
> - support for dynamic array constants (and variable initializations)

Addendum: the support for the "+" operator is now coupled to a new
modeswitch "ArrayOperators".

If the modeswitch is enabled, then the operator can not be overloaded
and it also won't be used even if an overload from a unit without the
modeswitch is in scope (the compiler however issues a warning if this is
the case, so you can either remove the overload or disable the
modeswitch). If the modeswitch is not enabled then everything behaves as
before.

The modeswitch is enabled by default in Delphi modes as this feature was
added for Delphi compatibility. If you want to disable it in those modes
then you need to do this:

=== code begin ===

{$mode delphi}
{$modeswitch arrayoperators-} // Note the "-"

=== code end ===

Regards,
Sven
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Mark Morgan Lloyd-5
On 20/06/18 20:00, Sven Barth via fpc-pascal wrote:

> Addendum: the support for the "+" operator is now coupled to a new
> modeswitch "ArrayOperators".
> If the modeswitch is enabled, then the operator can not be overloaded
> and it also won't be used even if an overload from a unit without the
> modeswitch is in scope (the compiler however issues a warning if this is
> the case, so you can either remove the overload or disable the
> modeswitch). If the modeswitch is not enabled then everything behaves as
> before.
> The modeswitch is enabled by default in Delphi modes as this feature was
> added for Delphi compatibility. If you want to disable it in those modes
> then you need to do this:
> === code begin ===
> {$mode delphi}{$modeswitch arrayoperators-} // Note the "-"
> === code end ===

Sven, please could we have a further update when you decide what version
of the compiler will get the new facility, so that those of us with code
that might clash can make sure we have version-specific conditionals in
place.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Mattias Gaertner
In reply to this post by Free Pascal - General mailing list
On Wed, 20 Jun 2018 21:56:56 +0200
Sven Barth via fpc-pascal <[hidden email]> wrote:

>[...]
> The modeswitch is enabled by default in Delphi modes as this feature was
> added for Delphi compatibility.

I would like to use that for objfpc.
Is there a command line flag to enable a modeswitch?

Mattias
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Ryan Joseph
In reply to this post by Free Pascal - General mailing list


> On Jun 21, 2018, at 2:56 AM, Sven Barth via fpc-pascal <[hidden email]> wrote:
>
> Addendum: the support for the "+" operator is now coupled to a new modeswitch "ArrayOperators".

Finally got to try this. It’s really satisfying being able to do:

var
        arr: array of integer = (1, 2, 3);
        i: integer;
begin
        arr += [4];
        for i in arr do
                writeln(i);


thanks!



Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Mattias Gaertner
On Thu, 21 Jun 2018 15:17:39 +0700
Ryan Joseph <[hidden email]> wrote:

>[...]
> Finally got to try this. It’s really satisfying being able to do:
>[...]
> for i in arr do

Note: This was supported for a long time.

Mattias
 
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Ryan Joseph
I mean the += [] part. :)

> On Jun 21, 2018, at 3:37 PM, Mattias Gaertner <[hidden email]> wrote:
>
> Note: This was supported for a long time.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Free Pascal - General mailing list
In reply to this post by Mark Morgan Lloyd-5
Mark Morgan Lloyd <[hidden email]> schrieb am Do., 21. Juni 2018, 09:11:
On 20/06/18 20:00, Sven Barth via fpc-pascal wrote:

> Addendum: the support for the "+" operator is now coupled to a new
> modeswitch "ArrayOperators".
> If the modeswitch is enabled, then the operator can not be overloaded
> and it also won't be used even if an overload from a unit without the
> modeswitch is in scope (the compiler however issues a warning if this is
> the case, so you can either remove the overload or disable the
> modeswitch). If the modeswitch is not enabled then everything behaves as
> before.
> The modeswitch is enabled by default in Delphi modes as this feature was
> added for Delphi compatibility. If you want to disable it in those modes
> then you need to do this:
> === code begin ===
> {$mode delphi}{$modeswitch arrayoperators-} // Note the "-"
> === code end ===

Sven, please could we have a further update when you decide what version
of the compiler will get the new facility, so that those of us with code
that might clash can make sure we have version-specific conditionals in
place.

New language features are normally not merged to a fixes branch (there were exceptions in the past, but they weren't too big). So if you check for >= 3.1.1 you're set. 

Regards, 
Sven 

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Dynamic array extensions

Free Pascal - General mailing list
In reply to this post by Mattias Gaertner
Mattias Gaertner <[hidden email]> schrieb am Do., 21. Juni 2018, 09:24:
On Wed, 20 Jun 2018 21:56:56 +0200
Sven Barth via fpc-pascal <[hidden email]> wrote:

>[...]
> The modeswitch is enabled by default in Delphi modes as this feature was
> added for Delphi compatibility.

I would like to use that for objfpc.
Is there a command line flag to enable a modeswitch?

Not tested, but -MArrayOperators *should* work (it takes both modes and modeswitches). 
One point to keep in mind however: any single modeswitch setting is overwritten once a mode is encountered, this includes both the $mode directive and the -M option (or e.g. -S2). 

Regards, 
Sven 

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
12345