Feature announcement: Management Operators

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Feature announcement: Management Operators

Maciej Izak
Hi,

I'm pleased to finally announce (again but now officially ;) ) the additional record operators: Initialize, Finalize, AddRef and Copy. Available in latest FPC trunk (r35485):

1. operator Initialize - is called after memory allocation for record (called *after* FPC internal call recordrtti(data,typeinfo,@int_initialize);)

2. operator Finalize -  is called when record goes out of scope (called *before* FPC internal call recordrtti(data,typeinfo,@int_finalize);)

3. operator AddRef - is called after the contents of a record has been duplicated by copying the contents byte by byte (called *after* FPC internal call recordrtti(data,typeinfo,@int_addref);  )

4. operator Copy - Copy operator (if exists) is called *instead of* default copy behavior. Operator is responsible for copying everything that's needed from the source to the target.

AddRef and Copy might be at beginning a little hard for understanding but we have good explanation provided by Thorsten Engler in topic "Management operators : Copy and Clone confusion..." :

=== quote begin ===
the reason for both to exist is that there are cases (e.g.
passing an interface parameter) where the callee is responsible for any
lifetime management implications (and the caller might not even be
implemented using fpc). The caller just passes the interface pointer on the
stack (or in a register), which, while it creates a copy of the bytes, does
not actually invoke the copy operator. In cases where reference counting is
required (parameter not marked as const), the callee may decide to simply
use the value that was passed on the stack as the backing for the parameter,
so it calls AddRef on that, without performing another copy.

Given this context, I find the terms AddRef and Copy to be perfectly
adequate, even when applied to types that aren't strictly speaking
performing reference counting; e.g. IIRC, AddRef for a BSTR (ole string type
for windows which does NOT have a reference count) actually involves making
a copy of the string contents and replacing the BSTR pointer.

AddRef here means "this is now an *additional* *reference* to whatever these
bytes might refer to".
=== quote end ===

my 2 cents for above: AddRef exist to speed up things.

New set of operators is unique and is called "management operators". That is because:

A. each of record (even non managed or even empty) with management operator becomes managed type.

B. is possible to implement new custom types (also thanks to other operators) with own management of memory (new string types, fast TValue implementations without hacks on RTL etc.).

C. "management operators" have no result.

D. For management operators is generated simple VMT (thanks to this is possible to work with management operators together with all RTL functions like InitializeArray/FinalizeArray).

=== example declaration begin ===
  PFoo = ^TFoo;
  TFoo = record
  private
    class operator Initialize(var a: TFoo);
    class operator Finalize(var a: TFoo);
    class operator AddRef(var a: TFoo);
    class operator Copy(constref aSrc: TFoo; var aDst: TFoo);
  end;
=== example declaration end ===

Management Operators feature can be used for many things:

* support for value types
* nullable types
* some custom ARC variations
* speed up existing types
* very fast RTTI.TValue implementation
* as replacement for manually called Init/Done record methods like in mORMot for many types (for example SynCommons.TSynLocker).
* auto init/finit for pointers/classes/simple types or whatever we have in Pascal

It works correctly in all possible ways with RTL: 

* New (Initialize)
* Dispose (Finalize)
* Initialize (Initialize)
* Finalize (Finalize)
* InitializeArray (Initialize)
* FinalizeArray (Finalize)
* SetLength (Initialize/Finalize)
* Copy (AddRef)
* RTTI.IsManaged

Managements operators are often called implicitly in many cases. Few examples:

* global variables (Initialize/Finalize)
* local variables (Initialize/Finalize)
* for fields inside: records, objects, classes (Initialize/Finalize)
* variable assignment (Copy)
* for parameters for routines (AddRef/Finalize/none - depends on modifiers like var/constref/cons)

Own experiments recommended! :)

Complex example and test:

Note: Lazarus (trunk version) has support for all new operators syntax (thanks for Mattias Gaertner).

Future directions and opened doors (where management operators can be used but not directly):

+ "default" field for better handling smart pointers / nullable types (maybe will be introduced new type called proxy record for handling "default" modifier) (eg. https://github.com/maciej-izak/PascalSmartPointers compilable with NewPascal)
+ New VMT entry "vmtManagedInit" for "Initialize operator" for record fields inside classes (optimization)
+ Base for Nullable types syntax sugar
+ Probably base in some way for ARC classes (DELPHINEXTGEN compiler mode)
+ More Oxygene compatibility
+ Aspects

Note for myself: document the feature on the wiki in New Features Trunk and the changes to the Rtti in User Changes Trunk :)

--
Best regards,
Maciej Izak

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

denisgolovan
Thanks a lot.
I guess http://bugs.freepascal.org/view.php?id=30687 should be closed now :)
 
-- 
Regards,
Denis Golovan
 

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Maciej Izak
2017-02-28 12:28 GMT+01:00 denisgolovan <[hidden email]>:
Thanks a lot.
I guess http://bugs.freepascal.org/view.php?id=30687 should be closed now :)

I don't have such powers ;)

--
Best regards,
Maciej Izak

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Michael Van Canneyt


On Tue, 28 Feb 2017, Maciej Izak wrote:

> 2017-02-28 12:28 GMT+01:00 denisgolovan <[hidden email]>:
>
>> Thanks a lot.
>> I guess http://bugs.freepascal.org/view.php?id=30687 should be closed now
>> :)
>>
>
> I don't have such powers ;)

You do now. I took care of that :)

But I set it to fixed for you. Seeing you were the reporter, you can set it
to closed :)

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Michael Van Canneyt


On Tue, 28 Feb 2017, Michael Van Canneyt wrote:

>
>
> On Tue, 28 Feb 2017, Maciej Izak wrote:
>
>> 2017-02-28 12:28 GMT+01:00 denisgolovan <[hidden email]>:
>>
>>> Thanks a lot.
>>> I guess http://bugs.freepascal.org/view.php?id=30687 should be closed now
>>> :)
>>>
>>
>> I don't have such powers ;)
>
> You do now. I took care of that :)
>
> But I set it to fixed for you. Seeing you were the reporter, you can set it
> to closed :)

By the way, thanks for the contribution.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Maciej Izak
In reply to this post by Michael Van Canneyt

2017-02-28 13:19 GMT+01:00 Michael Van Canneyt <[hidden email]>:
You do now. I took care of that :)

Thanks ^^ now I can officially fix bugs related to new feature. Mhm. Great... ;)

--
Best regards,
Maciej Izak

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

noreply
In reply to this post by Maciej Izak
On 2017-02-28 05:20, Maciej Izak wrote:

> It works correctly in all possible ways with RTL:
>
> * New (Initialize)
> * Dispose (Finalize)

What happens with a stack allocated record?
(no new() required, nor dispose() so is the record ever
initialized/finalized?)

Or stack records are forbidden?

One reason I love records is because you can create a stack record
without any worry of freeing and creating it on heap, a huge advantage
over OOP programming in delphi/fpc where everything is heap object
(except old turbo pascal objects).

Sorry, I've been out of touch with what's happening with Records lately
in modern pascal/delphi/etc.
(If it sounds like a silly newbie question, forgive me, I am not up on
the latest and greatest!)
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Jonas Maebe-3
On 28/02/17 21:40, [hidden email] wrote:
> What happens with a stack allocated record?
> (no new() required, nor dispose() so is the record ever
> initialized/finalized?)

Yes, just like records that contain e.g. an ansistring field.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Graeme Geldenhuys-6
In reply to this post by noreply
On 2017-02-28 20:40, [hidden email] wrote:
> Sorry, I've been out of touch with what's happening with Records lately
> in modern pascal/delphi/etc.

Don't worry, I'm with you on that one. All I know is that
Borland/CodeGear/EMBT broken TP Objects somewhere in Delphi. Then they
decided they want that functionality back, so instead of fixing TP
Objects, the decided to butcher Records to act just like TP Objects
(well, pretty much)! Go figure.

I mostly code with classes, but do still us TP Objects mixed in with my
code for simpler data structures where I don't really want to
instantiate objects. Records - well, I use them like they were defined
in Delphi 7 and earlier. I really don't see the need for Records with
Methods (ie: TP Objects) and such.

Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

noreply
On 2017-02-28 14:50, Graeme Geldenhuys wrote:

> On 2017-02-28 20:40, [hidden email] wrote:
>> Sorry, I've been out of touch with what's happening with Records
>> lately
>> in modern pascal/delphi/etc.
>
> Don't worry, I'm with you on that one. All I know is that
> Borland/CodeGear/EMBT broken TP Objects somewhere in Delphi. Then they
> decided they want that functionality back, so instead of fixing TP
> Objects, the decided to butcher Records to act just like TP Objects
> (well, pretty much)! Go figure.
>
> I mostly code with classes, but do still us TP Objects mixed in with my
> code for simpler data structures where I don't really want to
> instantiate objects. Records - well, I use them like they were defined
> in Delphi 7 and earlier. I really don't see the need for Records with
> Methods (ie: TP Objects) and such.
>
> Regards,
>   Graeme
>

There are many cases where I do not want to free and create an object
and just use something on the stack instead. TP objects kind of offered
a solution to this, but not really, since if you used an old TP object
which was designed to be on the heap, but you used in on the stack
instead - problems would occur...

I'm fond of what Oberon and Golang are doing to try and solve the issue,
but, that's garbage collected rather than stack.  And in those languages
you have to send the self parameter in as your own (there is no hidden
self, you create your own self shortform)
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

noreply
In reply to this post by Jonas Maebe-3
On 2017-02-28 14:48, Jonas Maebe wrote:
> On 28/02/17 21:40, [hidden email] wrote:
>> What happens with a stack allocated record?
>> (no new() required, nor dispose() so is the record ever
>> initialized/finalized?)
>
> Yes, just like records that contain e.g. an ansistring field.
>
>
> Jonas


do records which contain ansistring fields have some special stack
initialization system (sorry, I should really read the source code in
this case - still interested in theory of how this works)
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Michael Van Canneyt


On Sat, 4 Mar 2017, [hidden email] wrote:

> On 2017-02-28 14:48, Jonas Maebe wrote:
>> On 28/02/17 21:40, [hidden email] wrote:
>>> What happens with a stack allocated record?
>>> (no new() required, nor dispose() so is the record ever
>>> initialized/finalized?)
>>
>> Yes, just like records that contain e.g. an ansistring field.
>>
>>
>> Jonas
>
>
> do records which contain ansistring fields have some special stack
> initialization system (sorry, I should really read the source code in
> this case - still interested in theory of how this works)

Yes. They are initialized.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Benito van der Zander
Hi,

with a function call: http://bugs.freepascal.org/view.php?id=30333


I had benchmark runs where it spend more time in the initializing function than doing any calculations

Best,
Benito 



On 03/04/2017 06:41 PM, Michael Van Canneyt wrote:


On Sat, 4 Mar 2017, [hidden email] wrote:

On 2017-02-28 14:48, Jonas Maebe wrote:
On 28/02/17 21:40, [hidden email] wrote:
What happens with a stack allocated record?
(no new() required, nor dispose() so is the record ever
initialized/finalized?)

Yes, just like records that contain e.g. an ansistring field.


Jonas


do records which contain ansistring fields have some special stack initialization system (sorry, I should really read the source code in this case - still interested in theory of how this works)

Yes. They are initialized.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Hans-Peter Suter
In reply to this post by Graeme Geldenhuys-6
2017-02-28 21:50 GMT+01:00 Graeme Geldenhuys <[hidden email]>:
 Records - well, I use them like they were defined
in Delphi 7 and earlier. I really don't see the need for Records with
Methods (ie: TP Objects) and such.

Go with modern times! It's the 21st century now ;-)

(For inspiration have e.g. a look at Julia [1] and how they develop incl. syntax changes. - Many discussions/decisions are being done using github issues, here e.g. a quite fundamental keyword change [2, 3]. As you see in this issue, it is often not easy to find *the* good solution, but it's great, how opinions have a place to flow in and can be considered/integrated by the core or other talented devs. - There are branches: people can help but are also told to wait if the timing is not good [4]. - There is a manual, stdlib, devdoc [5] which gets you started quickly and one can easily contribute small things like typos. - A one-stop discourse based forum [6] which is a joy to use and much easier than e.g. the multitude of fpc-maillist, lazarus-forum, newpascal-/mse-<don't-know-atm>-lists. - There's a one-stop binary download location [7] and/or install master from source, just a ~ 'git clone, make install' away (llvm is not as fast as fpc though;). - In general uniform syntax (even with `end` like in pascal:) which doesn't make you guess which style/convention to choose. - Organisations which integrate people interested in a specific subject. e.g. [8]. - Package management is currently a bit 'suboptimal' but a new one is being developed, here the propositions [9] where interested people can give input in the design phase, and later while implementing. --- Of course one cannot and should not compare !!!   Julia doesn't have a 25+ year long legacy with many (partly a bit obscure) platforms to conserve and maintain. (It 'stole' the compiler, à la MSElang;-)  It has developers which work since ~7 years ~professionally and there is quite some funding and many many contributors. But as a maybe longterm inspiration, why not? Maybe some parts are worth stealing? And, with tool-support, maybe possible to steal?)

(Ähem, I digressed a bit from the Delphi 7 record topic... I am sorry (more so, considering I forked from you))

(I agree with what you wrote about 'polluting pascal code'. But what can one do? For me, atm, it is super-important that fpc is very compatible with delphi, e.g. generics (and that they work well). Thus, very grateful to people who contribute such things). 


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Feature announcement: Management Operators

Graeme Geldenhuys-6
On 2017-03-04 18:59, Hans-Peter Suter wrote:
> Go with modern times! It's the 21st century now ;-)

I am, that's why I am starting to use Java more and more. ;-) A very
nice language, multiple choices in IDE's, fantastic debugger, fantastic
contributions and ecosystem.

When I think Records, I think of trying to read a binary file off disk
in a complex layout. Records were (and still are) used to read such
complex binary files. Methods in records have no place there.

Now if I work with objects, I use Classes or Objects - both were
designed for that purpose.


> (I agree with what you wrote about 'polluting pascal code'. But what can
> one do?

Don't be a lemming! We can all see in what trouble Delphi has been for
the last 15+ years. They are grasping at straws trying to stay above
water. They f*cked up so many things during that time and became a very
buggy platform to develop in.

Free Pascal and Lazarus has a large following already, and is well
established and a very solid compiler. Even EMBT proved that by using
FPC is one of the Delphi releases (XE2 I think).

Free Pascal doesn't need to follow Delphi over the cliff. My personal
experience has proven that FPC is just as good... no wait, better that
Delphi (multiple CPU support, multiple OS support, mobile support,
embedded support etc). So I simply use FPC for everything! I don't need
cross-compiler compatibility.


> e.g. generics (and that they work well).

I'm warming up to Generics (in Object Pascal) - but that is again more
so because I'm also using Java, and in Java, generics is very often used.


Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal