reference-counted function results not initialized to nil

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

reference-counted function results not initialized to nil

David Emerson
Hi all,

After upgrading fpc 2.6.4 -> 3.0.0, I'm seeing a bug where (as noted in
subject) reference-counted function results are not being initialized to
nil.

Is this a known bug?

I am seeing it in ansistrings and also dynamic arrays. Here is an
example program to see it with ansistrings:

function new_string : ansistring;
   begin
     new_string := new_string + 'x';
   end;

begin
  writeln (new_string); // outputs: x
  writeln (new_string); // outputs: xx
  writeln (new_string); // outputs: xxx
end.

Reproducing with dynamic arrays seems to be a bit trickier, I haven't
made a test case yet, but I think I will be able to if required.

What to do?

Thanks,
David


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Michalis Kamburelis-3
> After upgrading fpc 2.6.4 -> 3.0.0, I'm seeing a bug where (as noted in
> subject) reference-counted function results are not being initialized to
> nil.

They were never guaranteed to be initialized to nil.

Reference-counted types (unlike other types) cannot contain memory
garbage. But it doesn't mean that they are always initialized empty
when the function starts. You need to explicitly do Result := '' if
your code reads the Result later.

See similar questions for Delphi:
http://stackoverflow.com/questions/5336863/what-is-the-default-value-of-result-in-delphi
http://stackoverflow.com/questions/5314918/do-i-need-to-setlength-a-dynamic-array-on-initialization/5315254#5315254

Luckily, FPC warns about it, at least in my simple test:

$ fpc -vw a.lpr && ./a
a.lpr(5,20) Warning: function result variable of a managed type does
not seem to be initialized
blabla
blablablabla
blablablablablabla

Regards,
Michalis

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

a.lpr (204 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

David Emerson
On 06/24/2016 08:19 PM, Michalis Kamburelis wrote:

>> After upgrading fpc 2.6.4 -> 3.0.0, I'm seeing a bug where (as noted in
>> subject) reference-counted function results are not being initialized to
>> nil.
>
> They were never guaranteed to be initialized to nil.
>
> Reference-counted types (unlike other types) cannot contain memory
> garbage. But it doesn't mean that they are always initialized empty
> when the function starts. You need to explicitly do Result := '' if
> your code reads the Result later.
 >
 > See similar questions for Delphi:
 >
http://stackoverflow.com/questions/5336863/what-is-the-default-value-of-result-in-delphi
 >
http://stackoverflow.com/questions/5314918/do-i-need-to-setlength-a-dynamic-array-on-initialization/5315254#5315254

OUCH!!

I never dreamed that

local_var := some_function (param);

can result in the old value of local_var being used inside
some_function. That's what's happening here. I fully expected the result
dynamic array (embedded in a record type) to be initialized to nil, and
have never taken care to initialize dynamic arrays and ansistrings. I'm
astonished that this didn't bite me sooner, actually. Perhaps it is a
culprit in occasional crashes.

I can't see any mention of this in the documentation:
http://www.freepascal.org/docs-html/ref/refse47.html
http://www.freepascal.org/docs-html/ref/refse89.html
http://www.freepascal.org/docs-html/ref/refse91.html

> Luckily, FPC warns about it, at least in my simple test:

it also warns me in a simple test, but it doesn't give any warning in my
not-simple program.


Is there any compiler option that I can use to force nil initialization
of ref-counted function results?


Thanks,
David



_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Maciej Izak
In reply to this post by Michalis Kamburelis-3
2016-06-25 5:19 GMT+02:00 Michalis Kamburelis <[hidden email]>:
They were never guaranteed to be initialized to nil.

Result has special logic for string, dynamic array, method pointer and variant (well documented ;) ):

"For a string, dynamic array, method pointer, or variant result, the effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result."


--
Best regards,
Maciej Izak

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Sven Barth-2

Am 25.06.2016 09:23 schrieb "Maciej Izak" <[hidden email]>:
>
> 2016-06-25 5:19 GMT+02:00 Michalis Kamburelis <[hidden email]>:
>>
>> They were never guaranteed to be initialized to nil.
>
>
> Result has special logic for string, dynamic array, method pointer and variant (well documented ;) ):
>
> "For a string, dynamic array, method pointer, or variant result, the effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result."
>
> http://docwiki.embarcadero.com/RADStudio/Berlin/en/Program_Control#Handling_Function_Results

And them being var-parameters basically states that they retain whatever value they had.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Jürgen Hestermann
In reply to this post by Maciej Izak
Am 2016-06-25 um 09:23 schrieb Maciej Izak:
 > 2016-06-25 5:19 GMT+02:00 Michalis Kamburelis <[hidden email]>:
 >     They were never guaranteed to be initialized to nil.
 > Result has special logic for string, dynamic array, method pointer and variant (well documented ;) ):
 > "For a string, dynamic array, method pointer, or variant result, the effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result."
 > http://docwiki.embarcadero.com/RADStudio/Berlin/en/Program_Control#Handling_Function_Results

Does that mean that we now need to read the whole documenation of Free Pascal *and* also the whole documentation of Delphi?
And what version of Delphi?
Where is this documented within Free Pascal?

This has definitely changed with Free Pascal 3 as my programs suddenly did not work anymore and I had to add Setlength() all over my code.
Before we could rely on that managed types where always initialized (well, that's the purpose of managed types, no?).
It took me quite a while to find out why my programs failed as I did not expect such a (hidden) change.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Sven Barth-2

Am 25.06.2016 13:17 schrieb "Jürgen Hestermann" <[hidden email]>:
>
> Am 2016-06-25 um 09:23 schrieb Maciej Izak:
> > 2016-06-25 5:19 GMT+02:00 Michalis Kamburelis <[hidden email]>:
> >     They were never guaranteed to be initialized to nil.
> > Result has special logic for string, dynamic array, method pointer and variant (well documented ;) ):
> > "For a string, dynamic array, method pointer, or variant result, the effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result."
> > http://docwiki.embarcadero.com/RADStudio/Berlin/en/Program_Control#Handling_Function_Results
>
> Does that mean that we now need to read the whole documenation of Free Pascal *and* also the whole documentation of Delphi?
> And what version of Delphi?
> Where is this documented within Free Pascal?

No one said that FPC's documentation is perfect, we do our best, but that might still mean that there are missing pieces of information that we try to fix as soon as they are known to be missing.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Karoly Balogh (Charlie/SGR)
In reply to this post by Jürgen Hestermann
Hi,

On Sat, 25 Jun 2016, Jürgen Hestermann wrote:

> This has definitely changed with Free Pascal 3 as my programs suddenly
> did not work anymore and I had to add Setlength() all over my code.
> Before we could rely on that managed types where always initialized
> (well, that's the purpose of managed types, no?).

No. Local variables are not initialized by default. Managed type or not,
doesn't make a difference. Result is just another local variable in this
case. I think the compiler will even warn about using uninitialized local
variables, at least with -O3 and above (or when Data Flow Analysis is
enabled).

> It took me quite a while to find out why my programs failed as I did not
> expect such a (hidden) change.

It wasn't an intentional change for sure. However, the generated code
might have changed (hopefully it became more optimal) which simply exposed
some of these issues within the compiled code.

Actually, such issues are much easier to notice on other architectures,
aside from i386, because of the different calling convention and register
layout they use. I spent the better part of the last 3 years fixing code
which was originally intended for i386, to get it working better on ARM
(and others), where I ran into this exact "result remains uninitialized"
issue, along with many other "my code is good, fix your compiler" issues,
where it turned out the code was basically working accidentally, and it
was built on false assumptions...

Charlie
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Jürgen Hestermann
Am 2016-06-26 um 01:19 schrieb Karoly Balogh (Charlie/SGR):
 > No. Local variables are not initialized by default. Managed type or not,
 > doesn't make a difference.

This is not true.
Local variables of managed types are of course initialized!
If you declare

var S : String;

then it is of course initialized to nil.
Otherwise the (random) pointer stored in S would be used
to change the reference counter when you assign:

S := '';

The only exception (at least since FPC 3) is the function result
which is totally unexpected.
Why such an exception?

And it is not even documented (at least not within FPC)!
Which I find a bit strange because it is very important to know.
Omitting such information in the documentation is not good.
Here
http://www.freepascal.org/docs-html/ref/refsu15.html
it says:

"When declaring a variable of a dynamic array type, the initial length of the array is zero."

but no word about that this is not true when this array is a the result of a function.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

C Western-2
On 26/06/16 11:09, Jürgen Hestermann wrote:

> Am 2016-06-26 um 01:19 schrieb Karoly Balogh (Charlie/SGR):
>> No. Local variables are not initialized by default. Managed type or not,
>> doesn't make a difference.
>
> This is not true.
> Local variables of managed types are of course initialized!
> If you declare
>
> var S : String;
>
> then it is of course initialized to nil.
> Otherwise the (random) pointer stored in S would be used
> to change the reference counter when you assign:
>
> S := '';

My understanding (having been bitten by this) is that the requirement is
only that S has a "sensible" value, so that the heap is not corrupted on
the assignment, as you indicate. The compiler can re-use another
variable instead, so the unset value is some other valid string.

Colin
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Sven Barth-2
In reply to this post by Jürgen Hestermann

Am 26.06.2016 12:10 schrieb "Jürgen Hestermann" <[hidden email]>:
>
> Am 2016-06-26 um 01:19 schrieb Karoly Balogh (Charlie/SGR):
> > No. Local variables are not initialized by default. Managed type or not,
> > doesn't make a difference.
>
> This is not true.
> Local variables of managed types are of course initialized!
> If you declare
>
> var S : String;
>
> then it is of course initialized to nil.
> Otherwise the (random) pointer stored in S would be used
> to change the reference counter when you assign:
>
> S := '';
>
> The only exception (at least since FPC 3) is the function result
> which is totally unexpected.
> Why such an exception?
>
> And it is not even documented (at least not within FPC)!
> Which I find a bit strange because it is very important to know.
> Omitting such information in the documentation is not good.
> Here
> http://www.freepascal.org/docs-html/ref/refsu15.html
> it says:
>
> "When declaring a variable of a dynamic array type, the initial length of the array is zero."
>
> but no word about that this is not true when this array is a the result of a function.

But also no word that it is true either, cause you're not declaring a variable for the Result.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Karoly Balogh (Charlie/SGR)
In reply to this post by Jürgen Hestermann
Hi,

On Sun, 26 Jun 2016, Jürgen Hestermann wrote:

Ok, scrap what I wrote before... :|

You are right about the managed types initialization. But then the
documentation needs to be corrected there too. But actually, result is not
a local var, but in fact a "special" parameter of the function. My bad.
See below.

> The only exception (at least since FPC 3) is the function result
> which is totally unexpected.
> Why such an exception?

There's no exception. At least not specifically for Result. It is simply
variable passed by reference from the caller side, therefore it's not
initialized inside the function. And this is A., consistent with other var
parameters B., apparently also how Delphi does it. Actually, since managed
types are always passed by reference, this is not really a surprise, nor
an exception.

Actually, I tend to agree that it is very misleading. Also, you are also
right that this is new behavior in current compiler. At least 2.6.2 (I
don't have 2.6.4 at hand) decreased the reference to the result parameter
before handing it to the called function, which no longer seems to happen.
(I'm looking at the generated code.)

I have vague memories of a lengthy thread related to this, which changed
the refcounting behavior of reference passed managed types/strings, maybe
we're observing a sideeffect of this? (I think it was related to what
happens if the same reference counted variable is passed to multiple var
types and/or results, maybe also out parameters were involved. It's a
pretty grey area, which is not really documented anywhere...)

To show the problem, what happens with:

var
  a: ansistring;
a:=some_function(a);

If the caller code destroys the result variable (by decreasing the
reference count, because it's a result, so lets initialize it) it won't be
valid by the time you pass it as parameter "a', because only the function
itself will increase its reference then... Also, depending on the order of
parameter passing, you might get different results on different CPU
platforms... (Fix me?) Of course the function itself could also initialize
the result, but we're back to square one with that, if result and one of
the parameters are the same, weird things might happen.

I wish someone with real clue of calling conventions would comment. Jonas
maybe?

Charlie
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Martin Schreiber-2
In reply to this post by Jürgen Hestermann
On Sunday 26 June 2016 12:09:58 Jc3bcrgen Hestermann wrote:
>
> The only exception (at least since FPC 3) is the function result
> which is totally unexpected.
> Why such an exception?
>
I don't think one could treat a function result as a normal local variable.
The reason why not to initialise the result variable is because in many cases
it is not necessary and reduces performance.

Martin
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Jürgen Hestermann
In reply to this post by Sven Barth-2
Am 2016-06-26 um 14:13 schrieb Sven Barth:
 > But also no word that it is true either, cause you're not declaring a variable for the Result.

Of course I am declaring a variable:

function X : String;

Now the (local) variable "Result" is of type "String".
I use it in the same way as any other local variable.
And because it is a manged type I could rely on the documentation
which says that managed types are initialized (with zero length/nil).
"Result" is used in the same way as a (local) variable.
What should make me think that it is different?

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Jürgen Hestermann
In reply to this post by Karoly Balogh (Charlie/SGR)
Am 2016-06-26 um 14:27 schrieb Karoly Balogh (Charlie/SGR):
 > There's no exception. At least not specifically for Result. It is simply
 > variable passed by reference from the caller side, therefore it's not
 > initialized inside the function.

You mean when I have the following:

--------------------------
function X : ansistring;
begin
end;

var S : ansistring;

S := X;
--------------------------

Then S is a dangling pointer to nowhere?


 > And this is A., consistent with other var
 > parameters B., apparently also how Delphi does it. Actually, since managed
 > types are always passed by reference, this is not really a surprise, nor
 > an exception.

Of course it is a surprise because the documenation says that managed types are always initialized.
If I use a var parameter I am forced to use a variable (declared somewhere else) but
the function result can also be used in expressions without any (direct) assignment to a variable.
So it must behave like a locally declared variable IMO.

If I use an intermediate variable like:

--------------------------
function X : ansistring;
var X1 : ansistring;
begin
Result := X1+'A';
end;
--------------------------

then it is initialized while here

--------------------------
function X : ansistring;
begin
Result := Result+'A';
end;
--------------------------

I get garbage?
That's a very strange behaviour and no longer Pascal, it's C with all its funny side effects.

 > var
 >   a: ansistring;
 > a:=some_function(a);

When you have a parameter then of course it depends
on how the parameter is declared.

The Result variable within a function is different
and not a parameter. You cannot feed in any data here.
I would consider this more like an out parameter
but in case of managed types an initialization
must take place somewhere (IMO in the function like
for all other local variables too, where else?).

And it is definitly a change with FPC 3 because all my programs relied
on that managed types are initialized and suddenly they are not anymore.
I am wondering what other surprises lurk here and there which I have just not found out.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Dimitrios Chr. Ioannidis-2
In reply to this post by Jürgen Hestermann

Hi,

On 26/6/2016 7:20 μμ, Jürgen Hestermann wrote:
And because it is a manged type I could rely on the documentation
which says that managed types are initialized (with zero length/nil).
"Result" is used in the same way as a (local) variable.
What should make me think that it is different?

as Barry Kelly pointed out there http://stackoverflow.com/questions/861045/which-variables-are-initialized-when-in-delphi/861178#861178,
initialization differs on what mechanism was used to allocate memory . AFAIU, "initialized" <> "zero-filled" .

regards,

--
Dimitrios Chr. Ioannidis
www.nephelae.eu

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Karoly Balogh (Charlie/SGR)
In reply to this post by Jürgen Hestermann
Hi,

On Sun, 26 Jun 2016, Jürgen Hestermann wrote:

> Am 2016-06-26 um 14:27 schrieb Karoly Balogh (Charlie/SGR):
> > There's no exception. At least not specifically for Result. It is simply
> > variable passed by reference from the caller side, therefore it's not
> > initialized inside the function.
>
> You mean when I have the following:
>
> --------------------------
> function X : ansistring;
> begin
> end;
>
> var S : ansistring;
>
> S := X;
> --------------------------
>
> Then S is a dangling pointer to nowhere?
No. In this case, S was initialized (to nil), because it's a global var.
This just won't be changed by the X function call. If you had S:='test';
Before S:=X; then it would still contain 'test' after it.

Charlie
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

Sven Barth-2
In reply to this post by Jürgen Hestermann
On 26.06.2016 18:20, Jürgen Hestermann wrote:

> Am 2016-06-26 um 14:13 schrieb Sven Barth:
>> But also no word that it is true either, cause you're not declaring a
> variable for the Result.
>
> Of course I am declaring a variable:
>
> function X : String;
>
> Now the (local) variable "Result" is of type "String".
> I use it in the same way as any other local variable.
> And because it is a manged type I could rely on the documentation
> which says that managed types are initialized (with zero length/nil).
> "Result" is used in the same way as a (local) variable.
> What should make me think that it is different?

Variables are declared with "var", a function result is not declared
with "var".


Regards,
Sven

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: reference-counted function results not initialized to nil

David Emerson
In reply to this post by David Emerson
Philosophical arguments aside, I filed a bug against the documentation,
since one thing is clear-- that I and others were previously misled into
believing something that is not true.

http://mantis.freepascal.org/view.php?id=30321

In the report, I included several references where the documentation
could be improved to make the compiler's behavior more clear. If anyone
wants to expand on that list, please feel free :)

Also perhaps the wiki could use some work in this regard.

~David.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal