Memory leaks in FPC 2.3.1 SVN 13188

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory leaks in FPC 2.3.1 SVN 13188

bruce
When building an application with Lazarus (r20153) using FPC (r13188)
(on a Debian Lenny system) we are experiencing significant memory leaks.
Earlier revisions of both Lazarus and FPC 2.3.1 experience the same problem.

Building and running our app with heaptrc enabled we find a very large
number of blocks reporting memory allocated by fpc_dynarray_copy and
fpc_dynarray_setlength as remaining allocated at application exit even
though the variables to which the arrays are assigned either go out of
scope, are reused or are destroyed at application exit (or before).

When the same application is built with with Delphi6, Kylix3 and Turbo
Delphi there are no such memory leaks. With are building using FPC 2.3.1
with $MODE DELPHI.

Are we missing something here or might this be an FPC bug?

We'll file a bug report if necessary.

Many thanks for any advice!

Cheers, Bruce.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

Jonas Maebe-2

On 24 May 2009, at 09:29, Bruce Tulloch wrote:

> Are we missing something here or might this be an FPC bug?
>
> We'll file a bug report if necessary.

A bug report with a compilable example program that demonstrates the  
problem would be helpful, thanks.


Jonas

PS: your message was held for moderation because you are not  
subscribed with this address to this list (it is subscribed to fpc-
devel, though). If you're subscribed with another address, there's no  
problem. If you're not subscribed, it's best to mention this in mails  
you send, because otherwise people (other than moderators who saw that  
your message was held for moderation) won't add you in CC when  
replying. In any case, messages from this address to fpc-pascal will  
also be automatically accepted from now on.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

leledumbo
Administrator
> Are we missing something here or might this be an FPC bug?

It depends on how you use HeapTrc. If via -gh, this might a bug. If you manually include in a uses clause, you have to be sure that you put it as the first one of your program. Otherwise, HeapTrc won't be able to detect what have been done by units loaded before it.
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

bruce
Thanks for that. We're using -gh. We're aware of the usage requirements
of heaptrc.

The cause of this problem in our case is very subtle. So far we have
been unable to reproduce it outside our (very large) application but
we're working on isolating it in a smaller test program.

The problem looks like a dynamic array, created by Copy() inside an
object's method, which is returned as the result of the method and
assigned to a property (not a field) of another object (of a different
class), is never released.

This is the case even when the property to which it's assigned, is
assigned a new value later on (at which point by our accounting, the
reference count of the first array should have dropped to zero).

We'll post more details or an example program when we have abstracted
this problem from the rest of our application (in which application we
have already eliminated all other known memory leaks using heaptrc).

Are we correct to assume that as soon as no variable in our program, be
it global, static, object field or property, refers to a dynamic array,
the dynamic array will be released. Are we also correct to assume that
the reassignment of any variable referring to a dynamic array to a new
value will cause the previous array value to be released (presuming
nothing else refers to this array)?

Cheers, Bruce.

leledumbo wrote:
>> Are we missing something here or might this be an FPC bug?
>>    
>
> It depends on how you use HeapTrc. If via -gh, this might a bug. If you
> manually include in a uses clause, you have to be sure that you put it as
> the first one of your program. Otherwise, HeapTrc won't be able to detect
> what have been done by units loaded before it.
>  

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

Jonas Maebe-2

On 25 May 2009, at 07:38, Bruce Tulloch wrote:

> Are we correct to assume that as soon as no variable in our program,  
> be
> it global, static, object field or property, refers to a dynamic  
> array,
> the dynamic array will be released. Are we also correct to assume that
> the reassignment of any variable referring to a dynamic array to a new
> value will cause the previous array value to be released (presuming
> nothing else refers to this array)?

Yes.

One caveat is that the memory may not be immediately released after  
the last reference that you know of disappears, because there may  
still be temporary memory locations (implicitly created by the  
compiler while evaluating expressions) containing a reference. Such  
temporary locations will be finalised either when they are reused by  
the compiler later on, or when the function in which they were created  
by the compiler exits (such temporary locations can never survive the  
scope in which they were created by the compiler).

However, this could only cause the behaviour you are seeing if you  
were to call halt() somewhere in a deeply nested function and some  
temporary locations in a parent stack frame still contained  
references, or of you call halt() while some threads are still running  
that have local variables/temporaries that point to such variables.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

bruce
Ok, I don't think that caveat does not applies in this case, we neither
call halt nor share these particular variables between threads (we do
have threads in this application however). What we do see when freeing
(ie, assigning to nil) some of these variables is a jump from recount =
1 (as we expect at this point in our application) to refcount = 7 on
some blocks. There is no way (in our application) that there are 7
references to these blocks. We are investigating and report further as
soon as we have some more concrete data to feed back to you.

Bruce.

PS: does the attribute "compilerproc" mean the function is defined to be
called by the compiler implicitly? We have rebuilt the FPC RTL with
debug enabled to further diagnose what's happening here (which is why we
came across it).

Jonas Maebe wrote:

>
> On 25 May 2009, at 07:38, Bruce Tulloch wrote:
>
>> Are we correct to assume that as soon as no variable in our program, be
>> it global, static, object field or property, refers to a dynamic array,
>> the dynamic array will be released. Are we also correct to assume that
>> the reassignment of any variable referring to a dynamic array to a new
>> value will cause the previous array value to be released (presuming
>> nothing else refers to this array)?
>
> Yes.
>
> One caveat is that the memory may not be immediately released after
> the last reference that you know of disappears, because there may
> still be temporary memory locations (implicitly created by the
> compiler while evaluating expressions) containing a reference. Such
> temporary locations will be finalised either when they are reused by
> the compiler later on, or when the function in which they were created
> by the compiler exits (such temporary locations can never survive the
> scope in which they were created by the compiler).
>
> However, this could only cause the behaviour you are seeing if you
> were to call halt() somewhere in a deeply nested function and some
> temporary locations in a parent stack frame still contained
> references, or of you call halt() while some threads are still running
> that have local variables/temporaries that point to such variables.
>
>
> Jonas
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

Jonas Maebe-2

On 25 May 2009, at 10:30, Bruce Tulloch wrote:

> PS: does the attribute "compilerproc" mean the function is defined  
> to be
> called by the compiler implicitly? We have rebuilt the FPC RTL with
> debug enabled to further diagnose what's happening here (which is  
> why we
> came across it).

It means that the procedure becomes invisible to regular Pascal code,  
and that it gets a particular internal symbol name. Such routines are  
indeed called implicitly by the compiler for various purposes.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

bruce
I think I've nailed and example of the problem; when Copy() is called
with the result of another Copy() the reference count for the returned
array is not managed correctly when compiled with -MDelphi.

A trivial example program, copytest.pas:

    program copytest;

    var
    S, D : array of Integer;
 
    begin
      SetLength(S,4000);
      D := Copy(Copy(S));
    end.

which when compiled with:

    fpc -MDelphi -gl -gh copytest.pas

and run produces the following:

    lenin:$ ./copytest
    Heap dump by heaptrc unit
    3 memory blocks allocated : 48024/48024
    2 memory blocks freed     : 32016/32016
    1 unfreed memory blocks : 16008
    True heap size : 262144
    True free heap : 246048
    Should be : 246072
    Call trace for block $B7F4BF50 size 16008
       $080480EA  main,  line 8 of copytest.pas
       $080657A7

I hope this helps diagnose what's happening here.

Note that if compiled without -MDelphi it works correctly but in our
case we need to use -MDelphi because our application is also built
with Delphi and Kylix3.

Cheers, Bruce.

Jonas Maebe wrote:

>
> On 25 May 2009, at 10:30, Bruce Tulloch wrote:
>
>> PS: does the attribute "compilerproc" mean the function is defined to be
>> called by the compiler implicitly? We have rebuilt the FPC RTL with
>> debug enabled to further diagnose what's happening here (which is why we
>> came across it).
>
> It means that the procedure becomes invisible to regular Pascal code,
> and that it gets a particular internal symbol name. Such routines are
> indeed called implicitly by the compiler for various purposes.
>
>
> Jonas
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

bruce
I've reported this as an FPC bug:

  http://bugs.freepascal.org/view.php?id=13820

Cheers, Bruce.

Bruce Tulloch wrote:

> I think I've nailed and example of the problem; when Copy() is called
> with the result of another Copy() the reference count for the returned
> array is not managed correctly when compiled with -MDelphi.
>
> A trivial example program, copytest.pas:
>
>     program copytest;
>
>     var
>     S, D : array of Integer;
>  
>     begin
>       SetLength(S,4000);
>       D := Copy(Copy(S));
>     end.
>
> which when compiled with:
>
>     fpc -MDelphi -gl -gh copytest.pas
>
> and run produces the following:
>
>     lenin:$ ./copytest
>     Heap dump by heaptrc unit
>     3 memory blocks allocated : 48024/48024
>     2 memory blocks freed     : 32016/32016
>     1 unfreed memory blocks : 16008
>     True heap size : 262144
>     True free heap : 246048
>     Should be : 246072
>     Call trace for block $B7F4BF50 size 16008
>        $080480EA  main,  line 8 of copytest.pas
>        $080657A7
>
> I hope this helps diagnose what's happening here.
>
> Note that if compiled without -MDelphi it works correctly but in our
> case we need to use -MDelphi because our application is also built
> with Delphi and Kylix3.
>
> Cheers, Bruce.
>
> Jonas Maebe wrote:
>  
>> On 25 May 2009, at 10:30, Bruce Tulloch wrote:
>>
>>    
>>> PS: does the attribute "compilerproc" mean the function is defined to be
>>> called by the compiler implicitly? We have rebuilt the FPC RTL with
>>> debug enabled to further diagnose what's happening here (which is why we
>>> came across it).
>>>      
>> It means that the procedure becomes invisible to regular Pascal code,
>> and that it gets a particular internal symbol name. Such routines are
>> indeed called implicitly by the compiler for various purposes.
>>
>>
>> Jonas
>> _______________________________________________
>> fpc-pascal maillist  -  [hidden email]
>> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>>    
>
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>  

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

bruce
Confirmed fixed in r13195 as tested with the application in which this
was originally discovered, thanks Jonas.

Bruce Tulloch wrote:
> I've reported this as an FPC bug:
>
>   http://bugs.freepascal.org/view.php?id=13820
>
> Cheers, Bruce.
>  
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

Joost van der Sluis
Op dinsdag 26-05-2009 om 10:16 uur [tijdzone +1000], schreef Bruce
Tulloch:
> Confirmed fixed in r13195 as tested with the application in which this
> was originally discovered, thanks Jonas.

In that case, can you close the bug report? So we know that we don't
have to look at this issue anymore?

Joost.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in FPC 2.3.1 SVN 13188

bruce
Hmm, I did. Apparently mantis did not register this? Done it again.
Closed now. -b

Joost van der Sluis wrote:

> Op dinsdag 26-05-2009 om 10:16 uur [tijdzone +1000], schreef Bruce
> Tulloch:
>  
>> Confirmed fixed in r13195 as tested with the application in which this
>> was originally discovered, thanks Jonas.
>>    
>
> In that case, can you close the bug report? So we know that we don't
> have to look at this issue anymore?
>
> Joost.
>
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>  

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal