Question on how to avoid memory trouble

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Question on how to avoid memory trouble

Jason P Sage
>The error is not from a lack of stack space, either.  The error is
>EOutOfMemory that indicates a less easily solved problem.

>Can anyone recommend a method to search a whole drive, of arbitrary
>size, without running out of memory.

>Stu Cox

This is exactly the kind of thing my string class works great with.

It's a dynamic double linked list - and the way its set up - there are three
versions - the simple tracks just pointers, the next has name, value and
Description ansistrings and I make pretty huge lists with it, and it has
Append, insert, findfirst, findnext, etc. I have an unofficial release on
www.jegas.org and you need to use the compile switch that allows +=
construct.

Fpc -Sc  yourprogram.pas  

Worth a shot.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

RE: Question on how to avoid memory trouble

Cox, Stuart TRAN:EX
Shucks everybody...

I think that its great that the hornet's nest that I stirred up might
mean that a new unit of container classes gets written.  I didn't think
that my seemingly simple problem would create such a cascade of
interest.  Lists are catalysts for invention!

My trouble remains that I seem unable to devise or use canned routines
to get the names of all the files on my drive.  Just writing them out to
con: in a console application would be acceptable to me to get me
started.  TurboPower's technique in their SysTools seemed simple enough
but I seem to have created a monster.

Does somebody have something in source that they'd share with this poor
guy, please?

Stu Cox

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jason P
Sage
Sent: Sat, February 3, 2007 5:13 PM
To: [hidden email]
Subject: [fpc-pascal] Question on how to avoid memory trouble

>The error is not from a lack of stack space, either.  The error is
>EOutOfMemory that indicates a less easily solved problem.

>Can anyone recommend a method to search a whole drive, of arbitrary
>size, without running out of memory.

>Stu Cox

This is exactly the kind of thing my string class works great with.

It's a dynamic double linked list - and the way its set up - there are
three versions - the simple tracks just pointers, the next has name,
value and Description ansistrings and I make pretty huge lists with it,
and it has Append, insert, findfirst, findnext, etc. I have an
unofficial release on www.jegas.org and you need to use the compile
switch that allows += construct.

Fpc -Sc  yourprogram.pas  

Worth a shot.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Wayne Sherman-4
> I think that its great that the hornet's nest that I stirred up might
> mean that a new unit of container classes gets written...
>
> ...My trouble remains...

Getting a better containers unit is nice, but the root of the problem
seems to be the memory manager.  Delphi's old memory manager had
fragmentation issues also.  That is why NexusMM, BucketMM and FastMM we
created.  FastMM is now included with Delphi as the main memory manager.
  Not only does it solve memory fragmentation issues, it is much faster
than the old memory manager (especially in multithreaded apps).

FastMM is open source, has anyone ported it to FPC?

I seem to recall that when it began to be used in Delphi, it uncovered
lots of memory bugs in the IDE and runtime.  When these were fixed,
Delphi became a much more stable product.

Wayne
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Micha Nelissen
Wayne Sherman wrote:
> Getting a better containers unit is nice, but the root of the problem
> seems to be the memory manager.  Delphi's old memory manager had

No.

> fragmentation issues also.  That is why NexusMM, BucketMM and FastMM we

The memory manager cannot fix fragmentation in TStringList. Delphi's
memory manager fragmented memory with (repeated) different memory size
usages.

Micha
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

RE: Question on how to avoid memory trouble

Jeff Pohlmeyer
In reply to this post by Jason P Sage
> Does somebody have something in source that they'd share
> with this poor guy, please?

Maybe you can find something here?
  http://torry.net/pages.php?id=256


 - Jeff
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Peter Vreman
In reply to this post by Wayne Sherman-4
At 23:39 5-2-2007, you wrote:

>>I think that its great that the hornet's nest that I stirred up might
>>mean that a new unit of container classes gets written...
>>...My trouble remains...
>
>Getting a better containers unit is nice, but the root of the
>problem seems to be the memory manager.  Delphi's old memory manager
>had fragmentation issues also.  That is why NexusMM, BucketMM and
>FastMM we created.  FastMM is now included with Delphi as the main
>memory manager.  Not only does it solve memory fragmentation issues,
>it is much faster than the old memory manager (especially in
>multithreaded apps).
>
>FastMM is open source, has anyone ported it to FPC?

There are users that have got FastMM working with FPC. But it is not
needed. The standard FPC heap manager is for single-threaded
applications as fast as FastMM.


Peter

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

RE: Question on how to avoid memory trouble

Helmut Hartl
In reply to this post by Jason P Sage


 > >FastMM is open source, has anyone ported it to FPC?
 >
 > There are users that have got FastMM working with FPC. But
 > it is not needed. The standard FPC heap manager is for
 > single-threaded applications as fast as FastMM.

A little bit shortsighted (IMHO),as "everyone seriously concerning
performance"
is concentrating on multithreaded, multiprocessor applications, wich are
lock and waitfree, right now.

Is there anyone around who is using FastMM in FPC?
A guide what is necessary to get FastMM working with FPC
is much appreciated.

Thanks, helmut
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Florian Klämpfl
Helmut Hartl schrieb:
>
>  > >FastMM is open source, has anyone ported it to FPC?
>  >
>  > There are users that have got FastMM working with FPC. But
>  > it is not needed. The standard FPC heap manager is for
>  > single-threaded applications as fast as FastMM.
>
> A little bit shortsighted (IMHO),as "everyone seriously concerning
> performance"

Well, depending on FastMM is short sight too because FastMM is using
heavily assembler, so it is i386 only.

I think a better approach is to extend the current fpc memory manager to
perform better in multithreaded environments.

> is concentrating on multithreaded, multiprocessor applications, wich are
> lock and waitfree, right now.

Well, the lockless multithreaded memory managers are a myth. Atomic
exchange operations take a also a lot of time and are locked by the cpu
as well.

>
> Is there anyone around who is using FastMM in FPC?
> A guide what is necessary to get FastMM working with FPC
> is much appreciated.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

George Lober
Florian Klaempfl wrote:

> Helmut Hartl schrieb:
>  
>>  > >FastMM is open source, has anyone ported it to FPC?
>>  >
>>  > There are users that have got FastMM working with FPC. But
>>  > it is not needed. The standard FPC heap manager is for
>>  > single-threaded applications as fast as FastMM.
>>
>> A little bit shortsighted (IMHO),as "everyone seriously concerning
>> performance"
>>    
>
> Well, depending on FastMM is short sight too because FastMM is using
> heavily assembler, so it is i386 only.
>
> I think a better approach is to extend the current fpc memory manager to
> perform better in multithreaded environments.
>
>  
>> is concentrating on multithreaded, multiprocessor applications, wich are
>> lock and waitfree, right now.
>>    
>
> Well, the lockless multithreaded memory managers are a myth. Atomic
> exchange operations take a also a lot of time and are locked by the cpu
> as well.
>
>  
>> Is there anyone around who is using FastMM in FPC?
>> A guide what is necessary to get FastMM working with FPC
>> is much appreciated.
>>    
>
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>
>  

One nice side benefit of FastMM is that it automatically lets you know
of memory leaks with no fuss and bother on the programmers part.

--
George

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Jonas Maebe-2

On 12 Feb 2007, at 20:27, George Lober wrote:

> One nice side benefit of FastMM is that it automatically lets you  
> know of memory leaks with no fuss and bother on the programmers part.

FPC has support for that built in. Just compile your application with  
-ghl (that also does a lot of other memory sanity checking)


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

RE: Question on how to avoid memory trouble

Helmut Hartl
In reply to this post by Jason P Sage
 >Florian Klaempfl schrieb:
 > Well, the lockless multithreaded memory managers are a myth.
 > Atomic exchange operations take a also a lot of time and are
 > locked by the cpu as well.
 >

The XEN Developers at Cambridge have nice papers to read ...

http://www.cl.cam.ac.uk/~kaf24/

(Others are researching this topic too, General Infos at: ACM / IBM /
Linux Kernel)

http://www.research.ibm.com/people/m/michael/pldi-2004.pdf

I agree that as always in life it depends on the situation on
where to use lock free techniques, and where not.
Memory Management is a field which can take advantages of
such techniques in MT Apps.

helmut




_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Vincent Snijders
Helmut Hartl schreef:

> http://www.research.ibm.com/people/m/michael/pldi-2004.pdf
>

This one is not lock free, because it uses atomic instructions used by
the cpu, which are essentially fine grained locks.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

George Lober
In reply to this post by Jonas Maebe-2
Jonas Maebe wrote:
>
> On 12 Feb 2007, at 20:27, George Lober wrote:
>
>> One nice side benefit of FastMM is that it automatically lets you
>> know of memory leaks with no fuss and bother on the programmers part.
>
> FPC has support for that built in. Just compile your application with
> -ghl (that also does a lot of other memory sanity checking)
>

Good to know, thanks for the tip

--
George

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Florian Klämpfl
In reply to this post by Vincent Snijders
Vincent Snijders schrieb:
> Helmut Hartl schreef:
>
>> http://www.research.ibm.com/people/m/michael/pldi-2004.pdf
>>
>
> This one is not lock free, because it uses atomic instructions used by
> the cpu, which are essentially fine grained locks.

Exactly, and cmpxchg etc. are really expensive too. Since the original
FPC heap uses already several pools, it should be easily possible to
make it scale better in multi threaded applications.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

RE: Question on how to avoid memory trouble

Helmut Hartl
In reply to this post by Jason P Sage
 > >
 > > This one is not lock free, because it uses atomic
 > instructions used by
 > > the cpu, which are essentially fine grained locks.
 >
 > Exactly, and cmpxchg etc. are really expensive too. Since
 > the original FPC heap uses already several pools, it should
 > be easily possible to make it scale better in multi threaded
 > applications.

Fine that sounds really nice - Do you already have a idea what
is not optimal and can be enhanced ?
Maybe I will find the time to give it a try

helmut
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Question on how to avoid memory trouble

Jonas Maebe-2

On 13 feb 2007, at 13:12, Helmut Hartl wrote:

>> Exactly, and cmpxchg etc. are really expensive too. Since
>> the original FPC heap uses already several pools, it should
>> be easily possible to make it scale better in multi threaded
>> applications.
>
> Fine that sounds really nice - Do you already have a idea what
> is not optimal and can be enhanced ?

The default memory manager is always entirely locked, while  
differently sized blocks are allocated from different pools. So it  
should be possible to lock only one pool per allocation.  
Additionally, getting a block from a pool may even be possibly using  
a few atomic operations rather than by acquiring a lock.

See rtl/inc/heap.inc


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal