FPC Heap Management : sub-allocation ?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

FPC Heap Management : sub-allocation ?

Brian
Does FPC implement sub-allocation  , in which the user program allocates a block of memory from the heap , and only "plays" in that sub-allocated block , such that if the user program has a serious memory leak , the user program may crash but it cannot exhaust the OS memory and cause the OS to crash?

... sorry for the long winded sentence.

Thanks in advance
Brian
Reply | Threaded
Open this post in threaded view
|

Re: FPC Heap Management : sub-allocation ?

Sven Barth-2

Am 31.10.2014 15:21 schrieb "Brian" <[hidden email]>:
>
> Does FPC implement sub-allocation  , in which the user program allocates a
> block of memory from the heap , and only "plays" in that sub-allocated block
> , such that if the user program has a serious memory leak , the user program
> may crash but it cannot exhaust the OS memory and cause the OS to crash?
>
> ... sorry for the long winded sentence.

FPC's default heap manager allocates until the OS reports that the process' address space is exhausted which for 32-bit processes on Windows without LargeAddressAware is at around 2 GB. After that you'll receive EOutOfMemory exceptions.

You can however implement a heap manager that works differently.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FPC Heap Management : sub-allocation ?

Brian
Thanks Sven.

Do you know how it behaves when running under Linux?
Reply | Threaded
Open this post in threaded view
|

Re: FPC Heap Management : sub-allocation ?

Michael Van Canneyt


On Fri, 31 Oct 2014, Brian wrote:

> Thanks Sven.
>
> Do you know how it behaves when running under Linux?

What do you mean with this question ?

Large memory blocks are allocated directly from the OS.
For small memory blocks, pools are allocated from the OS,
and the small memory blocks are then taken from the pool.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FPC Heap Management : sub-allocation ?

Frederic Da Vitoria
2014-10-31 16:33 GMT+01:00 Michael Van Canneyt <[hidden email]>:


On Fri, 31 Oct 2014, Brian wrote:

Thanks Sven.

Do you know how it behaves when running under Linux?

What do you mean with this question ?

Large memory blocks are allocated directly from the OS.
For small memory blocks, pools are allocated from the OS, and the small memory blocks are then taken from the pool.

I think that Brian is asking if a memory leak could eat the system's memory irreversibly. IIUC in Windows, when the program is stopped, all it's memory is freed, even if the program leaked memory. I don't know about Linux, but I'd be surprised if it weren't the same.

This does not mean the OS will not be almost as bad as crashed: if your program leaks too much, you may trigger so much thrashing that your system will become unusable. This did happen quite frequently to me with Windows XP 32 bits. I now use Windows 7 64 bits, and it hasn't happened to me since, but I guess that's only because I haven't tried hard enough :-)

--
Frederic Da Vitoria
(davitof)

Membre de l'April - « promouvoir et défendre le logiciel libre » - http://www.april.org

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FPC Heap Management : sub-allocation ?

Brian
Frederic Da Vitoria wrote
I think that Brian is asking if a memory leak could eat the system's memory
irreversibly. IIUC in Windows, when the program is stopped, all it's memory
is freed, even if the program leaked memory. I don't know about Linux, but
I'd be surprised if it weren't the same.

--
Frederic Da Vitoria
(davitof)

Membre de l'April - « promouvoir et défendre le logiciel libre » -
http://www.april.org

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Yes that is exactly what I was asking. For a high reliable system running 24/7 my fear using Object Oriented code vs procedural code is exactly what you mentioned . If there is a serious leak , or even a small one for an extended period of time , it can take down the OS. I have seen this happen with Sun OS and a 3rd party driver that leaked and caused the OS to crash after an extended period of time.
Reply | Threaded
Open this post in threaded view
|

Re: FPC Heap Management : sub-allocation ?

Michael Van Canneyt


On Fri, 31 Oct 2014, Brian wrote:

> Frederic Da Vitoria wrote
>> I think that Brian is asking if a memory leak could eat the system's
>> memory
>> irreversibly. IIUC in Windows, when the program is stopped, all it's
>> memory
>> is freed, even if the program leaked memory. I don't know about Linux, but
>> I'd be surprised if it weren't the same.
>>
>> --
>> Frederic Da Vitoria
>> (davitof)
>>
>> Membre de l'April - « promouvoir et défendre le logiciel libre » -
>> http://www.april.org
>>
>> _______________________________________________
>> fpc-pascal maillist  -
>
>> fpc-pascal@.freepascal
>
>> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>
>
> Yes that is exactly what I was asking. For a high reliable system running
> 24/7 my fear using Object Oriented code vs procedural code is exactly what
> you mentioned . If there is a serious leak , or even a small one for an
> extended period of time , it can take down the OS. I have seen this happen
> with Sun OS and a 3rd party driver that leaked and caused the OS to crash
> after an extended period of time.
First, Linux will simply kill off processes if memory is exhausted, to ensure system stability.

Second: a driver is something special. Most drivers run in kernel (OS) space.
If the kernel itself leaks memory, obviously it can not kill itself...

Third: on unix, you can tell the system how much memory a program is
allowed to consume. If the program tries to consume more, it will simply
not receive it. See the getrlimit/setrlimit calls.

So for a user-space program there really is nothing to worry about.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FPC Heap Management : sub-allocation ?

Mark Morgan Lloyd-5
In reply to this post by Brian
Brian wrote:
> Frederic Da Vitoria wrote> I think that Brian is asking if a memory leak could eat the system's> memory> irreversibly. IIUC in Windows, when the program is stopped, all it's> memory> is freed, even if the program leaked memory. I don't know about Linux, but> I'd be surprised if it weren't the same.> > -- > Frederic Da Vitoria> (davitof)> > Membre de l'April - « promouvoir et défendre le logiciel libre » -> http://www.april.org> > _______________________________________________> fpc-pascal maillist  -  
>> fpc-pascal@.freepascal
>> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>
> Yes that is exactly what I was asking. For a high reliable system running24/7 my fear using Object Oriented code vs procedural code is exactly whatyou mentioned . If there is a serious leak , or even a small one for anextended period of time , it can take down the OS. I have seen this happenwith Sun OS and a 3rd party driver that leaked and caused the OS to crashafter an extended period of time.

An operating system certainly /shouldn't/ crash if user processes
allocate excessive memory. The overall system might thrash which would
make it unresponsive to e.g. new SSH sessions, but that's hardly the
same thing.

I've got stuff here running 24/7 with uptime measured in months.
Historically, the memory leaks weren't in application code but were in
database backend stuff that managed its own heap, which made it very
difficult to monitor.

It's easy enough to code such that when you perform a complex activity
which /should/ free all memory on completion you check that it does so.
It's also easy enough to put something in your UI that reports how much
is currently on the heap, and if you see that it's growing you can
investigate in a development environment using e.g. HeapTrace.

In extremis, it's usually possible to get a program to restart itself if
something's obviously wrong. I don't know if it's still the case, but
Apache used to limit the number of requests each process would handle to
protect it from memory loss due to faulty CGI etc.

If necessary, you can use a custom heap manager e.g. if you don't want
it to expand beyond some preallocated size. In principle, you could code
yourself multiple heaps each with custom procedures/functions, but while
these might be useful for stuff that you're writing yourself you'll not
be able to use them with the standard OO classes.

I don't believe that there's an equivalent to mark/release in the
standard heap managers, and from discussion elsewhere I don't believe it
could be added in a compatible way. Some variant of Mark() that took a
parameter suggesting how much memory could be allocated could be useful,
but your focus should be on preventing leaks and runaway rather than
constantly recovering from them.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal