Heavy heap fragmentation issue

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Heavy heap fragmentation issue

Martok
Hi all,

I'm having a problem here in a sequential image processing application that
seems to test a particularly bad operation mode for the RTL heap manager (on
Windows, but I don't think this matters here).
The work load looks like this: load "normal sized" image, do some processing,
calculate a few values, create thumbnail in memory, next. By "normal sized" I
mean something like 35MB uncompressed (regular DSLR resolution) and smaller.
It's threaded code, but I'm describing the single worker operation here.

This appears to trigger insane memory fragmentation: after execution on 40 test
files, I'm left with 250MB working set, while GetHeapStatus only reports 600k
used (which seems correct, my data structures add up to 500k). This is never
released to the OS. In fact, I've had to set the LAA flag just so that the
program can work on the real data at all, with some 2.6GB working set for 1.07MB
of used memory.

Is there any tuning option that could use maybe fix that? Some growing or
reallocating flag or something?


As a quick test, I've tried my partial port of FastMM4 which works just fine, no
fragmentation and peaks at 40MB used memory, which fits with the largest image.

But this is such a reproducible test case, maybe there is something that can be
done to improve the RTL MM?

--
Regards,
Martok


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Ryan Joseph


> On Jun 2, 2019, at 12:49 PM, Martok <[hidden email]> wrote:
>
> Hi all,
>
> I'm having a problem here in a sequential image processing application that
> seems to test a particularly bad operation mode for the RTL heap manager (on
> Windows, but I don't think this matters here).
> The work load looks like this: load "normal sized" image, do some processing,
> calculate a few values, create thumbnail in memory, next. By "normal sized" I
> mean something like 35MB uncompressed (regular DSLR resolution) and smaller.
> It's threaded code, but I'm describing the single worker operation here.
>
> This appears to trigger insane memory fragmentation: after execution on 40 test
> files, I'm left with 250MB working set, while GetHeapStatus only reports 600k
> used (which seems correct, my data structures add up to 500k). This is never
> released to the OS. In fact, I've had to set the LAA flag just so that the
> program can work on the real data at all, with some 2.6GB working set for 1.07MB
> of used memory.
>
> Is there any tuning option that could use maybe fix that? Some growing or
> reallocating flag or something?
>
>
> As a quick test, I've tried my partial port of FastMM4 which works just fine, no
> fragmentation and peaks at 40MB used memory, which fits with the largest image.
>
> But this is such a reproducible test case, maybe there is something that can be
> done to improve the RTL MM?
>
> --
> Regards,
> Martok

can you break it down into a test program and post?

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Martok
> can you break it down into a test program and post?

I should probably put the project on Github anyway, it's rather small, and it
shouldn't have any dependencies. Give me an hour to clean it up a bit.

You'll have to supply your own images though (just copy one 10 times or
something), as I don't own the copyright to anything in my test dataset.



In the mean time, here's an overlay of Process Explorer for two test runs:
http://puu.sh/DAHMA/d83bfe7215.png
Red/Left axis is FastMM, Green/Right axis is the RTL MM. So the high memory use
occurs at the same time, only the RTLMM rarely returns memory so it builds up.


--
Regards,
Martok


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Martok
>> can you break it down into a test program and post?
>
> I should probably put the project on Github anyway, it's rather small, and it
> shouldn't have any dependencies. Give me an hour to clean it up a bit.
This should be good enough. It's been only used by like four people so the UI is
a bit of a mess, but you'll only need one button to see the problem.

So, here are the steps to reproduce:
- Get the code from
  <https://github.com/martok/ImageHash/tree/bug-fpcmm-fragment>
- Compile (originally written with FPC trunk >= late 2017, best use
  current trunk; LCL version shouldn't matter much)
- The default mode puts the exe in bin_i386, right next to the DLLs
- Run
- Provide data (~10 3MB JPEGs should do, or just paste one 10 times), either:
  - paste the full path to a directory containing a number of images
    in the top-center memo (no, there's no SelectDirectory dialog)
  - put the images in a folder 'data', that gets auto-added in this version
- Click 'Read'
- Wait for it to find visually similar images in your input
- Compare memory as reported by task manager or ProcExp to HeapStatus
  that is printed in the bottom-left log memo.

Note that the number of threads used (defaults to HostCPUs-1) doesn't matter
(can be changed down to 1), so it's not a concurrent alloc problem. You can also
apply the attached patch to even run the comparator in serial, but that doesn't
change anything either.

--
Regards,
Martok


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

test_serial_compare.patch (1002 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Ryan Joseph


> On Jun 2, 2019, at 2:49 PM, Martok <[hidden email]> wrote:
>
> This should be good enough. It's been only used by like four people so the UI is
> a bit of a mess, but you'll only need one button to see the problem.

Sorry this is just too much for me to look at without spending considerable time (and I’m on Mac also). Can’t you reduce it down to a single test program?

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Marco van de Voort-3
In reply to this post by Martok

Op 6/2/2019 om 6:49 PM schreef Martok:
> I'm having a problem here in a sequential image processing application that
> seems to test a particularly bad operation mode for the RTL heap manager (on
> Windows, but I don't think this matters here).
> The work load looks like this: load "normal sized" image, do some processing,
> calculate a few values, create thumbnail in memory, next. By "normal sized" I
> mean something like 35MB uncompressed (regular DSLR resolution) and smaller.
> It's threaded code, but I'm describing the single worker operation here.

Note that it is fairly typical that also frustrated pre fastmm D7. Back
then I wrote a simple pool-factory combo class for it, and with D2009 I
upgraded it to generics,so that I can easily create pools for many such
objects.

A possible issue might be that I use a simple tthreadlist for its
storage, but since D2009 all those classes level primitives work with
tmonitor spinlocks before them. That might be slower on FPC for high counts.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Martok
Am 03.06.2019 um 14:49 schrieb Marco van de Voort:
> Note that it is fairly typical that also frustrated pre fastmm D7. Back
> then I wrote a simple pool-factory combo class for it, and with D2009 I
> upgraded it to generics,so that I can easily create pools for many such
> objects.
Well, there is a general-purpose pooled allocator already. It's called the heap
manager ;-)

Jokes aside, I did a bunch of counting. The only thing that sticks around are
50kB TLazIntfImage instances. Everything else gets freed right after processing.
Just from the order of alloc/frees, there shouldn't even be gaps, so I would
expect the next iteration to just find those allocations to be empty and reuse
them. But for some reason, that doesn't happen.

I've compiled with FastMM for now, but it turns out this is in fact a bit slower
than the RTL MM, and I don't really trust my port yet...


--
Regards,
Martok


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Burkhard Carstens-3
Am Dienstag, 4. Juni 2019 18:00 schrieb Martok:

> Am 03.06.2019 um 14:49 schrieb Marco van de Voort:
> > Note that it is fairly typical that also frustrated pre fastmm D7.
> > Back then I wrote a simple pool-factory combo class for it, and
> > with D2009 I upgraded it to generics,so that I can easily create
> > pools for many such objects.
>
> Well, there is a general-purpose pooled allocator already. It's
> called the heap manager ;-)
>
> Jokes aside, I did a bunch of counting. The only thing that sticks
> around are 50kB TLazIntfImage instances. Everything else gets freed
> right after processing. Just from the order of alloc/frees, there
> shouldn't even be gaps, so I would expect the next iteration to just
> find those allocations to be empty and reuse them. But for some
> reason, that doesn't happen.
>
> I've compiled with FastMM for now, but it turns out this is in fact a
> bit slower than the RTL MM, and I don't really trust my port yet...

Try compiling the heap manager with "-dBESTMATCH". This makes it a bit
slower but greatly reduces fragmentation.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Martok
Am 13.07.2019 um 08:36 schrieb Burkhard Carstens:
> Try compiling the heap manager with "-dBESTMATCH". This makes it a bit
> slower but greatly reduces fragmentation.

Thanks, I'll give that a try!

Just to be clear, that needs to be set when compiling the RTL, right?


--
Regards,
Martok

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Burkhard Carstens-3
Am Montag, 15. Juli 2019 17:27 schrieb Martok:
> Am 13.07.2019 um 08:36 schrieb Burkhard Carstens:
> > Try compiling the heap manager with "-dBESTMATCH". This makes it a
> > bit slower but greatly reduces fragmentation.
>
> Thanks, I'll give that a try!
>
> Just to be clear, that needs to be set when compiling the RTL, right?
 
I think so. I usually set it when I "make all OPT=-dBESTMATCH" the compiler.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Heavy heap fragmentation issue

Martok
>>> Try compiling the heap manager with "-dBESTMATCH". This makes it a
>>> bit slower but greatly reduces fragmentation.
> I think so. I usually set it when I "make all OPT=-dBESTMATCH" the compiler.
Thanks.

BESTMATCH is only marginally better for the test case, but is much slower (i.e.
fully bootstrapping takes 50% longer). I guess there is something fundamentally
not very well suited to this type of workload in the RTL MM.

But it was worth a shot...


Best,

Sebastian

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

image.png (5K) Download Attachment