Caching a big file on memory

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Caching a big file on memory

Eduardo Morras
Hello:

        I have a little problem, i participe in the 'Large Compression
Benchmark' (http://cs.fit.edu/~mmahoney/compression/text.html) and
need to cache the whole enwik8 file (100MB) on memory but when i try
to  allocate space for it i get a compiler error, saying that maximum
is 64MB (67108864). I tried to use $MEMORY 200000000,500000000 (200MB
stack and 500MB heap) and the compiler switches, but no success.

        I need all the file on RAM beacause i want to do a BWT, which
implies a sort of the 100MB and uses 400 more MB, and doing so from
hard disk is very slow. The contest has a time limit of 10 hours on a
1Ghz machine, but i need all speed at BWT/sort phase, so i can spend
the time on getting better compression.

        Must i touch the system, rtl etc... files and recompile compiler? Or
is there another, and simpler, way?

P.D. Of course, my pc has more than 100 MB (1.25 GB)

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Caching a big file on memory

Jonas Maebe-2

On 11 Sep 2006, at 18:09, Eduardo wrote:

> I have a little problem, i participe in the 'Large Compression  
> Benchmark' (http://cs.fit.edu/~mmahoney/compression/text.html) and  
> need to cache the whole enwik8 file (100MB) on memory but when i  
> try to  allocate space for it i get a compiler error, saying that  
> maximum is 64MB (67108864). I tried to use $MEMORY  
> 200000000,500000000 (200MB stack and 500MB heap) and the compiler  
> switches, but no success.

How are you trying to allocate the memory, and for which target  
operating system are you compiling?


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Caching a big file on memory

Eduardo Morras
At 18:30 11/09/2006, Jonas wrote:

>On 11 Sep 2006, at 18:09, Eduardo wrote:
>
>>         I have a little problem, i participe in the 'Large Compression
>>Benchmark' (http://cs.fit.edu/~mmahoney/compression/text.html) and
>>need to cache the whole enwik8 file (100MB) on memory but when i
>>try to  allocate space for it i get a compiler error, saying that
>>maximum is 64MB (67108864). I tried to use $MEMORY
>>200000000,500000000 (200MB stack and 500MB heap) and the compiler
>>switches, but no success.
>
>How are you trying to allocate the memory, and for which target
>operating system are you compiling?

Oppsss, sorry, I have tried to allocate using pointer, direct array
var and pointer to type array

a)
var
     p : pointer of byte;
...
     GetMem(p,100000000); //100 MB
b)
var
     a : array[1..100000000] of byte; //100MB
c)
type
     a = array[1..100000000] of byte; //100MB
     pa = ^a;
var
     p : pa;
...
     GetMem(p,sizeof(a));

Also tried changing 100M of byte with 25M of cardinal but got the same error.

Operating system is  WinXP SP2. AthlonXP 2000+ and 1.25 GBRam

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Caching a big file on memory

Leonardo M. Ramé
In reply to this post by Eduardo Morras
I don't tested this with large files, but why don't you use TMemoryStream:

var
  myFile: TMemoryStream;

begin
  myFile := TMemoryStream.Create;
  myFile.LoadFromFile('c:\myfile');
  ...
end;

--- Eduardo <[hidden email]> wrote:

> Hello:
>
> I have a little problem, i participe in the 'Large Compression
> Benchmark' (http://cs.fit.edu/~mmahoney/compression/text.html) and
> need to cache the whole enwik8 file (100MB) on memory but when i try
> to  allocate space for it i get a compiler error, saying that maximum
> is 64MB (67108864). I tried to use $MEMORY 200000000,500000000 (200MB
> stack and 500MB heap) and the compiler switches, but no success.
>
> I need all the file on RAM beacause i want to do a BWT, which
> implies a sort of the 100MB and uses 400 more MB, and doing so from
> hard disk is very slow. The contest has a time limit of 10 hours on a
> 1Ghz machine, but i need all speed at BWT/sort phase, so i can spend
> the time on getting better compression.
>
> Must i touch the system, rtl etc... files and recompile compiler? Or
> is there another, and simpler, way?
>
> P.D. Of course, my pc has more than 100 MB (1.25 GB)
>
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>


Leonardo M. Ramé
http://leonardorame.blogspot.com

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Caching a big file on memory

Marco van de Voort
In reply to this post by Eduardo Morras
> >On 11 Sep 2006, at 18:09, Eduardo wrote:
> Also tried changing 100M of byte with 25M of cardinal but got the same error.
>
> Operating system is  WinXP SP2. AthlonXP 2000+ and 1.25 GBRam

What are you compiling _for_ ?  The plain dos compiler has a limit of 64MB
under Windows. Make sure you use the Windows (win32) compiler under windows.

fpc -iTO

will show the target. If it says "go32v2" you have the wrong (dos) compiler.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Caching a big file on memory

Eduardo Morras
In reply to this post by Leonardo M. Ramé
At 19:54 11/09/2006, you wrote:

>I don't tested this with large files, but why don't you use TMemoryStream:
>
>var
>   myFile: TMemoryStream;
>
>begin
>   myFile := TMemoryStream.Create;
>   myFile.LoadFromFile('c:\myfile');
>   ...
>end;

Not need OOP for do it, i need as much speed as i can and using a
plain array is faster than a stream. Also, now all compiles, perhaps
for a temp files cleaning or the dos<->win32 (i don't use "windows"
in uses clause, or my ide have dos as default) as Marco said.

Thanks again to all.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal