source file size ?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

source file size ?

Tony Pelton
hi all,

i wanted to ask a question, before i spend time going down a road to
implement it.

are there any know limits on how big a source file can be for the FPC compiler ?

i was thinking about experimenting with "code generating" a source
file that would become a memory resident database of sorts.

this would make my application, implemented as a shared library
"plugin" quite fast, and would potentially simplify the distribution
of my shared library to my user base.

are there any thoughts on having a source file that might be megabytes
in size that is basically just structured data implemented as source ?

tia,
Tony

--
X-SA user ? 0.4 is out !
http://x-plane.dsrts.com
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

L505



> are there any know limits on how big a source file can be for the FPC compiler ?

interesting question... hmm did you try a  3GB file which calls writeln('test') over
and over and over again? That's the stupid way of doing tests - the smart way would
be waiting for an answer from the FPC team.

Here are my prefered colors for your website (take it or leave it):

  body bgcolor="#1E1E1E" text="#D7D7D7" link="#006699" alink="#006699"
vlink="#006699"

Gotta love dark backgrounds.

hmmm website Sigs really do work because I went ahead and clicked on your site.

--
L505

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

Tony Pelton
On 1/5/06, L505 <[hidden email]> wrote:
>
> > are there any know limits on how big a source file can be for the FPC compiler ?
>
> interesting question... hmm did you try a  3GB file which calls writeln('test') over
> and over and over again?

> That's the stupid way of doing tests - the smart way would
> be waiting for an answer from the FPC team.

hmmm ... i'm sensing some sarcasm.

i'm not usually one to post dumb questions to a list because i'm lazy
or have nothing better to do.

also, i thought this was the "users" list ... not the compiler developers list.

in any event ... i have no idea if putting 3GB's of "writeln"
statements in a file is a good test.

in my case, the data would have to have structure.

it isn't just the size of the source file, but the size of the code as
generated by the compiler into the DLL.

are there limits to the "code segment" or other such internals of the
compiler and runtime file ?

i think the way i would have to lay it out would be to have 1000's of
Pascal "record" types, with members that are a mixture of "constant"
String and Double values.

i guess i would have to put all of these "record"s inside of a
procedure that alloced and assigned them all as members of an array,
upon invocation of the function at runtime ?

maybe i could put them in the initializer ?

it would actually take me some time to even lay the thing out to begin
with ... only to find that it doesn't work ... or that I _assume_ it
won't work ... because i took one path that didn't work, where there
might have been another that would have ... had i known.

so you see ... i feel like it is maybe more complicated that just
testing by echoing 3GB's of "writeln" statements into a source file
and compiling it.

if no one has any thoughts on it ... i'll go to the next step ... but
i didn't think it would hurt to ask the question.

but thanks for the help.

>
> Here are my prefered colors for your website (take it or leave it):
>
>   body bgcolor="#1E1E1E" text="#D7D7D7" link="#006699" alink="#006699"
> vlink="#006699"
>
> Gotta love dark backgrounds.
>
> hmmm website Sigs really do work because I went ahead and clicked on your site.

hmmm ... i just don't know if i'm reading into the intent of your
comments very well.

btw ... the screenshots you see there are screenshots of a plugin i've
written for a simulator i use from http://www.x-plane.com ... using
Free Pascal.

>
> --
> L505

Tony

--
X-SA user ? 0.4 is out !
http://x-plane.dsrts.com
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

Vincent Snijders
In reply to this post by Tony Pelton
Tony Pelton wrote:

> hi all,
>
> i wanted to ask a question, before i spend time going down a road to
> implement it.
>
> are there any know limits on how big a source file can be for the FPC compiler ?
>
> i was thinking about experimenting with "code generating" a source
> file that would become a memory resident database of sorts.
>
> this would make my application, implemented as a shared library
> "plugin" quite fast, and would potentially simplify the distribution
> of my shared library to my user base.
>
> are there any thoughts on having a source file that might be megabytes
> in size that is basically just structured data implemented as source ?

I think there are no hard limits on the source size. But you have to
keep in mind that procedures (and maybe data) cannot become infinitely
long, generating fast object code for them would be too complicated.

I would definitely try it.

BTW, lazarus is doing something similar, generating large strings that
act as resource, the largest several 100's of KB. But they are
statically linked.

Vincent

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

Marco van de Voort
> Tony Pelton wrote:
> > i wanted to ask a question, before i spend time going down a road to
> > implement it.
> >
> > are there any know limits on how big a source file can be for the FPC compiler ?
> >
> > i was thinking about experimenting with "code generating" a source
> > file that would become a memory resident database of sorts.
> >
> > this would make my application, implemented as a shared library
> > "plugin" quite fast, and would potentially simplify the distribution
> > of my shared library to my user base.
> >
> > are there any thoughts on having a source file that might be megabytes
> > in size that is basically just structured data implemented as source ?
>
> I think there are no hard limits on the source size. But you have to
> keep in mind that procedures (and maybe data) cannot become infinitely
> long, generating fast object code for them would be too complicated.

Correct, and extremely long sourcefiles will also use lots of memory, so
there is also a more practical limit there.
 
IIRC the codegenerator was tested using 6MB large procedures, if you go over
a few MBs per procedure you might get into trouble.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

Florian Klämpfl
In reply to this post by Tony Pelton
Tony Pelton wrote:

> hi all,
>
> i wanted to ask a question, before i spend time going down a road to
> implement it.
>
> are there any know limits on how big a source file can be for the FPC compiler ?
>
> i was thinking about experimenting with "code generating" a source
> file that would become a memory resident database of sorts.
>
> this would make my application, implemented as a shared library
> "plugin" quite fast, and would potentially simplify the distribution
> of my shared library to my user base.
>
> are there any thoughts on having a source file that might be megabytes
> in size that is basically just structured data implemented as source ?

What means structured data? Source data size doesn't matter, they can be
up to 2 GB, the important thing is the size of the resulting parsing
pieces. I.e. very long procedures (>10k lines), statements, expressions
call for trouble. If you create a 1M line file with 100k procedures or
declarations, I don't expect trouble.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

Tony Pelton
On 1/5/06, Florian Klaempfl <[hidden email]> wrote:

> Tony Pelton wrote:
> > hi all,
> >
> > i wanted to ask a question, before i spend time going down a road to
> > implement it.
> >
> > are there any thoughts on having a source file that might be megabytes
> > in size that is basically just structured data implemented as source ?
>
> up to 2 GB, the important thing is the size of the resulting parsing
> pieces. I.e. very long procedures (>10k lines), statements, expressions
> call for trouble.

this is the answer that answers my question i think.

while i hadn't thought it all the way through, in fact, the picture i
had in my head was one where there would be one massive function, that
i would call when my DLL was initialized, that would create all of the
"records" of data and add them to an array or collection for use by
the application at runtime.

i might be able to break that function into several somewhat smaller
functions, and aggregate the calls to each on via one.

procedure init();
begin
do100();
do200();
do300();
.
.
.
end;

procedure do100();
begin
{ add 1-100 of the N elements here }
.
.
.
end;

procedure do200();
begin
{ add 101-200 of the N elements here }
.
.
.
end;

thanks for the thoughts ...

Tony
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

L505
In reply to this post by Tony Pelton

>
> > are there any know limits on how big a source file can be for the FPC compiler ?
>
> interesting question... hmm did you try a  3GB file which calls writeln('test')
over
> and over and over again?

> That's the stupid way of doing tests - the smart way would
> be waiting for an answer from the FPC team.

> > hmmm ... i'm sensing some sarcasm.

None intended, I was serious.. personally, I tried the test and what will limit you
is the memory on your pc. Thousands of writeln on my windows machine said "you are
out of virtual memory". So this will only be your problem, as far as I tested..


>i'm not usually one to post dumb questions to a list because i'm lazy
>or have nothing better to do.

It wasn't a dumb question - I'm a user too, I'm not a developer (yet).


>in any event ... i have no idea if putting 3GB's of "writeln"
>statements in a file is a good test.

It sort of is, at least I found out something new.. the virtual memory blew up on my
PC at 180704 lines of source code ... with writeln('test');

> in my case, the data would have to have structure.

That too, you could try something like a simple record over and over again. Your data
wouldn't be over and over again, but it would still tell you quickly if FPC can
handle 400MB of source code, 4GB, etc. A writeln('test') would still cause you to
place thousands of strings into the exe and etc.

My idea was to let the FPC team tell you more about it than doing a silly test like
this.. because I'm just a user and they know more. But I suppose it's not that silly
because I learned something about virtual
memory when I tried it.

>
> Here are my prefered colors for your website (take it or leave it):
>
>   body bgcolor="#1E1E1E" text="#D7D7D7" link="#006699" alink="#006699"
> vlink="#006699"
>
> Gotta love dark backgrounds.
>
> hmmm website Sigs really do work because I went ahead and clicked on your site.

hmmm ... i just don't know if i'm reading into the intent of your
comments very well.

I've experimented with dark backgrounds and found an "off black" to be nice on
websites. I've found white on black is nice especially when it's an "off black" and
an "off white". I was complimenting your site.. here I have written about this a
while back:

http://z505.com/cgi-bin/qkcont/qkcont.cgi?p=black%20backgrounds

Only advantage with the old corporate black on white is that it is easier to send to
the printer.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Moz
Reply | Threaded
Open this post in threaded view
|

Re: source file size ?

Moz
In reply to this post by Tony Pelton
Tony Pelton said:
> are there any know limits on how big a source file can be for the FPC compiler ?

Why don't you just compile a binary resource into your program? I
assume that fpc can do the same thing as TP/Delphi somehow. That would
keep your source file nice and small but do what you want. Editing
large files is a considerable pain, they take ages to load.

Moz

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal