Placing binary data (resources) in object files?

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Placing binary data (resources) in object files?

Anthony Walter-3
I am trying to come up with an embedded binary resource system and was considering placing data in object files when I found this question on stack overflow:  http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using-gcc

I see that it's possible to directly turn any file into an object file and reference the memory of that as a resource in a C program using:

extern char binary_foo_bar_start[];
extern char binary_foo_bar_end[];

My question, is it possible to link extern char in free pascal in a way similar to above? Perhaps something similar to:

var binary_foo_bar_start: pointer; external;

{$L foo.bar.o}

Or some permutation of the above?

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Ewald-2
Once upon a time, Anthony Walter said:

> I am trying to come up with an embedded binary resource system and was
> considering placing data in object files when I found this question on
> stack overflow:
>  http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using-gcc
>
>
> I see that it's possible to directly turn any file into an object file
> and reference the memory of that as a resource in a C program using:
>
> extern char binary_foo_bar_start[];
> extern char binary_foo_bar_end[];
>
> My question, is it possible to link extern char in free pascal in a
> way similar to above? Perhaps something similar to:
>
> var binary_foo_bar_start: pointer; external;
>
> {$L foo.bar.o}
>
> Or some permutation of the above?

You can pass an option directly to the link ( add -k--format=binary
-k<binary files you wish to embed> -k--format=default to the fpc command
line), or you could use objcopy.

These two aproaches aren't very portable though, so it is best you write
a script or something that generates code that contains this file.
Something like this:

===php code begin===
<?php

function ObjCopy($file, $name, $outputname = 0)
{
    $contents = file_get_contents($file);
    $size = strlen($contents);
   
    if ($outputname === 0)
        $outputname = "blob_$name.o";
   
    $invalidsymbolcharacters = array('!', '@', '#', '$', '%', '^', '&',
'*', '(', ')', '_', '+', '-', '=', '[', ']', '{', '}', '\'', '\\', '"',
'|', ';', ':', ',', '.', '/', '<', '>', '?', '`', '~');
    $name = str_replace($invalidsymbolcharacters, '_', $name);

    $data = '';
    for ($cnt=0; $cnt < $size; $cnt++)
        $data.= '0x'.dechex(ord($contents[$cnt])).',';
    $data = substr($data, 0, -1);

    $tmpfile = "#include <cstddef>\n";
    $tmpfile.= "extern \"C\" const size_t _binary_{$name}_size = $size;\n";
    $tmpfile.= "extern \"C\" const char _binary_{$name}_start[] =
{{$data}};\n";
    $tmpfile.= "extern \"C\" const char* _binary_{$name}_end =
&_binary_{$name}_start[0] + $size-1;\n";

    fwrite($handle = popen("gcc -o '$outputname' -c -x c++ - -O3 -Wall
-pipe", 'w'), $tmpfile);
    pclose($handle);
}

?>
===php code end===

Usage: `ObjCopy('binaryinputfile', 'symol name',
'outputobjectfile.o');`. Then add `-koutputobjectfile.o` to the command
line of fpc. Next you can define your symbols in pascal as follows:

{$escapedpascalname}_start: array[0..<$length-1>] of cuint8; external
name '_binary_{$escapedname}_start';
{$escapedpascalname}_end: array[<1-$length>."..0] of cuint8; external
name '_binary_{$escapedname}_end';
{$escapedpascalname}_size: array[0..0] of cuint8; external name
'_binary_{$escapedname}_size';

Where {$escapedpascalname} is the name you gave as the second argument
to the function above. `$length` is the length of the original data in
bytes. To read the size, just use `csize_t(@{$escapedpascalname}_size)`.

Note that the above is a very quick draft (kind of a quickfix for some
other project -- but is works) and thus contains errors and lacks error
checking. Use at your own risk ;-)

PS: Sorry about the php code there; I know it's not the php mailing
list, but my buildtool happens to be written in php ;-)

--
Ewald

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Tomas Hajny-2
On Mon, June 3, 2013 18:09, Ewald wrote:

> Once upon a time, Anthony Walter said:
>> I am trying to come up with an embedded binary resource system and was
>> considering placing data in object files when I found this question on
>> stack overflow:
>>  http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using-gcc
>>
>>
>> I see that it's possible to directly turn any file into an object file
>> and reference the memory of that as a resource in a C program using:
>>
>> extern char binary_foo_bar_start[];
>> extern char binary_foo_bar_end[];
>>
>> My question, is it possible to link extern char in free pascal in a
>> way similar to above? Perhaps something similar to:
>>
>> var binary_foo_bar_start: pointer; external;
>>
>> {$L foo.bar.o}
>>
>> Or some permutation of the above?
>
> You can pass an option directly to the link ( add -k--format=binary
> -k<binary files you wish to embed> -k--format=default to the fpc command
> line), or you could use objcopy.
>
> These two aproaches aren't very portable though, so it is best you write
> a script or something that generates code that contains this file.
> Something like this:
 .
 .
> PS: Sorry about the php code there; I know it's not the php mailing
> list, but my buildtool happens to be written in php ;-)

Well, you can obviously use tool bin2obj created with/for FPC and
distributed with FPC releases - see fpcsrc/utils/bin2obj.pp.

Tomas


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Ewald-2
Once upon a time, Tomas Hajny said:
> On Mon, June 3, 2013 18:09, Ewald wrote:
> Well, you can obviously use tool bin2obj created with/for FPC and
> distributed with FPC releases - see fpcsrc/utils/bin2obj.pp.
Seriously? I should really start looking at included tools... I've got
this feeling that it will save me some time.

Anyway, reinventing the weel was quite fun ;-)

--
Ewald

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Rainer Stratmann
In reply to this post by Anthony Walter-3
 On Monday 03 June 2013 17:47:46 you wrote:

> I am trying to come up with an embedded binary resource system and was
> considering placing data in object files when I found this question on
> stack overflow:
> http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using
> -gcc
>
> I see that it's possible to directly turn any file into an object file and
> reference the memory of that as a resource in a C program using:
>
> extern char binary_foo_bar_start[];
> extern char binary_foo_bar_end[];
>
> My question, is it possible to link extern char in free pascal in a way
> similar to above? Perhaps something similar to:

There was a similar discussion about that.

See topic: 'Including binary data making easy'

The result of this discussion was that you can do this (inconvenient) with
existing tools.

And that it is not necessary to make improvements so that the compiler can do
(simple) things other compilers already can do for a long time.


> var binary_foo_bar_start: pointer; external;
>
> {$L foo.bar.o}
>
> Or some permutation of the above?
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Rainer Stratmann
In reply to this post by Anthony Walter-3
 On Monday 03 June 2013 17:47:46 you wrote:

> I am trying to come up with an embedded binary resource system and was
> considering placing data in object files when I found this question on
> stack overflow:
> http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using
> -gcc
>
> I see that it's possible to directly turn any file into an object file and
> reference the memory of that as a resource in a C program using:
>
> extern char binary_foo_bar_start[];
> extern char binary_foo_bar_end[];
>
> My question, is it possible to link extern char in free pascal in a way
> similar to above? Perhaps something similar to:
>
> var binary_foo_bar_start: pointer; external;
>
> {$L foo.bar.o}
>
> Or some permutation of the above?

For example: (see mentioned topic)

// get access to the data of file.dat
// have 1 executable file with containing all the data

{$ir file.dat mydata} // include resource or similar

// fpc compiler makes this
const
 mydata : array[ 0 .. 4 ] of byte =
 ($45,$44,$63,$76,$55); // file data

// the length can find out with sizeof( mydata );
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Anthony Walter-3
Wow, thanks for all the interesting replies. In the mean time here is what I came up with:

 ld -r -b binary -o binary.o helloworld.txt

#include <stdint.h>

extern char _binary_helloworld_txt_start[];
extern char _binary_helloworld_txt_end[];

void* helloworld(uintptr_t* i) {
uintptr_t a = (uintptr_t)&_binary_helloworld_txt_start;
uintptr_t b = (uintptr_t)&_binary_helloworld_txt_end;
(*i) = b - a;
return &_binary_helloworld_txt_start;
}

gcc -c -o test.o test.c
ar rcs libtest.a test.o binary.o

program testresources;

type
  TExternResource = function(out Size: UIntPtr): Pointer;

function ResString(Res: TExternResource): string;
var
  P: Pointer;
  I: UIntPtr;
begin
  P := Res(I);
  SetLength(Result, I);
  Move(PChar(P)[0], PChar(Result)[0], I);
end;

function helloworld(out Size: UIntPtr): Pointer; cdecl; external;

{$linklib test}

var
  S: string;
begin
  S := ResString(@helloworld);
  WriteLn(S);
end.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Rainer Stratmann

For me this seems (too) complicated to do an easy thing (including some data).


 On Monday 03 June 2013 18:51:59 you wrote:

> Wow, thanks for all the interesting replies. In the mean time here is what
> I came up with:
>
>  ld -r -b binary -o binary.o helloworld.txt
>
> #include <stdint.h>
>
> extern char _binary_helloworld_txt_start[];
> extern char _binary_helloworld_txt_end[];
>
> void* helloworld(uintptr_t* i) {
> uintptr_t a = (uintptr_t)&_binary_helloworld_txt_start;
> uintptr_t b = (uintptr_t)&_binary_helloworld_txt_end;
> (*i) = b - a;
> return &_binary_helloworld_txt_start;
> }
>
> gcc -c -o test.o test.c
> ar rcs libtest.a test.o binary.o
>
> program testresources;
>
> type
>   TExternResource = function(out Size: UIntPtr): Pointer;
>
> function ResString(Res: TExternResource): string;
> var
>   P: Pointer;
>   I: UIntPtr;
> begin
>   P := Res(I);
>   SetLength(Result, I);
>   Move(PChar(P)[0], PChar(Result)[0], I);
> end;
>
> function helloworld(out Size: UIntPtr): Pointer; cdecl; external;
>
> {$linklib test}
>
> var
>   S: string;
> begin
>   S := ResString(@helloworld);
>   WriteLn(S);
> end.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Ewald-2
Once upon a time, Rainer Stratmann said:
> For me this seems (too) complicated to do an easy thing (including some data).
That, and I don't think that ld on mac os x can handle `-b` (or --format).

--
Ewald

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Anthony Walter-3
I think I'll use Rainer's example:

{$ir file.dat mydata}

I just hope the final exe won't get bloated if resource are included in this manner and never used and also that it does slow down compilation every time unless I add to new resources or change code in the unit containing the resource.

What I basically have is some large jpgs, test scripts, compressed sound files and fonts, and I want them built into my library but discarded if no code references them. That is, if an app doesn't use a skybox (there are no references to the skybox class), then the default skybox jpeg resource is not linked into the exe.

1024*1024*6 pixel jpgs


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Rainer Stratmann
 On Monday 03 June 2013 20:01:45 you wrote:
> I think I'll use Rainer's example:
>
> {$ir file.dat mydata}

You can try, but I am not aware that this will work.

It was a suggestion. :-)
 
> I just hope the final exe won't get bloated if resource are included in
> this manner and never used and also that it does slow down compilation

Why should this slow down compilation.
It is only copy some data from a to b.

> every time unless I add to new resources or change code in the unit
> containing the resource.
>
> What I basically have is some large jpgs, test scripts, compressed sound
> files and fonts, and I want them built into my library but discarded if no
> code references them. That is, if an app doesn't use a skybox (there are no
> references to the skybox class), then the default skybox jpeg resource is
> not linked into the exe.
>
> http://www.mapcore.org/topic/2829-second-release-5-skies/
> 1024*1024*6 pixel jpgs
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Graeme Geldenhuys-3
In reply to this post by Ewald-2
On 2013-06-03 17:32, Ewald wrote:
>> Well, you can obviously use tool bin2obj created with/for FPC and
>> > distributed with FPC releases - see fpcsrc/utils/bin2obj.pp.
> Seriously? I should really start looking at included tools... I've got
> this feeling that it will save me some time.


That's how fpGUI includes all image and sound binaries into executables
for years. I do the same for my commercial apps. fpGUI includes a GUI
tool where you can simply drag-n-drop a file(s) onto the app, and it
converts that into binary array's or exports it to a file that can
easily be added to any project.


Regards,
  - Graeme -
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Anthony Walter-3
In reply to this post by Rainer Stratmann
Okay I found some interesting results after a bit of testing...

When using bin2obj you get a byte of array typed constant which you can include in your sources. The compiler will always include the resource whether it is referenced from code or not. Stripping the app (strip -s -g appname) does not remove the size the constants add to the app even if they aren't referenced by any other code.

Using an untyped constant such as "const Image = {$r image.base64};" still adds to the app size even if it's not referenced, but it can only be stripped out if no units are included which reference access to the constant. This means if a "function A" sets a string using part of the constant, then Image (in the example above) will be embedded and cannot be stripped, even if no code calls "function A".

image.base64 contains 480kb of encoded text 
'/9j/4AAQSkZJRgABAQEASABIAAD//gATQ3JlYXRlZCB3.....

What this means is that all string constants and typed constants are take up space in your executable whether your use them or not, which is not ideal IMO. The compiler ought to be able to detect when these constants are not referenced (much like not including the machine code of unused functions) and exclude them from the final app.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Jonas Maebe-2

On 03 Jun 2013, at 22:22, Anthony Walter wrote:

> What this means is that all string constants and typed constants are take
> up space in your executable whether your use them or not, which is not
> ideal IMO. The compiler ought to be able to detect when these constants are
> not referenced (much like not including the machine code of unused
> functions) and exclude them from the final app.

If you turn on "smart linking" (compile with -CX -XX), they will be removed by the linker.


Jonas_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Anthony Walter-3
Jonas,

It's not happening. I have a 700kb image base64 encoded in a file called orangesky.res. I never use "procedure TestResource;" and yet the exe is 700kb larger. The only way to get rid of that extra 700k is to delete the procedure. The smart linking option or using strip do not remove the 700k.

{ TestResource procedure is not called from any code }
procedure TestResource;
const
  Data = {$i orangesky.res}; // 700kb constant is always built into the app
var
  S: string;
  I: Integer;
begin
  S := Data;
  for I := 1 to 10 do
    WriteLn(S[I * 10]);
end; 

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Anthony Walter-3
Nevermind, Jonas you're right. I was missing the -C
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Rainer Stratmann
In reply to this post by Anthony Walter-3
 On Monday 03 June 2013 23:49:07 you wrote:

> Jonas,
>
> It's not happening. I have a 700kb image base64 encoded in a file called
> orangesky.res. I never use "procedure TestResource;" and yet the exe is
> 700kb larger. The only way to get rid of that extra 700k is to delete the
> procedure. The smart linking option or using strip do not remove the 700k.
>
> { TestResource procedure is not called from any code }
> procedure TestResource;
> const
>   Data = {$i orangesky.res}; // 700kb constant is always built into the app

This will include binary data???

Did I miss something?
How can you find out the legth of the included data?

> var
>   S: string;
>   I: Integer;
> begin
>   S := Data;
>   for I := 1 to 10 do
>     WriteLn(S[I * 10]);
> end;
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Rainer Stratmann
 On Tuesday 04 June 2013 12:16:24 you wrote:

>  On Monday 03 June 2013 23:49:07 you wrote:
> > Jonas,
> >
> > It's not happening. I have a 700kb image base64 encoded in a file called
> > orangesky.res. I never use "procedure TestResource;" and yet the exe is
> > 700kb larger. The only way to get rid of that extra 700k is to delete the
> > procedure. The smart linking option or using strip do not remove the
> > 700k.
> >
> > { TestResource procedure is not called from any code }
> > procedure TestResource;
> > const
> >
> >   Data = {$i orangesky.res}; // 700kb constant is always built into the
> >   app
>
> This will include binary data???
>
> Did I miss something?
> How can you find out the legth of the included data?

I see already. You have to make a resourcefile.

It would be too good if that would work as easy as possible.

> > var
> >
> >   S: string;
> >   I: Integer;
> >
> > begin
> >
> >   S := Data;
> >   for I := 1 to 10 do
> >  
> >     WriteLn(S[I * 10]);
> >
> > end;
>
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal 
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Anthony Walter-3
I came with a nice solution to all this and thought I'd share... 

I wrote a tool named makeres. In Lazarus edit the default project configuration adding Compilation | Execute Before | makeres $(CompPath). Then when ever you build/run a project the following happens:

* Two directories are created if they don't already exist, one named raw the other resources.
* Any files in raw get compressed and converted to base64 encoded res files placed in the resources folder
* A unit is automatically maintained for your project named ProjectName.Resources
* The ProjectName.Resources unit contains the following (example resources from raw)

unit ProjectName.Resources;

{$mode delphi}

interface

{doc off}
{ To strip unused resources turn off debug symbols and compile with -CX -XX }

const
  ResLargeFont = {$i resources/large.font.base64};
  ResShaderFrag = {$i resources/shader.frag.base64};
  ResShaderVert = {$i resources/shader.vert.base64};
  ResSkyboxJpeg = {$i resources/skybox.jpg.base64};
{doc on}

implementation

end.

* Then to decompressed and read a resources in your app you call call any of these functions (built into my library)

{ Convert a resource into a memory location }
function ResMemory(constref Resource: string): Pointer;
{ Convert a resource into a stream }
function ResStream(constref Resource: string): TStream;
{ Convert a resource into a string }
function ResString(constref Resource: string): string;

So for example you can say:

Font.LoadFromStream(ResStream(ResLargeFont));
or
Shader.Compile(ResString(ResShaderFrag), shaderFragment);

So in summary, you just drop files into your raw folder, and use the above two lines of code. That's all you have to do.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Placing binary data (resources) in object files?

Luiz Americo Pereira Camara
2013/6/4 Anthony Walter <[hidden email]>:
> I came with a nice solution to all this and thought I'd share...
>
> I wrote a tool named makeres.

Is it open source?

Luiz
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
12