Resource compilation

classic Classic list List threaded Threaded
47 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Michael Van Canneyt


On Sat, 4 Aug 2018, Martok wrote:

> Am 25.09.2017 um 14:24 schrieb Sven Barth via fpc-pascal:
>> The RC language itself isn't *that* difficult. Main difficulty would be to essentially implement a C-preprocessor-compatible
>> preprocessor.
>
> Just "a few" months later, I finally got around to checking this out.
>
> You were completely right, the language itself is not very difficult, and can be processed into a TResource in one pass in a
> tply-generated parser. The preprocessor is also just a subset of C's (i.e.: no macros), so it would not be too complicated (I've
> done the same for idlproc before). There are dialect issues between windres and Microsoft RC, but they are minor.
>
> The more difficult part is implementing the remaining resource access classes for fcl-res. Well, not difficult per se, but the
> copy-on-write-sharing thing done there adds a lot of complexity for (IMHO) very little benefit and makes it a bit awkward to
> work with. There's also a lot of endian conversions that could probably be factored out.
>
> That sounds challenging. How much interest would there be in having a full resource compiler, with the appropriate
> changes/extensions to fcl-res?

I think a lot. There is a request in the bugtracker for a resource compiler since ages.

Michael.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Marco van de Voort
In our previous episode, Michael Van Canneyt said:
> > changes/extensions to fcl-res?
>
> I think a lot. There is a request in the bugtracker for a resource compiler since ages.

It comes up regularly. Many people don't even pursue it further, and just
use resources generator by e.g. an old delphi project if needed.

But longterm, we need this for completeness sake.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Michael Van Canneyt


On Sat, 4 Aug 2018, Marco van de Voort wrote:

> In our previous episode, Michael Van Canneyt said:
>> > changes/extensions to fcl-res?
>>
>> I think a lot. There is a request in the bugtracker for a resource compiler since ages.
>
> It comes up regularly. Many people don't even pursue it further, and just
> use resources generator by e.g. an old delphi project if needed.
>
> But longterm, we need this for completeness sake.

Well, I could use it short term too :)

I imagine lazarus could use it as well to bind icons etc.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Giulio Bernardi
Hi everybody, I'm just chiming in after quite some years.
Just two notes:

* For lazarus, it shouldn't need to use a real resource compiler for standard
things like icons, manifests etc: see programs res6 and res7 at
https://www.freepascal.org/docs-html/fclres/basic%20usage.html

* For the "real" resource compiler, regarding Martok's comment:

> The more difficult part is implementing the remaining resource access
> classes for fcl-res. Well, not difficult per se, but the
> copy-on-write-sharing thing done there adds a lot of complexity for (IMHO)
> very little benefit and makes it a bit awkward to work with. There's also
> a lot of endian conversions that could probably be factored out.

Actually, fcl-res was designed as a library to be used by a resource compiler,
and I don't think you need to implement new classes at all: once you have
written a parser for .rc files you just have to read the files referred from the
.rc file (icons, raw data etc), create the right kind of resource and add to the
output TResources. IIRC all the standard resource types were already
implemented, so once you write a parser that handles .rc files you should be done.

Giulio


On 04/08/18 09:43, Michael Van Canneyt wrote:

>
>
> On Sat, 4 Aug 2018, Marco van de Voort wrote:
>
>> In our previous episode, Michael Van Canneyt said:
>>> > changes/extensions to fcl-res?
>>>
>>> I think a lot. There is a request in the bugtracker for a resource compiler
>>> since ages.
>>
>> It comes up regularly. Many people don't even pursue it further, and just
>> use resources generator by e.g. an old delphi project if needed.
>>
>> But longterm, we need this for completeness sake.
>
> Well, I could use it short term too :)
>
> I imagine lazarus could use it as well to bind icons etc.
>
> Michael.
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Resource compilation

Martok
Hi!

Good feedback so far. Given recent... conflicts, I wanted to ask first, just to be sure.

> * For lazarus, it shouldn't need to use a real resource compiler for standard
> things like icons, manifests etc: see programs res6 and res7 at
> https://www.freepascal.org/docs-html/fclres/basic%20usage.html
That is probably enough for simple resource editor, yes. I believe this is what Lazarus already does for VERSIONINFO resources (?)


> Actually, fcl-res was designed as a library to be used by a resource compiler,
> and I don't think you need to implement new classes at all: once you have
> written a parser for .rc files you just have to read the files referred from the
> .rc file (icons, raw data etc), create the right kind of resource and add to the
> output TResources. IIRC all the standard resource types were already
> implemented, so once you write a parser that handles .rc files you should be done.
In part, yes. Particularly the really difficult types are already implemented. I would attempt to implement all remaining RT_x
types, including those that are not commonly used in cross-platform targets. Things like DIALOG or MENU have a custom format in
their RawData, but that should be fairly straightforward, basically doing the same as the ACCELERATOR resource and tutorial.

--
Regards,
Martok

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Martok
Hi again,

a quick update: I have a somewhat-working version, but many things are still
missing.
We have: a preprocessor, full support for BITMAP, ICON, CURSOR, resource
attributes (LANGUAGE etc.), all RCDATA-like resources, including their
definition from inline literal statements.

What's missing:
 - UnicodeString Strings (L"foo")
 - various forms of escape sequences for strings
 - Concatenated adjacent strings ("ABC" "DEF")
 - numeric expressions (used for things like computing flags).
 - complex resources: ACCELERATORS DIALOG/EX MENU/EX STRINGTABLE VERSIONINFO

The first two are not that difficult, just awkward (as all encoding-aware
stringhandling is), adjacent strings are more of an issue: turns out this
creates a problem in the language:
    FOO BITMAP "900.bmp"
    "A NAME" TEXT BEGIN "A String" END
This can't parse right: the filename and resid get concatenated to "900.bmpA
NAME". While entirely correct, that's not very useful. I might leave that out
until someone needs it.
Of the complex resources, STRINGTABLE and VERSIONINFO are probably the more
important ones. Luckily, both already have resource types implemented, so that
shouldn't be too hard either.
It's also leaking memory like a rusty bucket, I have so far completely ignored
what transfers object ownership in fcl-res classes. Heaptrc will hopefully help
here.

I only get to work on this on about one day a week and spend half of that trying
to keep the grammar somewhat readable and with no bad reduce conflicts, but if
anyone wants to play along:
<https://github.com/martok/freepascal/compare/master...fpcres-rc>

--
Regards,
Martok

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Michael Van Canneyt


On Tue, 28 Aug 2018, Martok wrote:

> Hi again,
>
> a quick update: I have a somewhat-working version, but many things are still
> missing.
> We have: a preprocessor, full support for BITMAP, ICON, CURSOR, resource
> attributes (LANGUAGE etc.), all RCDATA-like resources, including their
> definition from inline literal statements.
>
> What's missing:
> - UnicodeString Strings (L"foo")
> - various forms of escape sequences for strings
> - Concatenated adjacent strings ("ABC" "DEF")
> - numeric expressions (used for things like computing flags).
> - complex resources: ACCELERATORS DIALOG/EX MENU/EX STRINGTABLE VERSIONINFO

Seems like you have the main things covered, though.
Nice job and keep up the good work :)

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Martok
In reply to this post by Martok
More Progress!

I've since finished STRINGTABLE and VERSIONINFO as well as mathematical expressions.

However, there's an issue I hope somebody can help with. Binutils has this test
case:

<https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=binutils/testsuite/binutils-all/windres/version_mlang.rc;hb=HEAD>

Every Resource compiler I have tried compiles this slightly differently, and
Resource Hacker doesn't understand one of them. Could somebody test this with MS
RC and share the res file?


Independently, TVersionResource emits one Translation block per translation
instead of one containing all translations. I take it that isn't formally wrong,
but it's also not optimal? Was this intentional?

--
Regards,
Martok

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Marco van de Voort
In our previous episode, Martok said:
> However, there's an issue I hope somebody can help with. Binutils has this test
> case:
>
> <https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=binutils/testsuite/binutils-all/windres/version_mlang.rc;hb=HEAD>
>
> Every Resource compiler I have tried compiles this slightly differently, and
> Resource Hacker doesn't understand one of them. Could somebody test this with MS
> RC and share the res file?

I happened to have an old win8 sdk around (vs community edition didn't seem
to have "RC"). Anyway:

http://www.stack.nl/~marcov/funkyres.res

 
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Martok

> I happened to have an old win8 sdk around (vs community edition didn't seem
> to have "RC"). Anyway:
>
> http://www.stack.nl/~marcov/funkyres.res
Thank you!

Funky indeed, I'll mark that down as a Reshacker bug then.

Off to String stuff then.

--
Regards,
Martok


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Martok
In reply to this post by Martok
Update time!

<https://github.com/martok/freepascal/compare/master...fpcres-rc>

>  - UnicodeString Strings (L"foo")Done

>  - various forms of escape sequences for strings
Done

>  - Concatenated adjacent strings ("ABC" "DEF")
This is weird: windres, brcc and rc all do something different when
concatenating mixed Unicode and CP strings, as in `L"a" "b"`, `"a" L"b"`.
I have no idea what the correct action would be.

>  - complex resources: ACCELERATORS DIALOG/EX MENU/EX STRINGTABLE VERSIONINFO
Probably going to leave out the Dialog-Related stuff, since it's not super
relevant for us anyway. Windows users can just use RC ;-)

> It's also leaking memory like a rusty bucket, I have so far completely ignored
> what transfers object ownership in fcl-res classes. Heaptrc will hopefully help
> here.
Fixed

All of windres's tests pass now, except for the Dialog related ones. I would
consider it more or less finished. Any more things to do? Otherwise, I'll stack
it into some patches.


A few lessons learned:
Using TStringList as a key-value-store (#define) is SLOW. Replacing it with a
TFPStringHashTable brought processing time for windows.h down from 4 seconds to
some 10ms. That's just insert operations.

Codepage Strings are weird if more than one CP is involved. There is almost no
way to carry a specific codepage anywhere: AnsiString[850] + Char =
AnsiString[CP_UTF8]... Also, ShortStrings are often converted to AnsiStrings, so
that doesn't reliably work either. Same for array of char. That made working
with plex awfully complicated, and I still think it's wrong in some cases. But
at least it works for most now.

Having a reentrant version of plex&pyacc would be nice. Flex and Bison can do it
these days.

To a lesser degree: something like Python's argparse would be nice. Everyone
builds their own argument parser, usually with very different concepts and
varying amounts of code duplication.


--
Regards,
Martok


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Resource compilation

Michael Van Canneyt


On Sun, 13 Jan 2019, Martok wrote:

>
> To a lesser degree: something like Python's argparse would be nice. Everyone
> builds their own argument parser, usually with very different concepts and
> varying amounts of code duplication.

Assuming you mean the command-line arguments:

What's wrong/missing with the functionality in TCustomApplication ?

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

CLI argument parsers (was: Resource compilation)

Martok
Am 13.01.2019 um 18:43 schrieb Michael Van Canneyt:
> Assuming you mean the command-line arguments:
>
> What's wrong/missing with the functionality in TCustomApplication ?
Other than that nobody in the wild seems to fully use it? ;-)
Even most of the examples in the compiler tree use homegrown parsers, or look
like [1].

I'm a fan of getopts and use it (with a small wrapper) for almost everything (i.
[2]), but it seems that many people are not aware it even exists. It's even
POSIX compliant, so the program's users won't be surprised by some
implementation particulars.
But even that is nowhere near as elegant as argparse. I just don't like retyping
usage statements when they could be autogenerated ;-)
And Subparsers are, while possible, always quite hacky.

Of course, if someone was to implement something now, we'd be in the
https://xkcd.com/927/ situation...


[1] https://github.com/graemeg/freepascal/blob/master/utils/svn2cvs/svn2cvs.pp#L494
[2] https://github.com/martok/buildtools/blob/master/buildutil.lpr

--
Regards,
Martok

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: CLI argument parsers (was: Resource compilation)

Michael Van Canneyt


On Mon, 14 Jan 2019, Martok wrote:

> Am 13.01.2019 um 18:43 schrieb Michael Van Canneyt:
>> Assuming you mean the command-line arguments:
>>
>> What's wrong/missing with the functionality in TCustomApplication ?
> Other than that nobody in the wild seems to fully use it? ;-)

Your count is a little biased. A total of 41 examples/tools use it.
14 utils distributed in the FPC tree use it.

home:~/fpc/packages> grep -i TCustomApplication */examples/*.* */tests/*.* ../utils/*.* ../utils/*/*.* | wc -l
41
home:~/fpc/packages> grep -i TCustomApplication  ../utils/*.* ../utils/*/*.* | wc -l
14

There is a lazarus IDE wizard for a console application.
Every Lazarus LCL application has it built-in. So it is used "in the wild".

The bare compiler tools do not use it because
a) They predate TCustomApplication
b) It introduces a dependency on something which is not in the RTL but in packages.

So that it's not used is simply incorrect. It's more used than the posix-y getopts
unit which is also present in the fpc tree.

You simply didn't know about it, most probably.

You're welcome to suggest improvements, but as your rightly observed,
we should not introduce another system.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: CLI argument parsers

Martok
I was 100% expecting that sort of answer.


Cheers,

Martok

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: CLI argument parsers

Michael Van Canneyt


On Mon, 14 Jan 2019, Martok wrote:

> I was 100% expecting that sort of answer.

My invitation to suggest improvements, you mean ?

I meant it: you're welcome to suggest improvements.

If you implement something like argpars and it can be used as backend for
TCustomApplication, I'm willing to rework TCustomApplication to use it.
This way we keep backwards compatibility and offer the possibility to extend
it.

I'm not blind to the shortcomings of TCustomApplication argument parsing.
It fully handles the unix '-a' '--aaa' scheme of doing things, but currently
does not handle options in command form
prog --globalopts command [--commandopts] args

So, if you can offer something to keep current functionality and allow this
as well: be my guest...

But e.g. the geptops way, which basically forces you to use a while loop is
really not acceptable, which is why the TCustomApplication interface is what
it is today.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: CLI argument parsers

Yann Mérignac
You can also try this: http://yann.merignac.free.fr/unit-cmdline.html

It's a command line parser I wrote when I needed it a few years ago.

I was inspired by these 2 articles:
- https://www.adacore.com/gems/gem-138-gnatcoll.command-line
- https://www.adacore.com/gems/gem-139-master-the-command-line-part-2

 

Le lun. 14 janv. 2019 à 09:41, Michael Van Canneyt <[hidden email]> a écrit :


On Mon, 14 Jan 2019, Martok wrote:

> I was 100% expecting that sort of answer.

My invitation to suggest improvements, you mean ?

I meant it: you're welcome to suggest improvements.

If you implement something like argpars and it can be used as backend for
TCustomApplication, I'm willing to rework TCustomApplication to use it.
This way we keep backwards compatibility and offer the possibility to extend
it.

I'm not blind to the shortcomings of TCustomApplication argument parsing.
It fully handles the unix '-a' '--aaa' scheme of doing things, but currently
does not handle options in command form
prog --globalopts command [--commandopts] args

So, if you can offer something to keep current functionality and allow this
as well: be my guest...

But e.g. the geptops way, which basically forces you to use a while loop is
really not acceptable, which is why the TCustomApplication interface is what
it is today.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: CLI argument parsers (was: Resource compilation)

Bart-48
In reply to this post by Martok
On Mon, Jan 14, 2019 at 12:56 AM Martok <[hidden email]> wrote:

> I'm a fan of getopts and use it (with a small wrapper) for almost everything (i.
> [2]), but it seems that many people are not aware it even exists. It's even
> POSIX compliant, so the program's users won't be surprised by some
> implementation particulars.

GetOpts gets it wrong sometimes: see
https://bugs.freepascal.org/view.php?id=19842

--
Bart
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: CLI argument parsers (was: Resource compilation)

Michael Van Canneyt


On Mon, 14 Jan 2019, Bart wrote:

> On Mon, Jan 14, 2019 at 12:56 AM Martok <[hidden email]> wrote:
>
>> I'm a fan of getopts and use it (with a small wrapper) for almost everything (i.
>> [2]), but it seems that many people are not aware it even exists. It's even
>> POSIX compliant, so the program's users won't be surprised by some
>> implementation particulars.
>
> GetOpts gets it wrong sometimes: see
> https://bugs.freepascal.org/view.php?id=19842

Hm. I must have missed this one.
I've assigned the bug to myself, will look at it ASAP.
I think getopts has changed since the bug was reported, but maybe the proposed fix is still valid.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: CLI argument parsers

Benito van der Zander
In reply to this post by Yann Mérignac

I wrote one, too: http://benibela.de/sources_en.html#rcmdline

Supports --linux=style and /windows style, and ' and " quotes



On 14.01.2019 10:49, Yann Mérignac wrote:
You can also try this: http://yann.merignac.free.fr/unit-cmdline.html

It's a command line parser I wrote when I needed it a few years ago.

I was inspired by these 2 articles:
  - https://www.adacore.com/gems/gem-138-gnatcoll.command-line
  - https://www.adacore.com/gems/gem-139-master-the-command-line-part-2

 

Le lun. 14 janv. 2019 à 09:41, Michael Van Canneyt <[hidden email]> a écrit :


On Mon, 14 Jan 2019, Martok wrote:

> I was 100% expecting that sort of answer.

My invitation to suggest improvements, you mean ?

I meant it: you're welcome to suggest improvements.

If you implement something like argpars and it can be used as backend for
TCustomApplication, I'm willing to rework TCustomApplication to use it.
This way we keep backwards compatibility and offer the possibility to extend
it.

I'm not blind to the shortcomings of TCustomApplication argument parsing.
It fully handles the unix '-a' '--aaa' scheme of doing things, but currently
does not handle options in command form
prog --globalopts command [--commandopts] args

So, if you can offer something to keep current functionality and allow this
as well: be my guest...

But e.g. the geptops way, which basically forces you to use a while loop is
really not acceptable, which is why the TCustomApplication interface is what
it is today.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
123