a suggestion...

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

a suggestion...

Krishna-33
Hi all,

Is there any particular reason for not compressing the release
tarballs (Linux f.e) with say bzip2 or even 7zip?  The uncompressed
tarball weighs in around 24M and I'm sure bzipping will reduce it by a
large margin.

Cheers,
Krishna

--
You think you know when you learn, are more
sure when you can write, even more when you
can teach, but certain when you can program
 - Alan Perlis
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: a suggestion...

Florian Klämpfl
Krishna wrote:
> Hi all,
>
> Is there any particular reason for not compressing the release
> tarballs (Linux f.e) with say bzip2 or even 7zip?  The uncompressed
> tarball weighs in around 24M and I'm sure bzipping will reduce it by a
> large margin.

Not really because it is simply a tar ball of several .tar.gz. Because
gzip is spread wider, we use this instead of bzip2/7zip.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: a suggestion...

Jonas Maebe-2

On 24 mei 2006, at 17:30, Florian Klaempfl wrote:

> Not really because it is simply a tar ball of several .tar.gz. Because
> gzip is spread wider, we use this instead of bzip2/7zip.

Isn't bzip2 available more or less everywhere nowadays? (at least  
where gzip is available, and in particular on Linux?)


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: a suggestion...

Florian Klämpfl
Jonas Maebe wrote:
>
> On 24 mei 2006, at 17:30, Florian Klaempfl wrote:
>
>> Not really because it is simply a tar ball of several .tar.gz. Because
>> gzip is spread wider, we use this instead of bzip2/7zip.
>
> Isn't bzip2 available more or less everywhere nowadays? (at least where
> gzip is available, and in particular on Linux?)
>

At least on debian woody it wasn't installed by default. No idea about
sarge.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re[2]: a suggestion...

Пётр Косаревский
FK> Jonas Maebe wrote:
>> On 24 mei 2006, at 17:30, Florian Klaempfl wrote:
>>
>>> Not really because it is simply a tar ball of several .tar.gz. Because
>>> gzip is spread wider, we use this instead of bzip2/7zip.
>> Isn't bzip2 available more or less everywhere nowadays? (at least where
>> gzip is available, and in particular on Linux?)
FK> At least on debian woody it wasn't installed by default. No idea about
FK> sarge.


Just wondering: is RAR considered "spread wide"?

I know that compressor is commercial, but uncompressor is not (sources
are available, if I remember right), and it
compresses sources better than gzip by 25% (often even more). There
are better compressors, they are free (sometimes even open source),
but support matters (dos, windows, pocket pc, linux, free bsd and
macos x versions are ready to download).

Or is it completely unacceptable because it isn't free?

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: a suggestion...

Jeff Pohlmeyer
In reply to this post by Krishna-33
> Because gzip is spread wider, we use this instead of bzip2/7zip.

I think the size saved by compressing FPC with bz2 would be much
greater than the size of downloading and installing a bzip2 extractor.


 - Jeff
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: a suggestion...

etrusco
In reply to this post by Пётр Косаревский
On 5/24/06, Пётр Косаревский с mail.ru <[hidden email]> wrote:
> FK> Jonas Maebe wrote:
> >> On 24 mei 2006, at 17:30, Florian Klaempfl wrote:
> >>
> >>> Not really because it is simply a tar ball of several .tar.gz. Because
> >>> gzip is spread wider, we use this instead of bzip2/7zip.
> >> Isn't bzip2 available more or less everywhere nowadays? (at least where
> >> gzip is available, and in particular on Linux?)
> FK> At least on debian woody it wasn't installed by default. No idea about
> FK> sarge.

AFAIR it is even used for .debs. Does priority "important" makes it
installed by default?

http://packages.qa.debian.org/b/bzip2.html
http://packages.debian.org/stable/utils/bzip2

> Just wondering: is RAR considered "spread wide"?
>

I would say definetely _no_.

> I know that compressor is commercial, but uncompressor is not (sources
> are available, if I remember right), and it

The open source version only supports old formats, the newest
format(s?) is only supported by a closed-source (but free as in beer)
version, which must be downloaded separately (even if it only requires
using synaptic...).

> compresses sources better than gzip by 25% (often even more). There
> are better compressors, they are free (sometimes even open source),
> but support matters (dos, windows, pocket pc, linux, free bsd and
> macos x versions are ready to download).

bzip2 has similar compression rates (except maybe for multimedia
files, which isn't the case) and 7zip/LZMA usually compresses better
than RAR.
7zip isn't installed by default in any distro AFAIK, but at least it
open source.

RAR would be my last option...

Cheers,
Flávio

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: a suggestion...

Florian Klämpfl
In reply to this post by Пётр Косаревский
ϸòð Êîñàðåâñêèé ñ mail.ru wrote:

> FK> Jonas Maebe wrote:
>>> On 24 mei 2006, at 17:30, Florian Klaempfl wrote:
>>>
>>>> Not really because it is simply a tar ball of several .tar.gz. Because
>>>> gzip is spread wider, we use this instead of bzip2/7zip.
>>> Isn't bzip2 available more or less everywhere nowadays? (at least where
>>> gzip is available, and in particular on Linux?)
> FK> At least on debian woody it wasn't installed by default. No idea about
> FK> sarge.
>
>
> Just wondering: is RAR considered "spread wide"?

No.

>
> I know that compressor is commercial,

Thus everybody building releases needs to buy a rar license?

> but uncompressor is not (sources
> are available, if I remember right), and it
> compresses sources better than gzip by 25% (often even more). There
> are better compressors, they are free (sometimes even open source),
> but support matters (dos, windows, pocket pc, linux, free bsd and
> macos x versions are ready to download).
>
> Or is it completely unacceptable because it isn't free?

So the fpc installer needs to deliver a unrar binary? Forget about this
on Linux, this requires more maintainance work than fpc itself ;)
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

Florian Klämpfl
In reply to this post by Jeff Pohlmeyer
Jeff Pohlmeyer wrote:
>> Because gzip is spread wider, we use this instead of bzip2/7zip.
>
> I think the size saved by compressing FPC with bz2 would be much
> greater than the size of downloading and installing a bzip2 extractor.

Yes, but it requires extra work :)
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

Marco van de Voort
> Jeff Pohlmeyer wrote:
> >> Because gzip is spread wider, we use this instead of bzip2/7zip.
> >
> > I think the size saved by compressing FPC with bz2 would be much
> > greater than the size of downloading and installing a bzip2 extractor.
>
> Yes, but it requires extra work :)

Just a side remark, FreeBSD has bzip2 in base since 5.x.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

adrian_m
In reply to this post by Florian Klämpfl
On 25/05/06, Florian Klaempfl <[hidden email]> wrote:
> Jeff Pohlmeyer wrote:
> >> Because gzip is spread wider, we use this instead of bzip2/7zip.
> >
> > I think the size saved by compressing FPC with bz2 would be much
> > greater than the size of downloading and installing a bzip2 extractor.
>
> Yes, but it requires extra work :)

What do you think about having two archives, to make everyone happy  :
one compressed with gzip ( for those concerned about compatibility)
and one compressed with bzip2  (for those concerned about the size) ?


--
Adrian Maier
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

Florian Klämpfl
Adrian Maier wrote:

> On 25/05/06, Florian Klaempfl <[hidden email]> wrote:
>> Jeff Pohlmeyer wrote:
>> >> Because gzip is spread wider, we use this instead of bzip2/7zip.
>> >
>> > I think the size saved by compressing FPC with bz2 would be much
>> > greater than the size of downloading and installing a bzip2 extractor.
>>
>> Yes, but it requires extra work :)
>
> What do you think about having two archives, to make everyone happy  :
> one compressed with gzip ( for those concerned about compatibility)
> and one compressed with bzip2  (for those concerned about the size) ?

Extra work for us :) We're already overloaded when doing releases.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

Tomas Hajny
Florian Klaempfl wrote:

> Adrian Maier wrote:
>> On 25/05/06, Florian Klaempfl <[hidden email]> wrote:
>>> Jeff Pohlmeyer wrote:
>>> >> Because gzip is spread wider, we use this instead of bzip2/7zip.
>>> >
>>> > I think the size saved by compressing FPC with bz2 would be much
>>> > greater than the size of downloading and installing a bzip2
>>> extractor.
>>>
>>> Yes, but it requires extra work :)
>>
>> What do you think about having two archives, to make everyone happy  :
>> one compressed with gzip ( for those concerned about compatibility)
>> and one compressed with bzip2  (for those concerned about the size) ?
>
> Extra work for us :) We're already overloaded when doing releases.

 ... plus it would have impact to required storage on all our mirrors.

Tomas

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

Jonas Maebe-2
In reply to this post by Florian Klämpfl

On 25 May 2006, at 12:43, Florian Klaempfl wrote:

>> What do you think about having two archives, to make everyone  
>> happy  :
>> one compressed with gzip ( for those concerned about compatibility)
>> and one compressed with bzip2  (for those concerned about the size) ?
>
> Extra work for us :) We're already overloaded when doing releases.

I really think requiring bzip2 for Linux users is not a problem.  
Those few Debian Woody users who might not yet have installed bzip2  
can easily do an "apt-get install bzip2"


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

Florian Klämpfl
Jonas Maebe wrote:

>
> On 25 May 2006, at 12:43, Florian Klaempfl wrote:
>
>>> What do you think about having two archives, to make everyone happy  :
>>> one compressed with gzip ( for those concerned about compatibility)
>>> and one compressed with bzip2  (for those concerned about the size) ?
>>
>> Extra work for us :) We're already overloaded when doing releases.
>
> I really think requiring bzip2 for Linux users is not a problem. Those
> few Debian Woody users who might not yet have installed bzip2 can easily
> do an "apt-get install bzip2"

I tend also to agree. Does anybody know something newer than Woody
coming without bzip2 by default?
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Re: a suggestion...

Felipe Monteiro de Carvalho
On 5/25/06, Florian Klaempfl <[hidden email]> wrote:
> I tend also to agree. Does anybody know something newer than Woody
> coming without bzip2 by default?

DamnSmallLinux

--
Felipe Monteiro de Carvalho
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re[4]: a suggestion...

Пётр Косаревский
In reply to this post by etrusco
> bzip2 has similar compression rates (except maybe for multimedia
> files, which isn't the case) and 7zip/LZMA usually compresses better
> than RAR.
> 7zip isn't installed by default in any distro AFAIK, but at least it
> open source.
> RAR would be my last option...

I tried to benchmark a little. Archivers were limited to 512 Mb. Timings only in the second test. Precision is kept 1 Mb/10 seconds intentionally.

There are many comprehensive benchmarks, but I tested nearly the last versions.

OpenOffice 2.0.2 sources, 1209 Mb (there are several .gz and other binary files inside).
(RAR and 7z had the advantage of managing files themselves to create "solid" archives, gzip and bzip2 were used on a 1209 Mb "ball" made by RAR.)

Rar (3.60 beta 2):  212 Mb
gzip (1.2.4):       276 Mb
bzip2 (1.0.3):      230 Mb
7z PPMd (4.4.2):    190 Mb
7z LZMA (--""--):   too slow (~100 Kb/s, more than three hours)

After that I tried to compress FPC SVN (153 Mb), and the numbers were in different order (FPC "ball" was created in 2 minutes by Rar):

Rar     21 Mb  5 minutes
gzip    29 Mb  1 minute 30 seconds
bzip2   20 Mb  3 minutes 40 seconds (is it optimized for pascal?)
7z PPMd 17 Mb  8 minutes 20 seconds
7z LZMA 16 Mb  25 minutes (well, it would be better, I believe it had memory problems)

I tried PPMonster archiver (open source, from the author of PPMd algorithm), while sources are less than 60 Kb in C, speed is often about 20 Kb/s. However, it promised 0.548 bits per byte of the FPC "ball" (but >1.5 for OpenOffice).

Naturally, there is still no good reason to change something.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re[4]: a suggestion...

Eduardo Morras

>
>I tried to benchmark a little. Archivers were limited to 512 Mb.
>Timings only in the second test. Precision is kept 1 Mb/10 seconds
>intentionally.
>
>There are many comprehensive benchmarks, but I tested nearly the
>last versions.
>
>OpenOffice 2.0.2 sources, 1209 Mb (there are several .gz and other
>binary files inside).
>(RAR and 7z had the advantage of managing files themselves to create
>"solid" archives, gzip and bzip2 were used on a 1209 Mb "ball" made by RAR.)
>
>Rar (3.60 beta 2):  212 Mb
>gzip (1.2.4):       276 Mb
>bzip2 (1.0.3):      230 Mb
>7z PPMd (4.4.2):    190 Mb
>7z LZMA (--""--):   too slow (~100 Kb/s, more than three hours)
>
>After that I tried to compress FPC SVN (153 Mb), and the numbers
>were in different order (FPC "ball" was created in 2 minutes by Rar):
>
>Rar     21 Mb  5 minutes
>gzip    29 Mb  1 minute 30 seconds
>bzip2   20 Mb  3 minutes 40 seconds (is it optimized for pascal?)
>7z PPMd 17 Mb  8 minutes 20 seconds
>7z LZMA 16 Mb  25 minutes (well, it would be better, I believe it
>had memory problems)
>
>I tried PPMonster archiver (open source, from the author of PPMd
>algorithm), while sources are less than 60 Kb in C, speed is often
>about 20 Kb/s. However, it promised 0.548 bits per byte of the FPC
>"ball" (but >1.5 for OpenOffice).

Then try one of the last compressors as PAQ8, your OO 2.0.2
sources  will take days in compress, but i think you can get 140-150
MB or even less and is opensource too. But it isn't used out of
comp.compression world. Also, the WinRK3.03 is a lot faster and gets
a bit more compression, but only works on windows. The tar.gz of
FreeBSD (90MB decompressed) goes to 12 MB with medium compression,
but takes 20 min in a 2GHz Athlon. Check
http://www.maximumcompression.com/ for details and the newsgroup
comp.compression or datacompression.info for more info.

Just change the gzip to a better zip program for make zip files, as
7z, and you'll get an improvement from current compression, but the
bzip2 suggestion is very good, even FreeBSD has changed the tar.gz
system to tar.bz2 (tbz) and the difference is notable.


A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re[5]: a suggestion...

Пётр Косаревский
Correction: Rar produced 212 Mb on OO 2.0.2 sources, but using "force text compression" makes it produce 206 Mb with not really big time overhead.

> Then try one of the last compressors as PAQ8, your OO 2.0.2
> sources  will take days in compress, but i think you can get 140-150
> MB or even less and is opensource too. But it isn't used out of

First thing: I did neither think or believe, I just tested.

Well, my benchmarks concerned bearable timings and memory requirements: no more than two hours and 512 Mb. I didn't test archivers, which were not mentioned in the thread (exception: PPMonstr, which actually was only an example of compression world only compressor).

> comp.compression world. Also, the WinRK3.03 is a lot faster and gets
> a bit more compression, but only works on windows. The tar.gz of

All four archivers tested by me work under many systems (bzip2, gzip, rar, 7z). I believe, paq8h has only minor problems porting. Rar is commercial, but it has good GUI (well, I use it and I like it) and is supported and developed continuously (and my benchmarks showed, that its compression ratios under given circumstances are not so good: losing to 7z is due to "dictionary" size: rar uses no more than 128 Mb of memory, but losing to bzip2 on fpcSVN is freaky).

> http://www.maximumcompression.com/ for details and the newsgroup
> comp.compression or datacompression.info for more info.

Oh, you forgot www.compression.ru :)

Didn't you pay attention, that the these sites didn't test sources? (And www.compression.ru has really outdated benchmarks :( )

> Just change the gzip to a better zip program for make zip files, as
> 7z, and you'll get an improvement from current compression, but the

Gzip does not produce zip files. It just uses deflate algorithm.

> bzip2 suggestion is very good, even FreeBSD has changed the tar.gz
> system to tar.bz2 (tbz) and the difference is notable.

Yes. And it uses only about 8 Mb memory. (But my test showed, that it loses 8.5% on OO 2.0.2 sources to Rar. And more than 11.5% with "force text compression" in rar.)

WinRK is unacceptable, because it is not only commercial, but runs only under windows. While Rar offers command line support under several linuces and unrar support under many systems.
AND WHILE ITS COMPRESSION ENGINE MAY BE GOOD, it simply does not work well (e.g. 3.0.3 b2 tells, that trial period has expired right after installation). IT'S PROGRAMMED BADLY.

Paq8 may be good, because it's open source and free. (By the way, if I get it right and speak it out very roughly, it is PPMd based with some dictionary improvements and other tweaks added.) But it's incredibly slow (in PPMonstr's category).

I understand, that experimental archivers are sometimes optimized for Calgary Challenge (not because of luaghable money prizes, but because of some sort of fame), so it's not that great for compressing OO 2.0.2 or FPC sources.

Benchmarks addendum (FPCsvn):

WinRK --- requires tweaks to run and may crash at many moments (GUI problems?). Could it be useful, "High" profile would be applied (<512 Mb). And it would be too long (as 7z in LZMA mode).
Paq-8H (on a "rar ball"): too long...



Well, bzip2 or 7z do seem most efficient. Nothing has changed. Probably, this thread is useless: if the developers find some time, they will use bzip2 or something they like.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re[5]: a suggestion...

Michael Van Canneyt


On Thu, 1 Jun 2006, ???? ??????????? wrote:

> Correction: Rar produced 212 Mb on OO 2.0.2 sources, but using "force text compression" makes it produce 206 Mb with not really big time overhead.
>
>> Then try one of the last compressors as PAQ8, your OO 2.0.2
>> sources  will take days in compress, but i think you can get 140-150
>> MB or even less and is opensource too. But it isn't used out of
>
> First thing: I did neither think or believe, I just tested.

To save your time, you should limit your tests to:

1. Command-line tools.

    Creation of installs is automated.
    GUI tools cannot be used in automated builds.

2. Completely cross-platform.

    For obvious reasons.

3. Completely Open source.

    For obvious reasons.

4. Compress time should be definitely less than 10 minutes.

    Sometimes we do 4-5 builds one after the other.
    We simply cannot afford to wait 2 hours, and we never know
    whether the next build will be the final one.

Any solution that falls outside of this is not acceptable in advance.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
12