old school crc32

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

old school crc32

wkitty42

this is driving me battier than i already am :(

i have a binary file that contains a record of data... the first field of the
data is a crc32 value of a string of characters... the string of characters is
lower cased before being fed to the crc32 function... i need to calculate the
same crc32 value as what is stored in the file...

i've already checked the polynomial ($edb88320) is the same in both, the
original implementation (converted to TP4 in 1988) and this implementation...

the original routine uses longint and speaks of initializing the variable to
$ffffffff before feeding it to the crc32 routine... the existing routine in FPC
says "Pre- and post-conditioning (one's complement) is performed within this
function so it shouldn't be done by the application." but that doesn't give me
the proper answer in this case... i've tried using cardinal types and longint
types to no avail...

the following is so so so very close but now i'm bald headed, completely lost in
the deep dark woods and can't find my breadcrumbs any more :(


program damncrc32;

{$MODE OBJFPC}
{$DEFINE DYNAMIC_CRC_TABLE}

uses
   crc, sysutils;

const
   mystring : pchar = 'aaaaaaaa aaaaaaaa';
   // desired crc32 checksum value 0xd728cba6
   // crc32 as seen in binary file 0xcba6d728
//  mystrsum : cardinal = 3609775014; // crc32 in decimal
   mystrsum : longint = -685192282; // crc32 in decimal

var
//  mycrc32 : cardinal;
   mycrc32 : longint;
   mystrln : longint;
   counter : longint;

begin
   // set the string length - subtract 1 later!
   mystrln := length(mystring);

   // prime the crc32 variable
   mycrc32 := crc32(0, nil, 0);
//  mycrc32 := $ffffffff;
//  mycrc32 := -1;

   // now calculate the crc32 of the string
   for counter := 0 to mystrln-1 do
     begin
       mycrc32 := crc32(mycrc32, @mystring[counter], 1);
       writeln('mystring[',inttostr(counter),'] : ',mystring[counter],'
',hexstr(mycrc32,8));
     end;

   // write the string and its length
   writeln('mystring : ',mystring,'  length : ',mystrln);

   // write the desired crc32 in decimal, hex and inverted hex
   writeln('  desired crc32 : ',mystrsum:12,'  (normal 0x',hexstr(mystrsum,8),'
- inverted 0x',hexstr(swap(mystrsum),8),')');

   // write the result crc32 in decimal, hex and inverted hex
   writeln('   result crc32 : ',mycrc32:12,'  (normal 0x',hexstr(mycrc32,8),' -
inverted 0x',hexstr(swap(mycrc32),8),')');

   // did we get it right???
   write('crc32 checksum');
   if mycrc32 <> mystrsum then
     writeln(' DOES NOT match.')
   else
     writeln(' matches! YAY!!!');
end.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

Mark Morgan Lloyd-5
[hidden email] wrote:

> this is driving me battier than i already am :(
>
> i have a binary file that contains a record of data... the first field
> of the data is a crc32 value of a string of characters... the string of
> characters is lower cased before being fed to the crc32 function... i
> need to calculate the same crc32 value as what is stored in the file...
>
> i've already checked the polynomial ($edb88320) is the same in both, the
> original implementation (converted to TP4 in 1988) and this
> implementation...
>
> the original routine uses longint and speaks of initializing the
> variable to $ffffffff before feeding it to the crc32 routine... the
> existing routine in FPC says "Pre- and post-conditioning (one's
> complement) is performed within this function so it shouldn't be done by
> the application." but that doesn't give me the proper answer in this
> case... i've tried using cardinal types and longint types to no avail...

Have you tried running your test case through a different
implementation? I use http://www.wolfgang-ehrhardt.de/crchash_en.html

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

Giuliano Colla
In reply to this post by wkitty42
Il 24/07/2016 01:18, [hidden email] ha scritto:

> this is driving me battier than i already am :(
>
> i have a binary file that contains a record of data... the first field
> of the data is a crc32 value of a string of characters... the string
> of characters is lower cased before being fed to the crc32 function...
> i need to calculate the same crc32 value as what is stored in the file...
>
> i've already checked the polynomial ($edb88320) is the same in both,
> the original implementation (converted to TP4 in 1988) and this
> implementation...
>
> the original routine uses longint and speaks of initializing the
> variable to $ffffffff before feeding it to the crc32 routine... the
> existing routine in FPC says "Pre- and post-conditioning (one's
> complement) is performed within this function so it shouldn't be done
> by the application." but that doesn't give me the proper answer in
> this case... i've tried using cardinal types and longint types to no
> avail...
>
> the following is so so so very close but now i'm bald headed,
> completely lost in the deep dark woods and can't find my breadcrumbs
> any more :(
>

What I'd do in your place would be to forget about strings, and use just
an array of bytes. This way you're sure that you don't have in the way
the string length value and some automatic encoding conversion, which
messes up things.

Giuliano

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

Tony Whyman
In reply to this post by wkitty42


On 24/07/16 00:18, [hidden email] wrote:
> i've already checked the polynomial ($edb88320) is the same in both,
> the original implementation (converted to TP4 in 1988) and this
> implementation...

Are you sure that you are dealing with a CRC algorithm? ISO 8073 TP4
uses an arithmetic checksum (Fletcher).

If you are using genuine CRC32 then

http://www.tty1.net/pycrc/

is a good reference for all the different variations. It will generate
'C' code for you but still a good baseline reference. I've used it
successfully for CRC implementations (some quite oddball in the aviation
sector).

Regards

Tony Whyman
MWA

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

geneb

I don't know if you've looked at SWAG yet, but there's a couple of 32 bit
CRC routines there.

See here: http://swag.delphidabbler.com/

g.
--
Proud owner of F-15C 80-0007
http://www.f15sim.com - The only one of its kind.
http://www.diy-cockpits.org/coll - Go Collimated or Go Home.
Some people collect things for a hobby.  Geeks collect hobbies.

ScarletDME - The red hot Data Management Environment
A Multi-Value database for the masses, not the classes.
http://scarlet.deltasoft.com - Get it _today_!
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

wkitty42
In reply to this post by Mark Morgan Lloyd-5
On 07/24/2016 05:03 AM, Mark Morgan Lloyd wrote:
> Have you tried running your test case through a different implementation? I
> use http://www.wolfgang-ehrhardt.de/crchash_en.html

i'm trying to use only what is available in FPC... i will take a look at others,
though...

[time passes]

ok, i looked at the one you recommended... the biggest plus is that it had a
self-test with a known crc32 for a given string... with that, i was able to
figure out what was going on...

the main thing is that i need the "inverted" crc32... in other words, i needed
the bytes all reversed so that i could visually match what i was seeing with
what i needed... i didn't realize that swap was only swapping the hi and lo
words of the longints... i had to specifically break that down and do it like
this...

   mystrsumi := swap(lo(mystrsum)) shl 16 + swap(hi(mystrsum));

that gave me the bytes in the same order i was seeing them in the binary file
and that told me that i had the proper result...

it also told me

1. i need to initialize the crc32 variable with longint($ffffffff).
2. feed each character to the routine instead of letting the routine walk the
string on its own.
3. do NOT xor the last value with longint($ffffffff).


i've found that i cannot use the crc32 function from the crc unit that comes
with FPC... if it is possible, i'm missing the very subtle difference... i
suspect that the automatic pre- and post-processing parts (items 1 and 3) are
getting in the way but it may also be how the routines walk the string if you
feed it all to them at once... even the crc32Update routine of wolfgang's
refuses to spit out the desired result if the whole string is given to it at
once instead of character by character...

thanks to everyone who offered input!

--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list* unless
        private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

wkitty42
In reply to this post by Giuliano Colla
On 07/24/2016 05:12 AM, Giuliano Colla wrote:
> What I'd do in your place would be to forget about strings, and use just an
> array of bytes.

that's why i'm using a pchar, starting at var[0] and length(var)-1 ;)

> This way you're sure that you don't have in the way the string length value
> and some automatic encoding conversion, which messes up things.

i can easily see how those would get in the way... in this case, there's none of
that fancyschmancy stuff being done ;)


--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list* unless
        private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

wkitty42
In reply to this post by Tony Whyman
On 07/24/2016 06:41 AM, Tony Whyman wrote:
> On 24/07/16 00:18, [hidden email] wrote:
>> i've already checked the polynomial ($edb88320) is the same in both, the
>> original implementation (converted to TP4 in 1988) and this implementation...
>
> Are you sure that you are dealing with a CRC algorithm? ISO 8073 TP4 uses an
> arithmetic checksum (Fletcher).

ahhh... TP4 in this case is Turbo Pascal v4... i've had a lot of code for a very
long time... especially code that was posted to the PASCAL and PASCAL_LESSONS
echos on Fidonet ;)

> If you are using genuine CRC32 then
>
> http://www.tty1.net/pycrc/
>
> is a good reference for all the different variations. It will generate 'C' code
> for you but still a good baseline reference. I've used it successfully for CRC
> implementations (some quite oddball in the aviation sector).

thanks for the info! if you've read my other posts by now, you'll see that the
function in FPC's crc unit does some things automatically that blow our desired
operation out of the water... we have to initialize the crc variable ourselves,
feed each character individually to the crc routine and then leave off the
finalization which flips all the bits over... it comes down to needing the
result in LSB to MSB format like was used with analogue UARTS which feed the
values to the serial port in reverse...

--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list* unless
        private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

wkitty42
In reply to this post by geneb
On 07/24/2016 10:15 AM, geneb wrote:
>
> I don't know if you've looked at SWAG yet, but there's a couple of 32 bit
> CRC routines there.

hey gene! fancy seeing you here instead of the flightgear areas ;) ;) ;)

i do have all of SWAG here... back in the day, we grabbed each new release as it
was available from Fidonet... we've even had some of our stuff published in SWAG
back in the day :)

--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list* unless
        private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

Mark Morgan Lloyd-5
In reply to this post by wkitty42
[hidden email] wrote:
> On 07/24/2016 05:03 AM, Mark Morgan Lloyd wrote:
>> Have you tried running your test case through a different
>> implementation? I
>> use http://www.wolfgang-ehrhardt.de/crchash_en.html
>
> i'm trying to use only what is available in FPC... i will take a look at
> others, though...

Bug 23472 on Mantis suggests adding that library as a package or whatever.

> [time passes]
>
> ok, i looked at the one you recommended... the biggest plus is that it
> had a self-test with a known crc32 for a given string... with that, i
> was able to figure out what was going on...
>
> the main thing is that i need the "inverted" crc32... in other words, i
> needed the bytes all reversed so that i could visually match what i was
> seeing with what i needed... i didn't realize that swap was only
> swapping the hi and lo words of the longints... i had to specifically
> break that down and do it like this...

Same old :-)

I'm using that library (plus at least one other) in a program to drive
an HP495x protocol analyser, so that it can look at a block of data and
generate "all known" checksums for investigation.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

geneb
In reply to this post by wkitty42
On Sun, 24 Jul 2016, [hidden email] wrote:

> On 07/24/2016 10:15 AM, geneb wrote:
>>
>> I don't know if you've looked at SWAG yet, but there's a couple of 32 bit
>> CRC routines there.
>
> hey gene! fancy seeing you here instead of the flightgear areas ;) ;) ;)
>
> i do have all of SWAG here... back in the day, we grabbed each new release as
> it was available from Fidonet... we've even had some of our stuff published
> in SWAG back in the day :)

Excellent. :)  Do you happen to have archives of the PASCAL and
PASCAL_LESSONS echos by chance?

g.

--
Proud owner of F-15C 80-0007
http://www.f15sim.com - The only one of its kind.
http://www.diy-cockpits.org/coll - Go Collimated or Go Home.
Some people collect things for a hobby.  Geeks collect hobbies.

ScarletDME - The red hot Data Management Environment
A Multi-Value database for the masses, not the classes.
http://scarlet.deltasoft.com - Get it _today_!
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: old school crc32

wkitty42
On 07/25/2016 11:22 AM, geneb wrote:

> On Sun, 24 Jul 2016, [hidden email] wrote:
>> On 07/24/2016 10:15 AM, geneb wrote:
>>> I don't know if you've looked at SWAG yet, but there's a couple of 32
>>> bit CRC routines there.
>>
>> hey gene! fancy seeing you here instead of the flightgear areas ;) ;) ;)
>>
>> i do have all of SWAG here... back in the day, we grabbed each new release
>> as it was available from Fidonet... we've even had some of our stuff
>> published in SWAG back in the day :)
>
> Excellent. :)  Do you happen to have archives of the PASCAL and PASCAL_LESSONS
> echos by chance?

sorry, i do not... my system only keeps the last three years of posts based on
their date... i wish that i had set numerous echomail areas to no purging but
back then (early '80s) the message base formats were rather limited and blew up
spectacularly when their limits were exceeded (think hudson message base)...
when i was able to switch to another message base format, much of the
interesting stuff was already long gone :(


--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list* unless
        private contact is specifically requested and granted.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal