Endian issues when porting FPC programs from x86 to other platforms

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Endian issues when porting FPC programs from x86 to other platforms

Bo Berglund
I am writing a serial communications program, which will connect to a
data acquisition instrument and download data. I am using FPC/Lazarus
in order to make the program platform independent so it can be
deployed also on an embedded Linux system even though it is developed
on Windows or Ubuntu on an x86 machine.

My problem is that the data downloaded are basically memory dumps of
an array of measure point records where the record contains various
data types including single precision floats, multi-byte integers and
character arrays. The instrument contains a Motorola microcontroller
so its endian is opposite that of the typical Windows PC.
When I wrote my Windows only program in Delphi 7 I had to introduce
swap functions to correct the endian of the various values after
downloading the memory dump. This worked fine but was a bit tedious to
do....

Now, however, the problem may be complicated because the program may
well run on a CPU that is using the *same* endian structure as the
original instrument in which case of course the swaps should not be
done.
So I need some detector or other to decide if swaps are needed.
Is there some commonly used function to handle this or is there some
conditional directive one can use to control the compiler to "do the
right thing" depending on the target CPU endian?

Or maybe this is a moot question if FPC is only so far running on
platforms that have the same endian as x86 CPU:s? I have no
information on this though...

I can see at least these targets:
- Windows and Linux on x86 CPU:s
- Embedded Linux on ARM CPU:s


Bo Berglund

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Marco van de Voort
In our previous episode, Bo Berglund said:
> Now, however, the problem may be complicated because the program may
> well run on a CPU that is using the *same* endian structure as the
> original instrument in which case of course the swaps should not be
> done.

Correct.

> So I need some detector or other to decide if swaps are needed.
> Is there some commonly used function to handle this or is there some
> conditional directive one can use to control the compiler to "do the
> right thing" depending on the target CPU endian?

See e.g. the CHM package. It has the same problem, the file structure is
little endian (since windows), but the package may run on e.g. PowerPC.

The routine in the url below is used for that. (in this case ntole, but
ntobe if your storage is bigendian)

http://www.freepascal.org/docs-html/rtl/system/ntole.html
http://www.freepascal.org/docs-html/rtl/system/ntobe.html

These routines are afaik inline and a no-op if the endianness is the same.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Thomas Schatzl
In reply to this post by Bo Berglund
Hi,

> Or maybe this is a moot question if FPC is only so far running on
> platforms that have the same endian as x86 CPU:s? I have no
> information on this though...
>
> I can see at least these targets:
> - Windows and Linux on x86 CPU:s
> - Embedded Linux on ARM CPU:s

Sparc, PowerPC are big endian, there is an ARM big endian target too.

Use BEtoN, NtoBE if the source is known to be big endian.

(http://www.freepascal.org/docs-html/rtl/system/beton.html)
(http://www.freepascal.org/docs-html/rtl/system/ntobe.html)

Hth,
  Thomas

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Mark Morgan Lloyd-5
Thomas Schatzl wrote:

> Hi,
>
>> Or maybe this is a moot question if FPC is only so far running on
>> platforms that have the same endian as x86 CPU:s? I have no
>> information on this though...
>>
>> I can see at least these targets:
>> - Windows and Linux on x86 CPU:s
>> - Embedded Linux on ARM CPU:s
>
> Sparc, PowerPC are big endian, there is an ARM big endian target too.
>
> Use BEtoN, NtoBE if the source is known to be big endian.
>
> (http://www.freepascal.org/docs-html/rtl/system/beton.html)
> (http://www.freepascal.org/docs-html/rtl/system/ntobe.html)

All good clean fun. See ENDIAN_LITTLE and ENDIAN_BIG in the FPC
Programmers' Manual, and also watch out for alignment issues.

I'd point out that if the embedded system is based on a Motorola chip
and that if this defines that the format transferred across the link is
big-endian, that this is basically the same as internet "network order"
and a convenient hack is to use the endian-conversion functions
associated with socket programming, i.e. HtoNL() etc.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Bo Berglund
In reply to this post by Bo Berglund
On Tue, 02 Nov 2010 09:09:05 +0100, Bo Berglund
<[hidden email]> wrote:

OK, I see that there are functions to convert a known BE or LE integer
value to the *native* order. This would in principle work across all
platforms where FPC is implemented so that is good news!

However now I have the semantics to deal with because of the way
people talk about big-endian and little-endian makes me think they are
opposite to my thinking...

So let me make a short example:
Assume a 4-byte integer value consisting of the B1, B2, B3 and B3
bytes where B1 is the MSB byte and B3 is the LSB byte.
They are arranged in memory at increasing addresses as:
B1 B2 B3 B4

Now comparing to how one writes a number 1234, this would look the
same i.e. the MSB comes first and the LSB last.

The end to me is the last (the right-most) byte. And that is the LSB.
So the end is LSB, the little part...

Therefore I would imagine that this is a LITTLE endian storage but is
it really what is the common definition?

And what happens when the memory itself holds 32 bits (or more) in
each address? Then everything is stuffed into the same single address
and the above example will fail...


Bo Berglund

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Jonas Maebe-2

On 02 Nov 2010, at 12:41, Bo Berglund wrote:

> The end to me is the last (the right-most) byte. And that is the LSB.
> So the end is LSB, the little part...

"Little endian" means "the little end comes first" (with "the little end" referring to the least significant byte). See http://en.wikipedia.org/wiki/Endianness#Etymology for the origins of the term.

> And what happens when the memory itself holds 32 bits (or more) in
> each address? Then everything is stuffed into the same single address
> and the above example will fail...

Endianess is defined in terms of byte-addressable memory. Software running on such an architecture would have to emulate the behaviour of a little or big endian byte-addressable architecture when exchanging data with the outside world.


Jonas_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Bo Berglund
In reply to this post by Thomas Schatzl
On Tue, 02 Nov 2010 09:32:36 +0100, Thomas Schatzl
<[hidden email]> wrote:

>Hi,
>
>> Or maybe this is a moot question if FPC is only so far running on
>> platforms that have the same endian as x86 CPU:s? I have no
>> information on this though...
>>
>> I can see at least these targets:
>> - Windows and Linux on x86 CPU:s
>> - Embedded Linux on ARM CPU:s
>
>Sparc, PowerPC are big endian, there is an ARM big endian target too.
>
>Use BEtoN, NtoBE if the source is known to be big endian.
>
>(http://www.freepascal.org/docs-html/rtl/system/beton.html)
>(http://www.freepascal.org/docs-html/rtl/system/ntobe.html)
>

Thanks, that helps a lot! Are there also overloaded BEtoN functions
for floating point values?
Like so:

function BEtoN(const AValue: Single):Single;

function BEtoN(const AValue: Double):Double;



--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Felipe Monteiro de Carvalho
On Tue, Nov 2, 2010 at 7:04 PM, Bo Berglund <[hidden email]> wrote:
> Thanks, that helps a lot! Are there also overloaded BEtoN functions
> for floating point values?

I think that single and double have always the same binary layout.

--
Felipe Monteiro de Carvalho
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Vinzent Höfler
In reply to this post by Bo Berglund
On Tue, 02 Nov 2010 21:39:31 +0100, Felipe Monteiro de Carvalho  
<[hidden email]> wrote:

> On Tue, Nov 2, 2010 at 7:04 PM, Bo Berglund <[hidden email]>  
> wrote:
>> Thanks, that helps a lot! Are there also overloaded BEtoN functions
>> for floating point values?
>
> I think that single and double have always the same binary layout.

Mostly, I'd guess. But I wouldn't count on that.


Vinzent.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Jonas Maebe-2
In reply to this post by Felipe Monteiro de Carvalho

On 02 Nov 2010, at 21:39, Felipe Monteiro de Carvalho wrote:

On Tue, Nov 2, 2010 at 7:04 PM, Bo Berglund <[hidden email]> wrote:
Thanks, that helps a lot! Are there also overloaded BEtoN functions
for floating point values?

I think that single and double have always the same binary layout.

That's incorrect, their endianess changes in the same way as that of integer data (except on ARM when using the FPA floating point unit, which uses its own special byte ordering).

There are no overloads of BEtoN() etc for floating point types in the system unit. You can convert them by putting them in a variant record with an overlaid floating point and integer field of the same size, and then swapping the integer field.


Jonas

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Bo Berglund
In reply to this post by Vinzent Höfler
On Tue, 02 Nov 2010 22:53:21 +0100, Vinzent Höfler
<[hidden email]> wrote:

>On Tue, 02 Nov 2010 21:39:31 +0100, Felipe Monteiro de Carvalho  
><[hidden email]> wrote:
>
>> On Tue, Nov 2, 2010 at 7:04 PM, Bo Berglund <[hidden email]>  
>> wrote:
>>> Thanks, that helps a lot! Are there also overloaded BEtoN functions
>>> for floating point values?
>>
>> I think that single and double have always the same binary layout.
>
>Mostly, I'd guess. But I wouldn't count on that.
>
I know for sure that on the Motorola system the floats are 4 bytes
(single precision) and have to be swapped in order to be used on
Windows. I have done it on Delphi already so I know this is a fact.
So now I need to find an endian corrector working with floating point
values as well...


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Bernd Mueller-3
In reply to this post by Jonas Maebe-2
Jonas Maebe wrote:
> On 02 Nov 2010, at 21:39, Felipe Monteiro de Carvalho wrote:
>
>> On Tue, Nov 2, 2010 at 7:04 PM, Bo Berglund <[hidden email]> wrote:
>>> Thanks, that helps a lot! Are there also overloaded BEtoN functions
>>> for floating point values?
>> I think that single and double have always the same binary layout.
>
> That's incorrect, their endianess changes in the same way as that of integer data (except on ARM when using the FPA floating point unit, which uses its own special byte ordering).

on ARM/OABI with double precision Netwinder FPE it is little endian with
low and high dwords swapped.

Regards, Bernd.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Thomas Schatzl
In reply to this post by Bo Berglund
Hi,

On Tue, 02 Nov 2010 23:49:03 +0100, Bo Berglund <[hidden email]>
wrote:

> On Tue, 02 Nov 2010 22:53:21 +0100, Vinzent Höfler
> <[hidden email]> wrote:
>
>>On Tue, 02 Nov 2010 21:39:31 +0100, Felipe Monteiro de Carvalho  
>><[hidden email]> wrote:
>>
>>> On Tue, Nov 2, 2010 at 7:04 PM, Bo Berglund <[hidden email]>  
>>> wrote:
>>>> Thanks, that helps a lot! Are there also overloaded BEtoN functions
>>>> for floating point values?
>>>
>>> I think that single and double have always the same binary layout.
>>
>>Mostly, I'd guess. But I wouldn't count on that.
>>
> I know for sure that on the Motorola system the floats are 4 bytes
> (single precision) and have to be swapped in order to be used on
> Windows. I have done it on Delphi already so I know this is a fact.
> So now I need to find an endian corrector working with floating point
> values as well...

Coming back to the suggestion from jonas:

function BEtoN(const AValue : single) : single;
type
  TData = packed record
    case integer of
      0 : (s : single);
      1 : (l : longint);
  end;
var
  d : TData;
begin
  d.s := AValue;
  d.l := system.BEtoN(d.l);
  result := d.s;
end;

A BEtoN function for doubles is similar, just replace the single type by
double, and longint by int64.

Hth,
  Thomas

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Bo Berglund
On Wed, 03 Nov 2010 10:50:10 +0100, Thomas Schatzl
<[hidden email]> wrote:

>Coming back to the suggestion from jonas:
>
>function BEtoN(const AValue : single) : single;
>type
>  TData = packed record
>    case integer of
>      0 : (s : single);
>      1 : (l : longint);
>  end;
>var
>  d : TData;
>begin
>  d.s := AValue;
>  d.l := system.BEtoN(d.l);
>  result := d.s;
>end;
>
>A BEtoN function for doubles is similar, just replace the single type by
>double, and longint by int64.
>

Hmm,
what would happen if I do this instead:

function BEtoN(const AValue: single): single;
begin
  result := single(BEtoN(longint(AValue)));
end;

I.e. I use typecasting to get the compiler to treat the same 4 bytes
in different ways inside the expression....
Is this legal in FPC?
(I can't test now because I am at work with no FPC system installed)

--

Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Endian issues when porting FPC programs from x86 to other platforms

Bo Berglund
On Fri, 05 Nov 2010 11:07:55 +0100, Bo Berglund
<[hidden email]> wrote:

>On Wed, 03 Nov 2010 10:50:10 +0100, Thomas Schatzl
><[hidden email]> wrote:
>
>>Coming back to the suggestion from jonas:
>>
>>function BEtoN(const AValue : single) : single;
>>type
>>  TData = packed record
>>    case integer of
>>      0 : (s : single);
>>      1 : (l : longint);
>>  end;
>>var
>>  d : TData;
>>begin
>>  d.s := AValue;
>>  d.l := system.BEtoN(d.l);
>>  result := d.s;
>>end;
>>
>>A BEtoN function for doubles is similar, just replace the single type by
>>double, and longint by int64.
>>
>
>Hmm,
>what would happen if I do this instead:
>
>function BEtoN(const AValue: single): single;
>begin
>  result := single(BEtoN(longint(AValue)));
>end;
>
>I.e. I use typecasting to get the compiler to treat the same 4 bytes
>in different ways inside the expression....
>Is this legal in FPC?
>(I can't test now because I am at work with no FPC system installed)

Now home and I put this function into a unit I am building and did a
test compile. No errors found...
But when I actually used it to convert a single value I got a runtime
exception...
The suggested function above, however, works fine! :-)
So I will use that to handle the conversions in my program.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal