Best way to insert bytes into a TBytes variable?

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Best way to insert bytes into a TBytes variable?

Bo Berglund
I am working on moving away from using AnsiString as communications
buffer in an old application, so I need to write efficient
replacements for certain string functions (Delete, Insert, Copy etc).

Now I am wondering if there is a better way to do the Insert() command
than this:

procedure InsertBytes(var Dest: TBytes; Src: TBytes; Index: integer);
var
  Len, LenA: integer;
begin
  Len := Length(Dest);
  LenA := Length(Src);
  if LenA = 0 then exit; //Do nothing
  if Index > Len then Index := Len;  //Make an append instead
  SetLength(Dest, Len + LenA);
  Move(Dest[Index], Dest[Index + LenA], LenA);  //Create space
  Move(Src[0], Dest[Index], LenA);  //Insert data
end;

Ideally the function should be portable between FPC and Delphi XE5...


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Michael Van Canneyt


On Tue, 25 Jul 2017, Bo Berglund wrote:

> I am working on moving away from using AnsiString as communications
> buffer in an old application, so I need to write efficient
> replacements for certain string functions (Delete, Insert, Copy etc).
>
> Now I am wondering if there is a better way to do the Insert() command
> than this:
>
> procedure InsertBytes(var Dest: TBytes; Src: TBytes; Index: integer);
> var
>  Len, LenA: integer;
> begin
>  Len := Length(Dest);
>  LenA := Length(Src);
>  if LenA = 0 then exit; //Do nothing
>  if Index > Len then Index := Len;  //Make an append instead
>  SetLength(Dest, Len + LenA);
>  Move(Dest[Index], Dest[Index + LenA], LenA);  //Create space
>  Move(Src[0], Dest[Index], LenA);  //Insert data
> end;
>
> Ideally the function should be portable between FPC and Delphi XE5...

There is no better way if you only want to use TBytes.
If you want less reallocations, you must reserve more memory for the array
from the start (so length is a "capacity"), and keep a second count, the
count of the number of bytes  actually used.

Then you're maybe better off working with a TMemory stream, which does all this
for you. Or write a TBuffer class.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Bo Berglund
On Tue, 25 Jul 2017 11:15:31 +0200 (CEST), Michael Van Canneyt
<[hidden email]> wrote:

>There is no better way if you only want to use TBytes.

Thanks, then I will continue using Move...



--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Martok
In reply to this post by Bo Berglund
> Ideally the function should be portable between FPC and Delphi XE5...
You'd only need your own functions for Delphi, FPC's intrinsics such as Insert()
can already work with arrays:

var
  b, c: TBytes;
begin
  b:= TBytes.Create(1,2,3);
  c:= TBytes.Create(10,11);
  Insert(c,b,2);

-> b is now [1,2,10,11,3]


--
Martok

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Free Pascal - General mailing list

Am 26.07.2017 10:59 schrieb "Martok" <[hidden email]>:
>
> > Ideally the function should be portable between FPC and Delphi XE5...
> You'd only need your own functions for Delphi, FPC's intrinsics such as Insert()
> can already work with arrays:
>
> var
>   b, c: TBytes;
> begin
>   b:= TBytes.Create(1,2,3);
>   c:= TBytes.Create(10,11);
>   Insert(c,b,2);
>
> -> b is now [1,2,10,11,3]

But only in trunk. And Delphi XE8 and newer also support them.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

noreply
In reply to this post by Martok
On 2017-07-25 11:00, Martok wrote:

>> Ideally the function should be portable between FPC and Delphi XE5...
> You'd only need your own functions for Delphi, FPC's intrinsics such as
> Insert()
> can already work with arrays:
>
> var
>   b, c: TBytes;
> begin
>   b:= TBytes.Create(1,2,3);
>   c:= TBytes.Create(10,11);
>   Insert(c,b,2);
>
> -> b is now [1,2,10,11,3]
>
>

The fpc wiki should probably be updated?

I cannot find "insert" on this page:
http://wiki.freepascal.org/Array

I cannot find it here either:
https://www.freepascal.org/docs-html/ref/refsu14.html

For the last 10 years or so I've been criticizing occasionally the fact
that everyone reinvents their own array algorithms for each array! So it
seems that may solve it. In fact this problem goes back to the 1970's
and 1980's when arrays were being used.

Please list other functions other than insert that are available? Maybe
someone add it to the wiki/docs
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

noreply
In reply to this post by Free Pascal - General mailing list
On 2017-07-26 06:34, Sven Barth via fpc-pascal wrote:

> Am 26.07.2017 10:59 schrieb "Martok" <[hidden email]>:
>>
>> > Ideally the function should be portable between FPC and Delphi
> XE5...
>> You'd only need your own functions for Delphi, FPC's intrinsics such
> as Insert()
>> can already work with arrays:
>>
>> var
>>   b, c: TBytes;
>> begin
>>   b:= TBytes.Create(1,2,3);
>>   c:= TBytes.Create(10,11);
>>   Insert(c,b,2);
>>
>> -> b is now [1,2,10,11,3]
>
> But only in trunk. And Delphi XE8 and newer also support them.
>
> Regards,
> Sven

Oh, that would explain why it is not in the wiki/docs yet

Only issue I can see, is with all Create's, programmers expect a
matching Free.
So that is a bit confusing, but so long as someone remembers arrays are
garbage (reference rather) collected...
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Free Pascal - General mailing list
In reply to this post by noreply

Am 27.07.2017 17:18 schrieb <[hidden email]>:
>
> On 2017-07-25 11:00, Martok wrote:
>>>
>>> Ideally the function should be portable between FPC and Delphi XE5...
>>
>> You'd only need your own functions for Delphi, FPC's intrinsics such as Insert()
>> can already work with arrays:
>>
>> var
>>   b, c: TBytes;
>> begin
>>   b:= TBytes.Create(1,2,3);
>>   c:= TBytes.Create(10,11);
>>   Insert(c,b,2);
>>
>> -> b is now [1,2,10,11,3]
>>
>>
>
> The fpc wiki should probably be updated?
>
> I cannot find "insert" on this page:
> http://wiki.freepascal.org/Array

Then update it.

> I cannot find it here either:
> https://www.freepascal.org/docs-html/ref/refsu14.html

The documentation is always for the latest release (currently 3.0.2, soon 3.0.4), but only trunk supports this as well as Delete() (Concat() is not yet implemented however).

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Michael Schnell
In reply to this post by Martok

On 25.07.2017 18:00, Martok wrote:
> ... FPC's intrinsics such as Insert() can already work with arrays:
Nonetheless, IMHO using single-Byte Strings (UTF-8 or RawByte, as a
proper "uncoded" string type brand does not exist), seems more
convenient, especially, as here we have lazy copy on top of ref counting.

-Michael
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Cyrax
In reply to this post by noreply
On 27/07/17 18:17, [hidden email] wrote:

> On 2017-07-25 11:00, Martok wrote:
>>> Ideally the function should be portable between FPC and Delphi XE5...
>> You'd only need your own functions for Delphi, FPC's intrinsics such
>> as Insert()
>> can already work with arrays:
>>
>> var
>>   b, c: TBytes;
>> begin
>>   b:= TBytes.Create(1,2,3);
>>   c:= TBytes.Create(10,11);
>>   Insert(c,b,2);
>>
>> -> b is now [1,2,10,11,3]
>>
>>
>
> The fpc wiki should probably be updated?
>
> I cannot find "insert" on this page:
> http://wiki.freepascal.org/Array
>
> I cannot find it here either:
> https://www.freepascal.org/docs-html/ref/refsu14.html
>
> For the last 10 years or so I've been criticizing occasionally the fact
> that everyone reinvents their own array algorithms for each array! So it
> seems that may solve it. In fact this problem goes back to the 1970's
> and 1980's when arrays were being used.
>
> Please list other functions other than insert that are available? Maybe
> someone add it to the wiki/docs
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

AFAIK, that is avaible only in the trunk version of FPC. You need to
build the trunk version of the documents by yourself.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Bo Berglund
In reply to this post by Bo Berglund
On Tue, 25 Jul 2017 10:54:30 +0200, Bo Berglund
<[hidden email]> wrote:

>so I need to write efficient
>replacements for certain string functions (Delete, Insert, Copy etc).

What I am now up aginst after a few days work is to find how to code
the Pos() function for strings...
The comm buffer apparently contains the 2-char string as indication of
the last and the existing code uses Pos() to find it.

What about something like:
function PosBytes(SearchItem, Target: TBytes): integer;
begin
  ?????
end;

What could I use inside the body of this function to mimik the string
function Pos (for AnsiStrings)?


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Michael Schnell
In reply to this post by Bo Berglund
On 25.07.2017 10:54, Bo Berglund wrote:
>   so I need to write efficient replacements for certain string functions (Delete, Insert, Copy etc).
Why do you think the string function (if using strictly just a single
UTF-8 or RawByte branded String type) are not efficient ?

If all string encoding brands in an operation are the same there will be
no conversion.

-Michael
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Bo Berglund
On Fri, 28 Jul 2017 16:35:05 +0200, Michael Schnell
<[hidden email]> wrote:

>On 25.07.2017 10:54, Bo Berglund wrote:
>>   so I need to write efficient replacements for certain string functions (Delete, Insert, Copy etc).
>Why do you think the string function (if using strictly just a single
>UTF-8 or RawByte branded String type) are not efficient ?
>
>If all string encoding brands in an operation are the same there will be
>no conversion.

Sorry,
I was not clear in my question...

What I meant to say is that I need to convert an existing application
that uses strings (AnsiString) as binary data buffers to instead use
TBytes buffers. Thus my quest is for functions operating like the
convenient string functions but on TBytes arrays.

All of this is because I have found that using AnsiString is
triggering data changes when the application is running in certain
countries (locales) and processing certain data values...

The application was started back in Delphi7 times when "string"
actually meant AnsiString and was a 1-byte per element container.

By efficient I meant that I probably would write inefficient code if I
had to create things like the mentioned functions myself and if the
compiler did the job it would be more efficient.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Free Pascal - General mailing list

Am 30.07.2017 08:07 schrieb "Bo Berglund" <[hidden email]>:
>
> On Fri, 28 Jul 2017 16:35:05 +0200, Michael Schnell
> <[hidden email]> wrote:
>
> >On 25.07.2017 10:54, Bo Berglund wrote:
> >>   so I need to write efficient replacements for certain string functions (Delete, Insert, Copy etc).
> >Why do you think the string function (if using strictly just a single
> >UTF-8 or RawByte branded String type) are not efficient ?
> >
> >If all string encoding brands in an operation are the same there will be
> >no conversion.
>
> Sorry,
> I was not clear in my question...
>
> What I meant to say is that I need to convert an existing application
> that uses strings (AnsiString) as binary data buffers to instead use
> TBytes buffers. Thus my quest is for functions operating like the
> convenient string functions but on TBytes arrays.
>
> All of this is because I have found that using AnsiString is
> triggering data changes when the application is running in certain
> countries (locales) and processing certain data values...
>
> The application was started back in Delphi7 times when "string"
> actually meant AnsiString and was a 1-byte per element container.

You could always use RawByteString or a string with a fixed codepage instead if plain AnsiString.

Just declare a type alias so that you can keep it compatible with older Delphi versions as well.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Bo Berglund
On Sun, 30 Jul 2017 09:33:59 +0200, Sven Barth via fpc-pascal
<[hidden email]> wrote:

>> The application was started back in Delphi7 times when "string"
>> actually meant AnsiString and was a 1-byte per element container.
>
>You could always use RawByteString or a string with a fixed codepage
>instead if plain AnsiString.

I asked about this problem over at Embarcadero too, but was flamed for
even thinking about using any kind of string for storing binary data.
THeyn did suggest RawByteString too, though so I went over the
application changing all buffers to RawByteString and it did work.
I also tried to set a fixed codepage for the application, but it is
hard to check if it actually does help.

The root problem is what happens with string conversions on different
locales and some functions could not be easily modified to use
RawByteString.
So I decided to bite the bullet and convert the whole unit to TBytes
instead. Some 7000 code lines to go over...

It has taken a good many hours now and I still have some intricate
problems to solve. It still won't compile without errors.

>
>Just declare a type alias so that you can keep it compatible with older
>Delphi versions as well.
>

Do you mean:
type
  TMyString = RawByteString;


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Free Pascal - General mailing list

Am 30.07.2017 12:37 schrieb "Bo Berglund" <[hidden email]>:
>
> On Sun, 30 Jul 2017 09:33:59 +0200, Sven Barth via fpc-pascal
> <[hidden email]> wrote:
>
> >> The application was started back in Delphi7 times when "string"
> >> actually meant AnsiString and was a 1-byte per element container.
> >
> >You could always use RawByteString or a string with a fixed codepage
> >instead if plain AnsiString.
>
> I asked about this problem over at Embarcadero too, but was flamed for
> even thinking about using any kind of string for storing binary data.
> THeyn did suggest RawByteString too, though so I went over the
> application changing all buffers to RawByteString and it did work.
> I also tried to set a fixed codepage for the application, but it is
> hard to check if it actually does help.

Not a fixed codepage for the application, but a string with fixed codepage, e.g. "String(CP_1242)" or so...

> The root problem is what happens with string conversions on different
> locales and some functions could not be easily modified to use
> RawByteString.
> So I decided to bite the bullet and convert the whole unit to TBytes
> instead. Some 7000 code lines to go over...
>
> It has taken a good many hours now and I still have some intricate
> problems to solve. It still won't compile without errors.
>
> >
> >Just declare a type alias so that you can keep it compatible with older
> >Delphi versions as well.
> >
>
> Do you mean:
> type
>   TMyString = RawByteString;

Yes.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Michael Schnell
In reply to this post by Bo Berglund
On 30.07.2017 08:06, Bo Berglund wrote:
> All of this is because I have found that using AnsiString is
> triggering data changes when the application is running in
> certaincountries (locales) and processing certain data values...
I am rather sure that this can be avoided.

-Michael
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Michael Schnell
In reply to this post by Bo Berglund
On 30.07.2017 12:37, Bo Berglund wrote:
> I asked about this problem over at Embarcadero too, but was flamed
> foreven thinking about using any kind of string for storing binary data.
They are silly and defending their completely silly implementation of
Code aware strings, forcing UTF-16 for any TStrings based classes.

The fpc implementation (forcing UTF-8 for any TStrings based classes) is
slightly less silly.

-Michael
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Michael Van Canneyt


On Mon, 31 Jul 2017, Michael Schnell wrote:

> On 30.07.2017 12:37, Bo Berglund wrote:
>> I asked about this problem over at Embarcadero too, but was flamed
>> foreven thinking about using any kind of string for storing binary data.
> They are silly and defending their completely silly implementation of
> Code aware strings, forcing UTF-16 for any TStrings based classes.
>
> The fpc implementation (forcing UTF-8 for any TStrings based classes) is
> slightly less silly.

Small correction:

FPC "forces" nothing. It offers a single-byte string TStrings implementation.
If you want to use that for Unicode, you indeed have no choice but to use UTF8
or use/write a separate class that uses UTF16.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Best way to insert bytes into a TBytes variable?

Michael Schnell
On 02.08.2017 09:29, Michael Van Canneyt wrote:
>
>
> FPC "forces" nothing. It offers a single-byte string TStrings
> implementation.
> If you want to use that for Unicode, you indeed have no choice but to
> use UTF8
> or use/write a separate class that uses UTF16.

That is why I said that it is less silly ! :)

But it does force TStrings siblings to handle single-byte strings (While
Delphi forces TStrings siblings to handle 16-bit-word Strings).

As a huge count of classes in the RTL (and the LCL and MSEGUI and ...,
and in user code) are or use TStrings siblings the compatibility issues
and certain other problems are due to happen.

(I hope I will not completely banned now, as I already am under
monitoring for this kind of posts...)

-Michael
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
12
Loading...