Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
I have created a simple application which relays data from an
application I am debugging in Windows7 to/from a remotely located
embedded data system.
I use the built-in fpc Serial unit to handle the serial data and an
Indy10 TIdTcpClient for the network part.
Data coming in on the serial port are mirrored out to the remote TCP
server and data returned from that server are mirrored to the serial
port.
The Windows application only uses RS232 for this type of
communication.
At the remote location I have a Raspberry Pi uint where I have
installed and configured ser2net to do the same job in the remote
location.

My relaying application displays the number of bytes received and
transmitted on the interfaces so I can see what is going on.

This scheme has worked fine for most commands and data transfers I
have checked, but now I have run against a brick wall...

There is a pair of commands designed to read and write a large section
of the system CMOS RAM memory (where the data file system resides).
When I try to write a 1 Mbytes big buffer the byte count in my relayer
does not reach the correct number. The Windows application I am
debugging sends all of the bytes out the serial port (I have logged
this), but the relayer seems to lose some data and therefore the
transfer fails. The binary protocol specifies at the start how many
bytes are to be transferred (0x0FF000 or 1044486 decimal), then it
sends the data followed by a two-byte checksum. The data system shall
respond with NAK or ACK depending on the outcome of the checksum
verification.
The problem is that when the Windows app is done sending the data
system is still missing many kilobytes....
So no ACK is returned, it is still in receive mode.

Now I am looking at the Serial unit in order to figure out how buffer
sizes influence the performance. Apparently both Tx and Rx buffers are
set (hardcoded) to 2048, which feels like a bit low to me.

Is there some hidden property that makes it possible to increase this
value?
What is the maximum size one can set it too?


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Mark Morgan Lloyd-5
On 06/09/17 18:45, Bo Berglund wrote:
> I have created a simple application which relays data from anapplication I am debugging in Windows7 to/from a remotely locatedembedded data system.I use the built-in fpc Serial unit to handle the serial data and anIndy10 TIdTcpClient for the network part.Data coming in on the serial port are mirrored out to the remote TCPserver and data returned from that server are mirrored to the serialport.The Windows application only uses RS232 for this type ofcommunication.At the remote location I have a Raspberry Pi uint where I haveinstalled and configured ser2net to do the same job in the remotelocation.
> My relaying application displays the number of bytes received andtransmitted on the interfaces so I can see what is going on.
> This scheme has worked fine for most commands and data transfers Ihave checked, but now I have run against a brick wall...
> There is a pair of commands designed to read and write a large sectionof the system CMOS RAM memory (where the data file system resides).When I try to write a 1 Mbytes big buffer the byte count in my relayerdoes not reach the correct number. The Windows application I amdebugging sends all of the bytes out the serial port (I have loggedthis), but the relayer seems to lose some data and therefore thetransfer fails. The binary protocol specifies at the start how manybytes are to be transferred (0x0FF000 or 1044486 decimal), then itsends the data followed by a two-byte checksum. The data system shallrespond with NAK or ACK depending on the outcome of the checksumverification.The problem is that when the Windows app is done sending the datasystem is still missing many kilobytes....So no ACK is returned, it is still in receive mode.
> Now I am looking at the Serial unit in order to figure out how buffersizes influence the performance. Apparently both Tx and Rx buffers areset (hardcoded) to 2048, which feels like a bit low to me.
> Is there some hidden property that makes it possible to increase thisvalue?What is the maximum size one can set it too?

Not that I was responsible for. Refer to the Windows API for any limits.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
On Wed, 6 Sep 2017 18:58:57 +0000, Mark Morgan Lloyd
<[hidden email]> wrote:

>> Is there some hidden property that makes it possible to increase thisvalue?
>>What is the maximum size one can set it too?
>
>Not that I was responsible for. Refer to the Windows API for any limits.

I might add that the problem manifests itself on transmission from the
Windows application (Delphi XE5).
I.e. the relaying Lazarus/FPC application loses incoming bytes after
some time of processing the streams. Small data sizes work just fine
but with the 1 Mbytes packet some kbytes are lost.


And here is the core procedure where the relaying happens:

procedure TfrmMain.RelayData;
{This procedure is responsible for transferring the data from one side
to the other.
It will run until one of the ports are closed as indicated by the
flags FTcpConnected
and FComOpen}
var
  Buf: TIdBytes;
  LenRd, BufLen: integer;
  RxSer, TxSer, RXTcp, TxTcp: integer;
begin
  RxSer := 0;
  TxSer := 0;
  RxTcp := 0;
  TxTcp := 0;
  BufLen := 50;
  FTcpComm.ReadTimeout := 30;
  while (FComOpen and FTcpConnected and FRelayOn) do
  begin
    //First check TCP data
    try
      SetLength(Buf, 0);
      if FTcpComm.IOHandler.CheckForDataOnSource(10) then
      begin
        FTcpComm.IOHandler.ReadBytes(Buf, -1, false);
        if Length(Buf) > 0 then
        begin
          Inc(RxTcp, Length(Buf));
          SerWrite(FComH, Buf[0], Length(Buf));
          Inc(TxSer, Length(Buf));
          LogHex('Rx', Buf);
          SetLength(Buf, 0);
        end;
      end;
    except
      on E: Exception do
         Showmessage('TCP read exception: '#13 + E.Message);
    end;

    //Next check serial data
    SetLength(Buf, BufLen);
    LenRd := SerRead(FComH, Buf[0], BufLen);
    if LenRd > 0 then
    begin
      SetLength(Buf, LenRd);
      Inc(RxSer, LenRd);
      try
        FTcpComm.IOHandler.Write(Buf, LenRd);
        Inc(TxTcp, LenRd);
        LogHex('Tx', Buf);
        SetLength(Buf, 0);
      except
        on E: Exception do
           Showmessage('TCP write exception: '#13 + E.Message);
      end;
    end;
    //Finally show data and check events
    stxSerRx.Caption := IntToStr(RxSer);
    stxSerTx.Caption := IntToStr(TxSer);
    stxTcpRx.Caption := IntToStr(RxTcp);
    stxTcpTx.Caption := IntToStr(TxTcp);
    Application.ProcessMessages;
  end;
end;

Where could I be losing incoming serial data?


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Marc Santhoff-2
On Mi, 2017-09-06 at 21:22 +0200, Bo Berglund wrote:
> Where could I be losing incoming serial data?

Only for completeness:

Can you rule out problems reagrding the hardware side? Mabe there are
some weak line drivers or a faulty cable involved?

Worth checking, at least.

HTH anyhow,
Marc


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
On Wed, 06 Sep 2017 22:27:49 +0200, Marc Santhoff
<[hidden email]> wrote:

>On Mi, 2017-09-06 at 21:22 +0200, Bo Berglund wrote:
>> Where could I be losing incoming serial data?
>
>Only for completeness:
>
>Can you rule out problems reagrding the hardware side? Mabe there are
>some weak line drivers or a faulty cable involved?
>
>Worth checking, at least.
>
Well,
I am using a 4-way USB to Serial unit and the cable (nullmodem)
connects two adjacent ports and is about 50 mm long...
This cable has never failed ever before.

I have now added extra binary logging to the relaying program only to
find that the data received and processed are missing about 50 k of
the 1M transmission.
Next I will edit serial.pp and up the buffers from 2K to 60K or so.

--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
On Wed, 06 Sep 2017 22:52:12 +0200, Bo Berglund
<[hidden email]> wrote:

>Next I will edit serial.pp and up the buffers from 2K to 60K or so.

Question:
If I edit the content of serial.pp (the one that is brought up by
right clicking and "Find declaration") such that it looks like this:

const   bufSize= 61440; //2048;

will it be used the next time I test run my application or must I do
some kind of "installation" like updating Lazarus or similar for it to
be used?


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
In reply to this post by Bo Berglund
On Wed, 06 Sep 2017 22:52:12 +0200, Bo Berglund
<[hidden email]> wrote:

>I have now added extra binary logging to the relaying program only to
>find that the data received and processed are missing about 50 k of
>the 1M transmission.
>Next I will edit serial.pp and up the buffers from 2K to 60K or so.

By the way:
I have sent the Delphi test program I am debugging over to people at
the remote location and they have connected their laptop directly to
the data system and with that setup there is no data loss.
So this leaves me with only the Lazarus/FPC network relaying app to
fix...

I could also create another test program, a serial sniffer that could
be set up to just log whatever appears on the serial lines. I need a
3-way cable for that of course.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Marc Santhoff-2
On Mi, 2017-09-06 at 23:06 +0200, Bo Berglund wrote:

> On Wed, 06 Sep 2017 22:52:12 +0200, Bo Berglund
> <[hidden email]> wrote:
>
> >I have now added extra binary logging to the relaying program only to
> >find that the data received and processed are missing about 50 k of
> >the 1M transmission.
> >Next I will edit serial.pp and up the buffers from 2K to 60K or so.
>
> By the way:
> I have sent the Delphi test program I am debugging over to people at
> the remote location and they have connected their laptop directly to
> the data system and with that setup there is no data loss.
> So this leaves me with only the Lazarus/FPC network relaying app to
> fix...

Could that be some sort of timeout?

You're setting

  FTcpComm.ReadTimeout := 30;

but I have no idea what order of magnitude this 30 is. Maybe streching
the timeout a little bit can help?

I had several cases of timeout problems using RS232 over TCP.

> I could also create another test program, a serial sniffer that could
> be set up to just log whatever appears on the serial lines. I need a
> 3-way cable for that of course.

Such thing does exists, I only forgot the name(s), ;)

HTH,
Marc


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

José Mejuto
In reply to this post by Bo Berglund
El 06/09/2017 a las 21:22, Bo Berglund escribió:
>            Inc(RxTcp, Length(Buf));
>            SerWrite(FComH, Buf[0], Length(Buf));
>            Inc(TxSer, Length(Buf));
> Where could I be losing incoming serial data?

Hello,

Where do you check that "Length(Buf)" has been sent ?

if SerWrite(FComH, Buf[0], Length(Buf))<>Length(Buf) then Raise
Exception.Create("KBOOM");

--

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Paul Breneman
In reply to this post by Marc Santhoff-2
On 09/06/2017 05:21 PM, Marc Santhoff wrote:

> On Mi, 2017-09-06 at 23:06 +0200, Bo Berglund wrote:
>> On Wed, 06 Sep 2017 22:52:12 +0200, Bo Berglund
>> <[hidden email]> wrote:
>>
>>> I have now added extra binary logging to the relaying program only to
>>> find that the data received and processed are missing about 50 k of
>>> the 1M transmission.
>>> Next I will edit serial.pp and up the buffers from 2K to 60K or so.
>>
>> By the way:
>> I have sent the Delphi test program I am debugging over to people at
>> the remote location and they have connected their laptop directly to
>> the data system and with that setup there is no data loss.
>> So this leaves me with only the Lazarus/FPC network relaying app to
>> fix...
>
> Could that be some sort of timeout?
>
> You're setting
>
>    FTcpComm.ReadTimeout := 30;
>
> but I have no idea what order of magnitude this 30 is. Maybe streching
> the timeout a little bit can help?
>
> I had several cases of timeout problems using RS232 over TCP.
>
>> I could also create another test program, a serial sniffer that could
>> be set up to just log whatever appears on the serial lines. I need a
>> 3-way cable for that of course.
>
> Such thing does exists, I only forgot the name(s), ;)
>
> HTH,
> Marc


There are two links in the bottom paragraph on this page:
http://ctrlterm.com/custom.htm

Regards,
Paul
www.ControlPascal.com
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Mark Morgan Lloyd-5
In reply to this post by Bo Berglund
On 06/09/17 21:15, Bo Berglund wrote:

> I could also create another test program, a serial sniffer that couldbe set up to just log whatever appears on the serial lines. I need a3-way cable for that of course.

I'd also suggest that that sort of thing works better with on-board or
at least PCI-connected serial ports rather than relying on USB-connected
ports.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Mark Morgan Lloyd-5
In reply to this post by Bo Berglund
On 06/09/17 19:30, Bo Berglund wrote:

> Where could I be losing incoming serial data?

FTcpComm.IOHandler.ReadBytes(Buf, -1, false);
..
   SerWrite(FComH, Buf[0], Length(Buf));

SerWrite() returns the actual number of bytes it's written, which could
be anywhere from the number passed as the parameter down to zero (if
hardware handshaking's being used at a lower level). An intermediate
value would indicate that a buffer's full somewhere, and that you should
adjust your source address and length and have another shot.

TCP is a stream protocol and you can't make any assumptions about the
amount of data you get in one call. It's not reasonable to assume that
buffers for serial comms are limitless, 2K is entirely reasonable to
support most serial comms protocols or an 80x25 screen.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
In reply to this post by Marc Santhoff-2
On Wed, 06 Sep 2017 23:21:17 +0200, Marc Santhoff
<[hidden email]> wrote:

>Could that be some sort of timeout?
>
>You're setting
>
>  FTcpComm.ReadTimeout := 30;
>
>but I have no idea what order of magnitude this 30 is. Maybe streching
>the timeout a little bit can help?
>
>I had several cases of timeout problems using RS232 over TCP.

No, the Tcp/IP part is not involved at all here. When the transfer to
the server starts there is no response expected until all data have
been transferred.
The problem is that the Win7 application sends RS232 data out a serial
port and another application (my relayer) is using another serial port
which is connected by wire to the first serial port.
This relaying application seems to lose incoming data, which are sent
at a speed of 38400 baud.
Before the data arrives to the TCP/IP part they have been corrupted.
The test on location proves that the case where the relayer is not
involved (wire goes from Win7 application directly to the data system)
is working fine.
But I cannot use that because I am in Sweden and the data system is in
Texas...

Also,
before I wrote my own relaying application in Lazarus I tried to find
something for Windows on-line like ser2net for Linux, which I use in
the receiving end on a Raspberry Pi. But I had no success.
The connection to the remote system goes via a VPN tunnel from my Win7
computer so I cannot use a Raspberry Pi in my end to handle the
relaying. It does not see the remote network...


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
In reply to this post by José Mejuto
On Thu, 7 Sep 2017 01:54:18 +0200, José Mejuto
<[hidden email]> wrote:

>El 06/09/2017 a las 21:22, Bo Berglund escribió:
>>            Inc(RxTcp, Length(Buf));
>>            SerWrite(FComH, Buf[0], Length(Buf));
>>            Inc(TxSer, Length(Buf));
>> Where could I be losing incoming serial data?
>
>Hello,
>
>Where do you check that "Length(Buf)" has been sent ?

Well as you can see I don't.
However, I have not had any problems *receiving* data from the remote
server (this is where the SerWrite would be used to send it into my
Win7 program).
What I see is losing data when sending to the server, i.e. not all
bytes on the wire from the Win7 program are actually received by my
relaying code:

    SetLength(Buf, BufLen);
    LenRd := SerRead(FComH, Buf[0], BufLen);

Here data are arriving but some are lost, so the total number of bytes
relayed are too few.
THey are not lost on the TCP/IP outgoing side because I am counting
and showing the number of received bytes and this counter does not
ever go above 1000000, it stops somewhere at 970000 or so.

>if SerWrite(FComH, Buf[0], Length(Buf))<>Length(Buf) then Raise
>Exception.Create("KBOOM");


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
In reply to this post by Mark Morgan Lloyd-5
On Thu, 7 Sep 2017 10:42:59 +0000, Mark Morgan Lloyd
<[hidden email]> wrote:

>SerWrite() returns the actual number of bytes it's written, which could
>be anywhere from the number passed as the parameter down to zero (if
>hardware handshaking's being used at a lower level). An intermediate
>value would indicate that a buffer's full somewhere, and that you should
>adjust your source address and length and have another shot.
>

But I am losing data in the *receive* end, not while sending back to
Win7.
As described I am sending 0x0FF000 bytes of data out of the Win7
serial port but only less than a million is counted in the serial
receive end of the relaying program. So too few bytes are sent over
the TCP/IP link.

I think I will have to create a bogus image file where I have a
regular pattern of bytes so I can see when the hickups occur in the
log. In a normal memory file most of the data is zero, so there are
long stretches of zero bytes sent. Therefore it is not easy to see
where the losses occur, I only see that the last part of the file
(which contains some data) has been displaced to lower addresses.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Marc Santhoff-2
In reply to this post by Bo Berglund
On Do, 2017-09-07 at 13:32 +0200, Bo Berglund wrote:

> On Wed, 06 Sep 2017 23:21:17 +0200, Marc Santhoff
> <[hidden email]> wrote:
>
> >Could that be some sort of timeout?
> >
> >You're setting
> >
> >  FTcpComm.ReadTimeout := 30;
> >
> >but I have no idea what order of magnitude this 30 is. Maybe streching
> >the timeout a little bit can help?
> >
> >I had several cases of timeout problems using RS232 over TCP.
>
> No, the Tcp/IP part is not involved at all here. When the transfer to
> the server starts there is no response expected until all data have
> been transferred.
> The problem is that the Win7 application sends RS232 data out a serial
> port and another application (my relayer) is using another serial port
> which is connected by wire to the first serial port.
> This relaying application seems to lose incoming data, which are sent
> at a speed of 38400 baud.
> Before the data arrives to the TCP/IP part they have been corrupted.

What structure does the Windows side have, ist your application running
on the same machine as the remote control program for the data system or
is there a second computer attached using cables?

> The test on location proves that the case where the relayer is not
> involved (wire goes from Win7 application directly to the data system)
> is working fine.
> But I cannot use that because I am in Sweden and the data system is in
> Texas...

The most important part is to see where exactly data is vanishing. Which
I don't at the moment. There is only one small part of the transmission
path involved, running from the RS232 out of the windows app to the
relaying app, as I understand. And this only happens in the direction
from the win app to the remote system. Did I get that right?

--
Marc Santhoff <[hidden email]>

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
In reply to this post by Bo Berglund
On Thu, 07 Sep 2017 13:48:27 +0200, Bo Berglund
<[hidden email]> wrote:

>I think I will have to create a bogus image file where I have a
>regular pattern of bytes so I can see when the hickups occur in the
>log. In a normal memory file most of the data is zero, so there are
>long stretches of zero bytes sent. Therefore it is not easy to see
>where the losses occur, I only see that the last part of the file
>(which contains some data) has been displaced to lower addresses.

I did so and discovered that loss of data started after some 48-50
Kbytes had been transferred. At that point a few bytes were missing.
But the losses got progressively worse as the transfer went on and in
the end there was about 50 K lost.

Then I tested a few things (like adding delays etc) but with no real
difference.

FINALLY FOUND A SOLUTION...
----------------------------
Finally I realized that the relayer was very busy during the transfer
since it was displaying the relayed data in a log window (a listbox) I
had put on the screen!

In the RelayData procedure there are two calls like this for the two
data directions:

  LogHex('Rx', Buf); //Incoming TCP/IP data
and
  LogHex('Tx', Buf); //Incoming RS232 data

The LogHex procedure does this:

procedure TfrmMain.LogHex(Prefix: AnsiString; var Buf: TIdBytes);
var
  i, l: integer;
  sLine,
  sTime: AnsiString;
begin
  sTime := FormatDateTime('yyyy-mm-dd hh:nn:ss.zzz', Now()) + '  ';
  l := Length(Buf);
  sLine := Prefix + ': ';
  for i := 0 to l-1 do
  begin
    sLine := sLine + IntToHex(Buf[i],2);
  end;
  lbxLog.Items.Add(sTime + ' ' + sLine);
  lbxLog.ItemIndex := lbxLog.Items.Count-1;
end;

Then I added a checkbox on the application form to switch on/off this
logging to the listbox.
I put this check as the first line in the LogHex function:

  if not ckbShowLogData.Checked then exit;

Next I unchecked the checkbox before doing the long transfer, and
lo-and-behold(!) now there are no losses anymore!

So the action of writing text to the listbox in the LogHex function
was actually causing the application to lose incoming serial data!

Serial buffer size?
-------------------
I thought that upping the serial buffers to 60 K would have helped but
apparently not. Maybe the change I did was not used after all?

What I did regarding buffer change:
1) Rightclicked SerSetParams() and selected "Find declaration"
2) This brought up the serial.pp unit at the declaration
3) Used Ctrl-Shft-downarrow to find the implementation
4) This had to be repeated for SerSetParamsPrivate()
5) In this function body the first line was this:
const   bufSize= 2048;

which I changed to:
const   bufSize= 61440;

6) Then I rebuilt my application, but I have no idea if this actually
caused any buffer change. I know too little about the inner workings
of the Lazarus/FPC system...

So it might still use the tiny 2K buffer???


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

el_es
On 08/09/17 09:34, Bo Berglund wrote:

> 6) Then I rebuilt my application, but I have no idea if this actually
> caused any buffer change. I know too little about the inner workings
> of the Lazarus/FPC system...
>
> So it might still use the tiny 2K buffer???
>
>

I would decouple the log display from the serial handling routines -
by delegating the serial task to a thread, only display of log is handled in
the main thread (this is true whether your application is cli-only or gui -
have the tasks that should not be interrupted, like bulk serial communication,
handled by a thread, and whatever the thread wants to display, use a buffer (like TThreadList) to
communicate to the main thread.

-L.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

Bo Berglund
On Fri, 8 Sep 2017 13:06:16 +0100, el es
<[hidden email]> wrote:

>
>I would decouple the log display from the serial handling routines -
>by delegating the serial task to a thread, only display of log is handled in
>the main thread (this is true whether your application is cli-only or gui -
>have the tasks that should not be interrupted, like bulk serial communication,
>handled by a thread, and whatever the thread wants to display, use a buffer (like TThreadList) to
>communicate to the main thread.
>

Yes, when I saw this I figured that the data handling should be done
in a thread.
However, I am not at all sure how this can be accomplished.
Basically all of the activity is handled inside my RelayData()
function. This is where the Tcp and RS232 data are crossfed and also
where I call the logging function.

But if I create a thread and put this function into it, how would that
work concerning access to the IdTcpClient and Serial objects? They
were set up and connected in the main application and nothing but the
actual data handling needs to be put inside the thread if it can
communicate the logging info in a way that makes it non-disturbed.

But how?
I have 4 TStaticText controls that show the accumulated byte counts in
the Rx and Tx channels of the socket and comport. These are updated in
RelayData(). And of course I have the listbox displaying the data
going back and forth. I think that the problem is in the listbox,
which will get more and more data as time goes by...
Probably I should have some other logging/display system here.

I have very limited experience with threads and then these did not
interact at all with any GUI functions.

--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using Serial in a TCP/RS232 gateway, how to set buffer sizes?

José Mejuto
In reply to this post by Bo Berglund
El 08/09/2017 a las 10:34, Bo Berglund escribió:

>    for i := 0 to l-1 do
>    begin
>      sLine := sLine + IntToHex(Buf[i],2);
>    end;
>    lbxLog.Items.Add(sTime + ' ' + sLine);
>    lbxLog.ItemIndex := lbxLog.Items.Count-1;

Hello,

This is a big "NO, NO!" for quite large buffers, that code can take a
lot of time for a 50K buffer, as it will render in the expected 50K
inttohex conversions, but also (in worst and impossible case scenario)
50K memory moves, 50K thread locks, 50K memory allocations and 50K
memory frees.

Much better something like:

SetLength(sLine, Length(buf) * 2);
Bin2Hex(@Buf[0],sLine[1],Length(Buf));
sLine := Prefix + ':' + sLine;

Please recheck params syntax for Bin2Hex, my memory is not as it used to
be :-)

A large listbox is not also very good.

--

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
12