Unbuffering I/O

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Unbuffering I/O

Mark Morgan Lloyd-5
I think I've seen this question asked before, my apologies if this was
recently.

I've got two programs intended to be functionally identical, one in Perl
and the other in FPC. They read a unix-domain datagram, decode the
message, and emit output; if this goes to a file then it's reasonable to
monitor it using  tail -f

Perl has a variable that you can set to force output to be unbuffered,
with the result that as soon as a message is output it's in the file in
its entirety.

Is there an equivalent for Pascal, or should I be using something like
fpSync(stdout) at opportune times?

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Henry Vermaak
On Wed, Aug 29, 2018 at 03:01:54PM +0000, Mark Morgan Lloyd wrote:

> I think I've seen this question asked before, my apologies if this was
> recently.
>
> I've got two programs intended to be functionally identical, one in Perl and
> the other in FPC. They read a unix-domain datagram, decode the message, and
> emit output; if this goes to a file then it's reasonable to monitor it using
> tail -f
>
> Perl has a variable that you can set to force output to be unbuffered, with
> the result that as soon as a message is output it's in the file in its
> entirety.
>
> Is there an equivalent for Pascal, or should I be using something like
> fpSync(stdout) at opportune times?

Does SetTextBuf() with a buffer of size 1 work?  I don't think there is
anything equivalent to setvbuf().  Otherwise you'll have to Flush() them
manually, which is a pain.

Henry
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Mark Morgan Lloyd-5
On 29/08/18 16:00, Henry Vermaak wrote:
> On Wed, Aug 29, 2018 at 03:01:54PM +0000, Mark Morgan Lloyd wrote:> I think I've seen this question asked before, my apologies if this was> recently.> > I've got two programs intended to be functionally identical, one in Perl and> the other in FPC. They read a unix-domain datagram, decode the message, and> emit output; if this goes to a file then it's reasonable to monitor it using> tail -f> > Perl has a variable that you can set to force output to be unbuffered, with> the result that as soon as a message is output it's in the file in its> entirety.> > Is there an equivalent for Pascal, or should I be using something like> fpSync(stdout) at opportune times?
> Does SetTextBuf() with a buffer of size 1 work?  I don't think there isanything equivalent to setvbuf().  Otherwise you'll have to Flush() themmanually, which is a pain.

I'm a bit wary of SetTextBuf() with a preopened handle. Flush() is
actually no big deal since the program has a well-defined processing
loop... but unfortunately is less than effective.

It's something I can live with, since this is really only a test program
I knocked together to look at how clock_gettime() behaves on different
platforms (the answer being "badly" :-)

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Marc Santhoff
In reply to this post by Mark Morgan Lloyd-5
On Wed, 2018-08-29 at 15:01 +0000, Mark Morgan Lloyd wrote:

> I've got two programs intended to be functionally identical, one in Perl
> and the other in FPC. They read a unix-domain datagram, decode the
> message, and emit output; if this goes to a file then it's reasonable to
> monitor it using  tail -f
>
> Perl has a variable that you can set to force output to be unbuffered,
> with the result that as soon as a message is output it's in the file in
> its entirety.
>
> Is there an equivalent for Pascal, or should I be using something like
> fpSync(stdout) at opportune times?

I don't know if it really helps, but since I fiddled with a similar topic:

FreeBSD has stdio channel non blocking by default. I had some problems with
that and did this:


procedure SwitchFdBlocking(var channel: Text);
var
  res: cint;
begin
  res := fpfcntl(TextRec(channel).Handle, F_GETFL);
  if ((O_NONBLOCK AND res)>0) then
  begin
    res := res AND (NOT O_NONBLOCK);
    res := fpfcntl(TextRec(channel).Handle, F_SETFL, res);
    if (res=-1) then writeln(stderr, 'ERROR on SETFL');
  end;
end;

According to the man page of fcntl using the flag "O_DIRECT" instead of
"O_NONBLOCK" for switching of caching as far as possible. At least on FreeBSD
it is like this, check on Linux yourself...

--
Marc Santhoff <[hidden email]>
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Free Pascal - General mailing list
In reply to this post by Mark Morgan Lloyd-5
Am 29.08.2018 um 18:25 schrieb Mark Morgan Lloyd:

> On 29/08/18 16:00, Henry Vermaak wrote:
>> On Wed, Aug 29, 2018 at 03:01:54PM +0000, Mark Morgan Lloyd wrote:> I
>> think I've seen this question asked before, my apologies if this was>
>> recently.> > I've got two programs intended to be functionally
>> identical, one in Perl and> the other in FPC. They read a unix-domain
>> datagram, decode the message, and> emit output; if this goes to a
>> file then it's reasonable to monitor it using> tail -f> > Perl has a
>> variable that you can set to force output to be unbuffered, with> the
>> result that as soon as a message is output it's in the file in its>
>> entirety.> > Is there an equivalent for Pascal, or should I be using
>> something like> fpSync(stdout) at opportune times?
>> Does SetTextBuf() with a buffer of size 1 work?  I don't think there
>> isanything equivalent to setvbuf().  Otherwise you'll have to Flush()
>> themmanually, which is a pain.
>
> I'm a bit wary of SetTextBuf() with a preopened handle. Flush() is
> actually no big deal since the program has a well-defined processing
> loop... but unfortunately is less than effective.
>
> It's something I can live with, since this is really only a test
> program I knocked together to look at how clock_gettime() behaves on
> different platforms (the answer being "badly" :-)
>
SetTextBuf() does not influence the handle itself, only the buffer of
the file/text record, so it should be safe to use.

Regards,
Sven
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

wkitty42
In reply to this post by Mark Morgan Lloyd-5
On 08/29/2018 11:01 AM, Mark Morgan Lloyd wrote:
> Is there an equivalent for Pascal, or should I be using something like
> fpSync(stdout) at opportune times?

flush();


i use it all the time on my programs that write to the logs... i hate having a
crash and be missing some log output because it was buffered and lost with the
crash... flush() everything...



--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list unless*
        *a signed and pre-paid contract is in effect with us.*
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

wkitty42
In reply to this post by Henry Vermaak
On 08/29/2018 11:49 AM, Henry Vermaak wrote:
> Otherwise you'll have to Flush() them manually, which is a pain.

really? i wrote a wrapper for write and writeln that simply calls them and then
does a flush()... nothing painful other than using mywrite() and mywriteln() or
similar ;)


--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list unless*
        *a signed and pre-paid contract is in effect with us.*
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Martin Schreiber-2
In reply to this post by Mark Morgan Lloyd-5
On Wednesday 29 August 2018 17:01:54 Mark Morgan Lloyd wrote:

> I think I've seen this question asked before, my apologies if this was
> recently.
>
> I've got two programs intended to be functionally identical, one in Perl
> and the other in FPC. They read a unix-domain datagram, decode the
> message, and emit output; if this goes to a file then it's reasonable to
> monitor it using  tail -f
>
> Perl has a variable that you can set to force output to be unbuffered,
> with the result that as soon as a message is output it's in the file in
> its entirety.
>
> Is there an equivalent for Pascal, or should I be using something like
> fpSync(stdout) at opportune times?

In order to flush textfiles automatically I use
"
 ttextrec(<thefile>).flushfunc:= ttextrec(<thefile>).inoutfunc;
"
after it is opened.

Martin

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Henry Vermaak
On Thu, Aug 30, 2018 at 07:52:54AM +0200, Martin Schreiber wrote:

> On Wednesday 29 August 2018 17:01:54 Mark Morgan Lloyd wrote:
> > I think I've seen this question asked before, my apologies if this was
> > recently.
> >
> > I've got two programs intended to be functionally identical, one in Perl
> > and the other in FPC. They read a unix-domain datagram, decode the
> > message, and emit output; if this goes to a file then it's reasonable to
> > monitor it using  tail -f
> >
> > Perl has a variable that you can set to force output to be unbuffered,
> > with the result that as soon as a message is output it's in the file in
> > its entirety.
> >
> > Is there an equivalent for Pascal, or should I be using something like
> > fpSync(stdout) at opportune times?
>
> In order to flush textfiles automatically I use
> "
>  ttextrec(<thefile>).flushfunc:= ttextrec(<thefile>).inoutfunc;
> "
> after it is opened.

Reading text.inc this morning lead me to believe this is the correct
way.  This is what the RTL does when opening serial devices in
FileOpenFunc(), for example.  There's a comment inside Flush() that says
that InOutFunc() should be used to flush, since the FlushFunc() may not
be assigned.

Thanks for this, Martin.

Henry
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Mark Morgan Lloyd-5
On 30/08/18 09:00, Henry Vermaak wrote:
> On Thu, Aug 30, 2018 at 07:52:54AM +0200, Martin Schreiber wrote:

> In order to flush textfiles automatically I use> ">  ttextrec(<thefile>).flushfunc:= ttextrec(<thefile>).inoutfunc;> "> after it is opened.

> Reading text.inc this morning lead me to believe this is the correctway.  This is what the RTL does when opening serial devices inFileOpenFunc(), for example.  There's a comment inside Flush() that saysthat InOutFunc() should be used to flush, since the FlushFunc() may notbe assigned.

I've just checked this and unfortunately it doesn't do very much for the
standard device (?) used by WriteLn() etc., i.e. as would be used for a
quick-and-dirty program.

I've tried

     ttextrec(StdErr).flushfunc:= ttextrec(StdErr).inoutfunc;
//    Flush(StdErr);
     while true do begin
//      Flush(StdOut);                  // Has limited effect
       ttextrec(StdErr).flushfunc:= ttextrec(StdErr).inoutfunc;

and

program(input, output, error)

..
//    error.flushfunc:= error.inoutfunc;
//    Flush(StdErr);
     while true do begin
//      Flush(StdOut);                  // Has limited effect
       output.flushfunc:= output.inoutfunc;

(Is that latter form even valid any more?)

Please don't anybody put any effort into it- it's a very minor niggle
from my POV.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Henry Vermaak
On Thu, Aug 30, 2018 at 09:45:00AM +0000, Mark Morgan Lloyd wrote:

> On 30/08/18 09:00, Henry Vermaak wrote:
> >On Thu, Aug 30, 2018 at 07:52:54AM +0200, Martin Schreiber wrote:
>
> >In order to flush textfiles automatically I use> ">  ttextrec(<thefile>).flushfunc:= ttextrec(<thefile>).inoutfunc;> "> after it is opened.
>
> >Reading text.inc this morning lead me to believe this is the correctway.  This is what the RTL does when opening serial devices inFileOpenFunc(), for example.  There's a comment inside Flush() that saysthat InOutFunc() should be used to flush, since the FlushFunc() may notbe assigned.
>
> I've just checked this and unfortunately it doesn't do very much for the
> standard device (?) used by WriteLn() etc., i.e. as would be used for a
> quick-and-dirty program.

It definitely works for me.  I made a program flush.pas that looks like this:

begin
        ttextrec(output).flushfunc := ttextrec(output).inoutfunc;
        writeln('hi there');
        sleep(1000);
end.

Running `./flush > out.txt` and `tail -F out.txt` shows the output
immediately, while removing the flushfunc assignment causes a delay of a
second before the output appeared in the file.

Henry
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Mark Morgan Lloyd-5
On 30/08/18 10:15, Henry Vermaak wrote:
> On Thu, Aug 30, 2018 at 09:45:00AM +0000, Mark Morgan Lloyd wrote:> On 30/08/18 09:00, Henry Vermaak wrote:> >On Thu, Aug 30, 2018 at 07:52:54AM +0200, Martin Schreiber wrote:> > >In order to flush textfiles automatically I use> ">  ttextrec(<thefile>).flushfunc:= ttextrec(<thefile>).inoutfunc;> "> after it is opened.> > >Reading text.inc this morning lead me to believe this is the correctway.  This is what the RTL does when opening serial devices inFileOpenFunc(), for example.  There's a comment inside Flush() that saysthat InOutFunc() should be used to flush, since the FlushFunc() may notbe assigned.> > I've just checked this and unfortunately it doesn't do very much for the> standard device (?) used by WriteLn() etc., i.e. as would be used for a> quick-and-dirty program.
> It definitely works for me.  I made a program flush.pas that looks like this:
> begin ttextrec(output).flushfunc := ttextrec(output).inoutfunc; writeln('hi there'); sleep(1000);end.
> Running `./flush > out.txt` and `tail -F out.txt` shows the outputimmediately, while removing the flushfunc assignment causes a delay of asecond before the output appeared in the file.

Ah yes, that's it, thanks very much.

       WriteLn(StdErr, Format('# Socket %s, clock resolution %8.6f
uSec', [socketName, ts.tv_nsec / 1000]));
     ttextrec(StdErr).flushfunc:= ttextrec(StdErr).inoutfunc;
     while true do begin
       ttextrec(Output).flushfunc:= ttextrec(Output).inoutfunc;
       i := fprecv(client, @buff, 1024, 0);

Basically, I was getting confused by the lack of "Error" etc., hence the
different names.

I note this works the same for both  tail -f  and  tail -F  which have
slightly different ways of looking to see if the input file has changed.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Henry Vermaak
On Thu, Aug 30, 2018 at 10:57:21AM +0000, Mark Morgan Lloyd wrote:
> Ah yes, that's it, thanks very much.
>
>       WriteLn(StdErr, Format('# Socket %s, clock resolution %8.6f uSec',
> [socketName, ts.tv_nsec / 1000]));
>     ttextrec(StdErr).flushfunc:= ttextrec(StdErr).inoutfunc;
>     while true do begin
>       ttextrec(Output).flushfunc:= ttextrec(Output).inoutfunc;
>       i := fprecv(client, @buff, 1024, 0);

You only need to set flushfunc once at startup (for standard handles) or
just after opening a file.  The whole output vs stdout thing has
confused me in the past, too.

Henry
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Unbuffering I/O

Mark Morgan Lloyd-5
On 30/08/18 12:00, Henry Vermaak wrote:
> On Thu, Aug 30, 2018 at 10:57:21AM +0000, Mark Morgan Lloyd wrote:> Ah yes, that's it, thanks very much.> >       WriteLn(StdErr, Format('# Socket %s, clock resolution %8.6f uSec',> [socketName, ts.tv_nsec / 1000]));>     ttextrec(StdErr).flushfunc:= ttextrec(StdErr).inoutfunc;>     while true do begin>       ttextrec(Output).flushfunc:= ttextrec(Output).inoutfunc;>       i := fprecv(client, @buff, 1024, 0);
> You only need to set flushfunc once at startup (for standard handles) orjust after opening a file.  The whole output vs stdout thing hasconfused me in the past, too.

Thanks, noted and good point. That's a result of where the flushes
originally were.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal