TProcess usage and reading program output

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

TProcess usage and reading program output

Graeme Geldenhuys-6
Hi,

Can anybody see if there is something wrong with the code shown below.
The code is copied from one of my earlier projects where I call the FPC
compiler and it worked just fine in that project.

In the work I'm doing now, I'm calling the Delphi Command Line Compiler,
and made a few minor tweaks to the code, but this method never seems to
return when Delphi command line compiler is called, and I don't see any
output. If I call other utilities (eg: Git etc) then I do see output and
it works as expected. It's just the Delphi Command Line Compiler that
now seems to be giving troubles.

I would really appreciate it is anybody could spare a moment. Many thanks.

I marked two areas with "// ???" where I am unsure if I'm doing the
right thing.


==================================
function TBuildDelphiProject.RunTProcess(const Binary: string; args:
TStrings): boolean;
const
  BufSize = 1024;
var
  p: TProcess;
  Buf: string;
  Count: integer;
  i: integer;
  LineStart: integer;
  OutputLine: string;
begin
  p := TProcess.Create(nil);
  try
    p.Executable := Binary;
    // ??? Is poWaitOnExit needed here, it is called later down the code
    p.Options := [poUsePipes, poStdErrToOutPut, poWaitOnExit];
//    p.CurrentDirectory := ExtractFilePath(p.Executable);
    p.ShowWindow := swoShowNormal;  // ??? Is this needed?

    p.Parameters.Assign(args);
    DoLog(etInfo,'Running command "%s" with arguments
"%s"',[p.Executable, p.Parameters.Text]);
    p.Execute;

    { Now process the output }
    OutputLine:='';
    SetLength(Buf,BufSize);
    repeat
      if (p.Output<>nil) then
      begin
        Count:=p.Output.Read(Buf[1],Length(Buf));
      end
      else
        Count:=0;
      LineStart:=1;
      i:=1;
      while i<=Count do
      begin
        if Buf[i] in [#10,#13] then
        begin
          OutputLine:=OutputLine+Copy(Buf,LineStart,i-LineStart);
          writeln(OutputLine);
          OutputLine:='';
          if (i<Count) and (Buf[i+1] in [#10,#13]) and
(Buf[i]<>Buf[i+1]) then
            inc(i);
          LineStart:=i+1;
        end;
        inc(i);
      end;
      OutputLine:=Copy(Buf,LineStart,Count-LineStart+1);
    until Count=0;
    if OutputLine <> '' then
      writeln(OutputLine);
    p.WaitOnExit;
    Result := p.ExitStatus = 0;
    if Not Result then
      Writeln('Command ', p.Executable ,' failed with exit code: ',
p.ExitStatus);
  finally
    FreeAndNil(p);
  end;
end;
==================================


Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: TProcess usage and reading program output

Graeme Geldenhuys-6
On 2017-02-28 16:06, Graeme Geldenhuys wrote:
> //    p.CurrentDirectory := ExtractFilePath(p.Executable);


Just thought I would explain this. I don't change directory there (in
the TProcess instance), because the program itself changes the current
directory before calling RunTProcess().

As for the procedure not returning - from what I can see (I think) it
seems like if Delphi fails to compile the project, that's when my code
get stuck and never returns.

Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: TProcess usage and reading program output

Michael Van Canneyt
In reply to this post by Graeme Geldenhuys-6


On Tue, 28 Feb 2017, Graeme Geldenhuys wrote:

> Hi,
>
> Can anybody see if there is something wrong with the code shown below.
> The code is copied from one of my earlier projects where I call the FPC
> compiler and it worked just fine in that project.
>
> In the work I'm doing now, I'm calling the Delphi Command Line Compiler,
> and made a few minor tweaks to the code, but this method never seems to
> return when Delphi command line compiler is called, and I don't see any
> output. If I call other utilities (eg: Git etc) then I do see output and
> it works as expected. It's just the Delphi Command Line Compiler that
> now seems to be giving troubles.
>
> I would really appreciate it is anybody could spare a moment. Many thanks.
>
> I marked two areas with "// ???" where I am unsure if I'm doing the
> right thing.

poWaitOnExit should not be needed, as this will cause Execute to wait for process exit...

It seems likely that this will interfere with reading from output: when the
output buffer is full, the executed process will block.

The rest seems fine.

Also:
It may be that Delphi somehow writes directly to the console buffer, in that
case you will not catch any output. But then you should see the output on
the screen (just not caught by your process)

Michael.





>
>
> ==================================
> function TBuildDelphiProject.RunTProcess(const Binary: string; args:
> TStrings): boolean;
> const
>  BufSize = 1024;
> var
>  p: TProcess;
>  Buf: string;
>  Count: integer;
>  i: integer;
>  LineStart: integer;
>  OutputLine: string;
> begin
>  p := TProcess.Create(nil);
>  try
>    p.Executable := Binary;
>    // ??? Is poWaitOnExit needed here, it is called later down the code
>    p.Options := [poUsePipes, poStdErrToOutPut, poWaitOnExit];
> //    p.CurrentDirectory := ExtractFilePath(p.Executable);
>    p.ShowWindow := swoShowNormal;  // ??? Is this needed?
>
>    p.Parameters.Assign(args);
>    DoLog(etInfo,'Running command "%s" with arguments
> "%s"',[p.Executable, p.Parameters.Text]);
>    p.Execute;
>
>    { Now process the output }
>    OutputLine:='';
>    SetLength(Buf,BufSize);
>    repeat
>      if (p.Output<>nil) then
>      begin
>        Count:=p.Output.Read(Buf[1],Length(Buf));
>      end
>      else
>        Count:=0;
>      LineStart:=1;
>      i:=1;
>      while i<=Count do
>      begin
>        if Buf[i] in [#10,#13] then
>        begin
>          OutputLine:=OutputLine+Copy(Buf,LineStart,i-LineStart);
>          writeln(OutputLine);
>          OutputLine:='';
>          if (i<Count) and (Buf[i+1] in [#10,#13]) and
> (Buf[i]<>Buf[i+1]) then
>            inc(i);
>          LineStart:=i+1;
>        end;
>        inc(i);
>      end;
>      OutputLine:=Copy(Buf,LineStart,Count-LineStart+1);
>    until Count=0;
>    if OutputLine <> '' then
>      writeln(OutputLine);
>    p.WaitOnExit;
>    Result := p.ExitStatus = 0;
>    if Not Result then
>      Writeln('Command ', p.Executable ,' failed with exit code: ',
> p.ExitStatus);
>  finally
>    FreeAndNil(p);
>  end;
> end;
> ==================================
>
>
> Regards,
>  Graeme
>
> --
> fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
> http://fpgui.sourceforge.net/
>
> My public PGP key:  http://tinyurl.com/graeme-pgp
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: TProcess usage and reading program output

Graeme Geldenhuys-6
On 2017-02-28 16:28, Michael Van Canneyt wrote:
> poWaitOnExit should not be needed, as this will cause Execute to wait for process exit...
>
> It seems likely that this will interfere with reading from output: when the
> output buffer is full, the executed process will block.

Ah. I've also been reading the docs and this FPC wiki page...

   http://wiki.freepascal.org/Executing_External_Programs

In the wiki they talked about a dead-lock when poWaitOnExit is used and
the output buffer is full. It seems that is exactly what I experienced.
I'll increase the buffer size I use to 2048 and remove the poWaitOnExit
and see how it goes. The "Reading large output" example also uses a
TMemoryStream instead of a string buffer like I do, but I think we
should have the same end result.


http://wiki.freepascal.org/Executing_External_Programs#Reading_large_output

If my code still doesn't work, I'll switch to using a TMemorySteam for
storing the output and retest.

Thanks for the extra pair of eyes.

Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: TProcess usage and reading program output

noreply
In reply to this post by Graeme Geldenhuys-6
On 2017-02-28 10:06, Graeme Geldenhuys wrote:
> Hi,
>
> Can anybody see if there is something wrong with the code shown below.
> The code is copied from one of my earlier projects where I call the FPC
> compiler and it worked just fine in that project.


Did you end up resolving the issue?

I'd be interested in creating a build tool that not only compiles
projects in FPC but also compiles with dcc32 (delphi compiler)..

Is that something like what you are doing? I have a lot of projects that
compile both in fpc and delphi and compiling manually using to compilers
is a pain. One step just compiling it with a build tool that does both
compilers would be nice.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: TProcess usage and reading program output

Graeme Geldenhuys-6
In reply to this post by Michael Van Canneyt
On 2017-02-28 16:28, Michael Van Canneyt wrote:
> poWaitOnExit should not be needed, as this will cause Execute to wait for process exit...

This post is simply to close off this thread. Michael's suggestion was
the solution to my problem. I removed the poWaitOnExit and set my buffer
size to 2048 bytes and that seems to have resolved the issue.

Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: TProcess usage and reading program output

Graeme Geldenhuys-6
In reply to this post by noreply
On 2017-02-28 20:03, [hidden email] wrote:
> Did you end up resolving the issue?

Yes, see my "closing thoughts" reply to Michael. What he suggested fixed
the issue.


> I'd be interested in creating a build tool that not only compiles
> projects in FPC but also compiles with dcc32 (delphi compiler)..
>
> Is that something like what you are doing?

I'm converting a *very* complex (and massive) multi-project Makefile
system into a single "build binary". Configuration is controlled via
project.json files. eg: build dependencies, build lists (if a project
consists of more that one project), compiler parameters with macro
support (macros are global configurations that can be injected in
compiler parameters etc).

This is not a general tool, but a specific commercial solution. Saying
that, it should be possible building a general tool that does similar -
reading project.json files to feed the build system. But then again,
there are probably multiple such solutions around already (I think).


Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: TProcess usage and reading program output

Benito van der Zander
In reply to this post by Michael Van Canneyt
Hi,

 how would you handle large output and large stderr?


When you read from one and it writes to the other, the read blocks. Then it keeps writing to the other buffer, till that is full, and then its write is blocked, and it is deadlocked.

Probably check NumBytesAvailable before reading?

Bye,
Benito 



On 02/28/2017 05:28 PM, Michael Van Canneyt wrote:


On Tue, 28 Feb 2017, Graeme Geldenhuys wrote:

Hi,

Can anybody see if there is something wrong with the code shown below.
The code is copied from one of my earlier projects where I call the FPC
compiler and it worked just fine in that project.

In the work I'm doing now, I'm calling the Delphi Command Line Compiler,
and made a few minor tweaks to the code, but this method never seems to
return when Delphi command line compiler is called, and I don't see any
output. If I call other utilities (eg: Git etc) then I do see output and
it works as expected. It's just the Delphi Command Line Compiler that
now seems to be giving troubles.

I would really appreciate it is anybody could spare a moment. Many thanks.

I marked two areas with "// ???" where I am unsure if I'm doing the
right thing.

poWaitOnExit should not be needed, as this will cause Execute to wait for process exit...

It seems likely that this will interfere with reading from output: when the
output buffer is full, the executed process will block.

The rest seems fine.

Also:
It may be that Delphi somehow writes directly to the console buffer, in that
case you will not catch any output. But then you should see the output on
the screen (just not caught by your process)

Michael.







==================================
function TBuildDelphiProject.RunTProcess(const Binary: string; args:
TStrings): boolean;
const
 BufSize = 1024;
var
 p: TProcess;
 Buf: string;
 Count: integer;
 i: integer;
 LineStart: integer;
 OutputLine: string;
begin
 p := TProcess.Create(nil);
 try
   p.Executable := Binary;
   // ??? Is poWaitOnExit needed here, it is called later down the code
   p.Options := [poUsePipes, poStdErrToOutPut, poWaitOnExit];
//    p.CurrentDirectory := ExtractFilePath(p.Executable);
   p.ShowWindow := swoShowNormal;  // ??? Is this needed?

   p.Parameters.Assign(args);
   DoLog(etInfo,'Running command "%s" with arguments
"%s"',[p.Executable, p.Parameters.Text]);
   p.Execute;

   { Now process the output }
   OutputLine:='';
   SetLength(Buf,BufSize);
   repeat
     if (p.Output<>nil) then
     begin
       Count:=p.Output.Read(Buf[1],Length(Buf));
     end
     else
       Count:=0;
     LineStart:=1;
     i:=1;
     while i<=Count do
     begin
       if Buf[i] in [#10,#13] then
       begin
         OutputLine:=OutputLine+Copy(Buf,LineStart,i-LineStart);
         writeln(OutputLine);
         OutputLine:='';
         if (i<Count) and (Buf[i+1] in [#10,#13]) and
(Buf[i]<>Buf[i+1]) then
           inc(i);
         LineStart:=i+1;
       end;
       inc(i);
     end;
     OutputLine:=Copy(Buf,LineStart,Count-LineStart+1);
   until Count=0;
   if OutputLine <> '' then
     writeln(OutputLine);
   p.WaitOnExit;
   Result := p.ExitStatus = 0;
   if Not Result then
     Writeln('Command ', p.Executable ,' failed with exit code: ',
p.ExitStatus);
 finally
   FreeAndNil(p);
 end;
end;
==================================


Regards,
 Graeme

-- 
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal