How to use pipes ?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

How to use pipes ?

fredvs
Hello.

I have done lot of search on internet about pipes without success.
Even for Delphi. no demo nor explanation.
The same in fpc wki page.

For example, how to use bytesavailable with pipes ?

In:  
CreatePipeHandles(InHandle, OutHandle, APipeBufferSize);

InPipe.Read(Bufferout[0],BPipeBufferSize);

What is the the ratio with APipeBufferSize vs  BPipeBufferSize ?

And length of Bufferout[0] vs APipeBufferSize vs BPipeBufferSize ?

In short, a demo will be welcome.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

Mattias Gaertner
On Wed, 1 Feb 2017 05:02:05 -0700 (MST)
fredvs <[hidden email]> wrote:

> Hello.
>
> I have done lot of search on internet about pipes without success.
> Even for Delphi. no demo nor explanation.
> The same in fpc wki page.
>
> For example, how to use bytesavailable with pipes ?

http://wiki.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess

 
> In:  
> CreatePipeHandles(InHandle, OutHandle, APipeBufferSize);
>
> InPipe.Read(Bufferout[0],BPipeBufferSize);
>
> What is the the ratio with APipeBufferSize vs  BPipeBufferSize ?

APipeBufferSize is how much the other process can produce before
congestion.
BPipeBufferSize is how much you want to process/eat each
iteration.
Usually APipeBufferSize >= BPipeBufferSize.

 
> And length of Bufferout[0] vs APipeBufferSize vs BPipeBufferSize ?

Bufferout[0] is just the first value. The whole Bufferout can be
written by InPipe.Read. Therefore Bufferout must be >= BPipeBufferSize.


> In short, a demo will be welcome.


Mattias
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

fredvs
Many thanks Mattias.

Huh, and about bytesavailable ?

See here:

 CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);
InPipe := TInputPipeStream.Create(InHandle);
OutPipe := TOutputPipeStream.Create(OutHandle);

httpget := TThreadHttpGetter.Create(url, OutPipe);

// is length ok ?
 setlength(BufferURL, PipeBufferSize);
setlength(buffertemp, PipeBufferSize);

// Is it the right way to fill BufferURL ?
len := 1 ;
len2 := 0 ;
  while (len2 < PipeBufferSize) and (len > 0) do
  begin
 len := InPipe.Read(buffertemp[0],PipeBufferSize-len2);
  if len > 0 then  for i := 0 to len -1 do
    BufferURL[i+len2] := buffertemp[i] ;
      len2 := len2 + len;
   end;

Thanks.
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

fredvs
Hello.

Some more explanation.

With that code, only +- 10 loops are working, after, no more Ouframes... why ?


const
FramesWanted : 640;
PipeBufferSize : 10240;
BufferURL, BufferTemp : tbytes;
BufferFloat : array of float;
...

setlength(BufferURL, PipeBufferSize);
setlength(BufferTemp,  FramesWanted * 2);
setlength(BufferFloat, FramesWanted * 2);
 
CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);
InPipe := TInputPipeStream.Create(InHandle);
OutPipe := TOutputPipeStream.Create(OutHandle);
HandleOP :=  op_test_memory(BufferURL[0],PipeBufferSize, Err);

Outframes := 1;
outst := 1;
outst2 := 0;

// The loop, is it the correct way ?
while Outframes > 0 do
begin
   while (outst2 < FramesWanted)  and (outst > 0)  do
  begin
     outst := InPipe.Read(BufferTemp[0],FramesWanted-outst2);
    if outst > 0 then  for i := 0 to outst -1 do
   BufferURL[i+outst2] := BufferTemp[i] ;
     outst2 := outst2 + outst;
  end;
 
// Outframes > 0 only +- 10 loops, why ?
Outframes := op_read_float(HandleOP,  @Buffer[0], FramesWanted , nil));
...
end;

Thanks.

Fre;D

Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

Andrew Haines
On 02/01/2017 08:52 AM, fredvs wrote:
> Hello.
>
> Some more explanation.
>
> With that code, only +- 10 loops are working, after, no more Ouframes... why
> ?

Where is it stopping? It may be that you are asking for more bytes than
are available and it is blocking waiting for more data. Or the opposite
could be true. You are blocking processing data but the buffer from the
kernel is full. On many Linux pc's the kernel buffer size is 65536 bytes.

Regards,

Andrew
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

fredvs
Ho Andrew, nice to read you. ;-)

I battle hard to use your Thttpgetter for Opus, like you did for mpg123.

The difference with mpg123 is that Opus needs a buffer (mpg123 a named pipe).

I am nearly sure that the problem comes from my code to get the buffer from the pipe:

  while (outst2 < FramesWanted)  and (outst > 0)  do
  begin
     outst := InPipe.Read(BufferTemp[0],FramesWanted-outst2);
    if outst > 0 then  for i := 0 to outst -1 do
   BufferURL[i+outst2] := BufferTemp[i] ;
     outst2 := outst2 + outst;
  end;

> Where is it stopping?

In the main loop, it is stopping a legal way, after +- 10 loops, because Outframes =  0:
 
Outframes := op_read_float(HandleOP,  @Buffer[0], FramesWanted , nil);
if Outframes = 0 then exit;

> You are blocking processing data but the buffer from the kernel is full.
> On many Linux pc's the kernel buffer size is 65536 bytes.

Yes, it seems that this is the problem (but how to solve that?)

Thanks.

Fre;D



Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

José Mejuto
El 01/02/2017 a las 19:55, fredvs escribió:

> Outframes := op_read_float(HandleOP,  @Buffer[0], FramesWanted , nil);
> if Outframes = 0 then exit;

Hello,

if OutFrames < 0 then exit;

The fact that op_read_float is zero is not an error, simply the amount
of samples read by opus engine is zero, maybe because internal buffer is
full and do not need more data by now.

Reading from internet is usually faster than playing bitrate. Maybe you
should sleep for a bit and retry.

 From docs:

Returns:
The number of samples read per channel on success, or a negative value
on failure.


--

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

fredvs
Hello Jose.

>The fact that op_read_float is zero is not an error, simply the amount of samples read by opus engine is zero

Exactly. I know that isnot a error otherwise OutFrames < 0.

>simply the amount of samples read by opus engine is zero.

Exactly too and this because my conversion pipe into buffer is wrong.
I never worked with pipes and, to be honest, I do not get everything...
How to translate it to a "classical" buffer of a memory stream...

IMO, my code did read all the file from internet after +- 10 loops, the main buffer of the pipe is full.
I am in the dark, the mechanism of pipes escape to me.

Otherwise, many thanks to make me discover Opus.
It seems really the universal audio format.
And open source.
One of the developers is the one who created SoundFile library (a must).

Fre;D

 
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

fredvs
Re-hello.

Some more news:

OK, I make work a other method:

op_open_callbacks(InPipe, op_callbacks, BufferURL[0], PipeBufferSize, err);  

This method use a named pipe and can be use of the box with Andrew's THttpgetter.
No need to do complicated conversion of pipe into buffer.

op_callbacks does it for you.

Nice, it works, but just like with  op_open_memory, only  2 seconds of music then it stops.

The same reason: op_read_float => result after 2 seconds : Outframes = 0.

Houlala, I feel that we are near the goal.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: How to use pipes ?

fredvs
In reply to this post by fredvs
Huh, sorry to monopolize but...

I get it. ;-)

Changing the built-in read-callback method with this:

function OpusReadCB(stream: Pointer; var buffer; nbytes: cint): cint; cdecl;
begin
  if nbytes<>0
  then
 result := TInputPipeStream(stream^).read(Buffer, nbytes)
   else
    result := 0;
end;

Does the tick.

This is real web-streaming, for hours in super-high quality !
Many thanks to everybody for helping in this (hard) battle.
I will clean the code and commit it asap.

fpc always win.

Fre;D

Many thanks ;-)