Pipe vs Memory buffer.

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Pipe vs Memory buffer.

fredvs
Hello

In a C method:

OP_WARN_UNUSED_RESULT OggOpusFile *op_test_memory(const unsigned char *_data,
 size_t _size,int *_error);

translated in Pascal with this:

var
 op_test_memory: function(const _data; const size: cuint; out error: cint): TOggOpusFile;

How to use a pipe for _data ?

The pipe is a TInputPipeStream.

Using this does not work:
pipein was created and linked to a url.

var
pipein :TInputPipeStream; /
PipeBufferSize := $4000;
...
op_test_memory(pointer(InPipe),PipeBufferSize, Err);

The error is -129 which is a null pointer or other generic internal error.

Thanks.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

Michael Van Canneyt


On Wed, 25 Jan 2017, fredvs wrote:

> Hello
>
> In a C method:
>
> OP_WARN_UNUSED_RESULT OggOpusFile *op_test_memory(const unsigned char
> *_data,
> size_t _size,int *_error);
>
> translated in Pascal with this:
>
> var
> op_test_memory: function(const _data; const size: cuint; out error: cint):
> TOggOpusFile;
>
> How to use a pipe for _data ?
>
> The pipe is a TInputPipeStream.
>
> Using this does not work:
> pipein was created and linked to a url.
>
> var
> pipein :TInputPipeStream; /
> PipeBufferSize := $4000;
> ...
> op_test_memory(pointer(InPipe),PipeBufferSize, Err);
>
> The error is -129 which is a null pointer or other generic internal error.

That will never work.

OP_WARN_UNUSED_RESULT OggOpusFile *op_test_memory(const unsigned char *_data, size_t _size,int *_error);

expects a memory buffer; you cannot pass it a stream.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
>That will never work.

> OP_WARN_UNUSED_RESULT OggOpusFile *op_test_memory(const unsigned char *_data, size_t _size,int *_error);

> expects a memory buffer; you cannot pass it a stream.

Ha, ok.

And how to convert a pipe into a  memory buffer ?

Thanks.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

Michael Van Canneyt


On Wed, 25 Jan 2017, fredvs wrote:

>> That will never work.
>
>> OP_WARN_UNUSED_RESULT OggOpusFile *op_test_memory(const unsigned char
>> *_data, size_t _size,int *_error);
>
>> expects a memory buffer; you cannot pass it a stream.
>
> Ha, ok.
>
> And how to convert a pipe into a  memory buffer ?

Read from it and append the content to the buffer.
This probably means some polling mechanism:
fpSelect on unix, WaitForObject on windows.
Or maybe using a thread. But that will require some synchronisation.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
Thanks Michael.

OK, I know the way, I will try it (but code example is very welcome).

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

greim
In reply to this post by fredvs
Hello Fred,

I didn't understand the software structure / problem behind the C->
Pascal translation, but for me named pipes work very well for many kinds
of interprocess communication.

Markus





Am 25.01.2017 um 23:13 schrieb fredvs:

> Hello
>
> In a C method:
>
> OP_WARN_UNUSED_RESULT OggOpusFile *op_test_memory(const unsigned char
> *_data,
>  size_t _size,int *_error);
>
> translated in Pascal with this:
>
> var
>  op_test_memory: function(const _data; const size: cuint; out error: cint):
> TOggOpusFile;
>
> How to use a pipe for _data ?
>
> The pipe is a TInputPipeStream.
>
> Using this does not work:
> pipein was created and linked to a url.
>
> var
> pipein :TInputPipeStream; /
> PipeBufferSize := $4000;
> ...
> op_test_memory(pointer(InPipe),PipeBufferSize, Err);
>
> The error is -129 which is a null pointer or other generic internal error.
>
> Thanks.
>
> Fre;D
>
>
>
>
> -----
> Many thanks ;-)
> --
> View this message in context: http://free-pascal-general.1045716.n5.nabble.com/Pipe-vs-Memory-buffer-tp5727435.html
> Sent from the Free Pascal - General mailing list archive at Nabble.com.
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
> didn't understand the software structure / problem behind the C->
> Pascal translation

Huh, IMO, the translation is ok.
The problem is that the method ask for a memory buffer and I give a pipe.

> but for me named pipes work very well for many kinds of interprocess communication.

Yes, for example, mpg123 library is very happy with this:

PipeBufferSize := $4000 ;

CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);

InPipe := TInputPipeStream.Create(InHandle);
OutPipe := TOutputPipeStream.Create(OutHandle);
httpget := TThreadHttpGetter.Create(opus_url,OutPipe);

mpg123_open_handle(AHandle, Pointer(InPipe));

But for Opus library, there is error while using:

op_test_memory(pointer(InPipe),PipeBufferSize, Err);

Like Michael explained, the buffer of the pipe must be entered.

And I do not know how to do this... ;-(

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
Hello.

Some news from the front:

var
BufferURL : array of float;

.....

PipeBufferSize := $4000 ;

CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);

InPipe := TInputPipeStream.Create(InHandle);
OutPipe := TOutputPipeStream.Create(OutHandle);
httpget := TThreadHttpGetter.Create(opus_url,OutPipe);

setlength(BufferURL, PipeBufferSize);

// This to have the buffer as parameter ???
InPipe.Read(BufferURL,PipeBufferSize);

op_test_memory(BufferURL,PipeBufferSize, Err);

=> Gives as error:  -133 =>
A required header packet was not properly formatted, contained illegal values, or was missing altogether.

So it seems that the buffer was accepted, only miss the header ?

Difficult to explore when there is no documentation how to do nor examples.

Any idea is highly welcome.

Fe;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

silvioprog
On Thu, Jan 26, 2017 at 8:20 PM, fredvs <[hidden email]> wrote:
Hello.

Some news from the front:

var
BufferURL : array of float;

.....

PipeBufferSize := $4000 ;

CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);

InPipe := TInputPipeStream.Create(InHandle);
OutPipe := TOutputPipeStream.Create(OutHandle);
httpget := TThreadHttpGetter.Create(opus_url,OutPipe);

setlength(BufferURL, PipeBufferSize);

// This to have the buffer as parameter ???
InPipe.Read(BufferURL,PipeBufferSize);

op_test_memory(BufferURL,PipeBufferSize, Err);

=> Gives as error:  -133 =>
A required header packet was not properly formatted, contained illegal
values, or was missing altogether.

So it seems that the buffer was accepted, only miss the header ?

Difficult to explore when there is no documentation how to do nor examples.

I had never tried pipes before, but it seems FPC has a unit to handle that: `iostream`, so today I played a very simple test:

program project1;
uses sysutils, iostream;
const buf_sz = 1024;
var
  buf: tbytes;
  sz: longint;
  stdin: tiostream;
begin
  stdin := tiostream.create(iosinput);
  try
    setlength(buf, buf_sz);
    // it is just a test, so I limited it to 1024 bytes, please read the entire content by yourself! :-)
    sz := stdin.read(buf[0], buf_sz);
    if sz < buf_sz then
      setlength(buf, sz);
    writeln(tencoding.utf8.getstring(buf));
  finally
    stdin.free;
  end;
end.

in my terminal:

$ echo "Yes" | ./project1 
Yes

However, I don't know if you are searching exactly that, anyway you can check the iosinput sources. :-)

Any idea is highly welcome.

Fe;D

-----
Many thanks ;-)

--
Silvio Clécio

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
Thanks Silvio, I will try it tonight.
Write you later.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
Hello Silvio:

Yes, we are on the good way.
Following your advice, here from https-url-opus the result of :

var
  BufferURL: tbytes;

  setlength(BufferURL, PipeBufferSize);
  CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);
  InPipe := TInputPipeStream.Create(InHandle);
  OutPipe := TOutputPipeStream.Create(OutHandle);
  httpget := TThreadHttpGetter.Create(url, OutPipe);
  InPipe.Read(BufferURL[0],PipeBufferSize);
 
  writeln(tencoding.utf8.getstring(BufferURL));

======>

Lavf57.57.100encoder=Lavc57.65.100 libopustitle=Be Thankful%artist=For What You GotWilliam De Vau$album=Blaxploitation Vol.3 The Payba date=1997TRACKNUMBER=11OggS��d,����54�@BCDDDEDDDCR�����������������������������������������@*�m��|��G���"��>_4Zѡ�/kv��e"��bvZ6�j�
             �z2��PY�*��]luN��H�t�A5QR�s:3�0���:�^����͋�¢f��8�$5{w>f@r)YZ���K>�𠑡T�`�+������z�g
"F^�������a     h��«�o?"t�C�d]J=��kdw�Z3|�L��B���a���r�b�Ĉ}��u�L@�����W9��V�W �+r���{<�`�_��O&�t� G�g�܄~��oᐄQ|�~7�E��~!��<�$3�E��I����4����h;��\0L�|xE��x`�����͇G��a��@������P�[�����6,�rk��&������=����V��+�I3~�~j��^��g�x�a֣�,l'�_=����(��aԦ���u�ZҬ��
                                   �|N+9զ~e͓���2���r�a֥R�E����� �Fޡj��\��QA��s<6�Cal������ۨ��Ä�zE����!Z�&G�"���a֥��7�rɝnt�Q̊4�N��{�w[��zt��p��%�ƌ�ZYN����}䒀���7������

You may note that some opus infos are there and seems to be the header.
Sadly, op_test_memory(BufferURL,PipeBufferSize, Err) gives this error:

error -132: A purported Ogg Opus stream did not begin with an Ogg page, a purported header packet did not start with one of the required strings, "OpusHead" or "OpusTags", or a link in a chained file was encountered that did not contain any logical Opus streams.

But there is some light, we will win.

Fre;D

Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

silvioprog
On Thu, Jan 26, 2017 at 11:52 PM, fredvs <[hidden email]> wrote:
Hello Silvio:

Yes, we are on the good way.

Awesome. :-)
 
Following your advice, here from https-url-opus the result of :

var
  BufferURL: tbytes;

  setlength(BufferURL, PipeBufferSize);
  CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);
  InPipe := TInputPipeStream.Create(InHandle);
  OutPipe := TOutputPipeStream.Create(OutHandle);
  httpget := TThreadHttpGetter.Create(url, OutPipe);
  InPipe.Read(BufferURL[0],PipeBufferSize);

  writeln(tencoding.utf8.getstring(BufferURL));

======>

Lavf57.57.100  encoder=Lavc57.65.100 libopus title=Be Thankful%artist=For
What You GotWilliam De Vau$album=Blaxploitation Vol.3 The Payba
date=1997TRACKNUMBER=11OggS
[...]

Some streams requires you set its cursor to 0 before writing/reading buffer, so you need to check it:

  CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);
  InPipe := TInputPipeStream.Create(InHandle);
  OutPipe := TOutputPipeStream.Create(OutHandle);
  httpget := TThreadHttpGetter.Create(url, OutPipe);
  OutPipe.Seek(0, soBeginning);
  InPipe.Seek(0, soBeginning);
  InPipe.Read(BufferURL[0],PipeBufferSize);

Notice tencoding was just for testing. :-)

--
Silvio Clécio

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

José Mejuto
In reply to this post by fredvs
El 27/01/2017 a las 3:52, fredvs escribió:
> Hello Silvio:
>
> Yes, we are on the good way.
> Following your advice, here from https-url-opus the result of :
>

Hello,

The first thing to debug this problems is to determine the expected data
for opus_test_memory. Usually streams (audio, video, etc...) are
composed by two pieces the raw data and the transport envelope. In this
case opus usually uses Ogg as transport envelope but it could be
streamed in another container like MP4, MKV, etc.

Opus library should provide a play engine that decodes raw streams,
extracted from its transport layer and maybe another series of APIs for
the default transport layer, Ogg in this case. This last job seems to be
done by the OpusFile API.

So to resume, you file is a raw opus audio or an Ogg file ?

Before using pipes with http try pipes with a local file. If your pipes
on local file works but http does not maybe you are receiving the http
headers, or any other unexpected situation, so dump the piped data to a
local file and match it against the remote one.

Said that, I had never touched opus in my life :) so take my comments
with care.

--

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
Hello.

> Some streams requires you set its cursor to 0 before writing/reading buffer, so you need to check it:

> OutPipe.Seek(0, soBeginning);
> InPipe.Seek(0, soBeginning);
> InPipe.Read(BufferURL[0],PipeBufferSize);

=====>

"Exception at 000000000043A720: EPipeSeek:
Cannot seek on pipes."

> So to resume, you file is a raw opus audio or an Ogg file ?

Huh, I do not know. It was converted from a mp3.
Localy, from a hard-disk:

Using op_test_file("test.opus") + op_open_test() is working perfectly.

IMO, the problem comes from abuffer in op_test_memory(abuffer,lengthb).

I am not sure that a array of byte is what op_test_memory want.
It is not a pipe (like lot of other audio lib use), if it is not a array of byte, what does he want...?

> Said that, I had never touched opus in my life :)

So I am not the only one. ;-)

PS: On Opus forum, there is only one topic this year. Without answer. (And this topic is from me !).

Fre;D







Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
Hello.

Some other news from the front:

var
  BufferURL, buffadd: tbytes;
...
  setlength(BufferURL, PipeBufferSize);
  setlength(buffadd, PipeBufferSize);
  CreatePipeHandles(InHandle, OutHandle, PipeBufferSize);
  InPipe := TInputPipeStream.Create(InHandle);
  OutPipe := TOutputPipeStream.Create(OutHandle);
  httpget := TThreadHttpGetter.Create(url, OutPipe);

PipeBufferSize := 1024;
len2 := 0;
len := 1;

while (len2 < PipeBufferSize) and (len > 0) do
  begin
 len := InPipe.Read(buffadd[0],PipeBufferSize);
  if len > 0 then  for i := 0 to len -1 do
  BufferURL[i+len2] := buffadd[i] ;
     len2 := len2 + len;
   end;

 HandleOP := op_test_memory(BufferURL[0],PipeBufferSize, Err);
 
    if Err=0
    then begin
         Err := op_test_open(HandleOP);
....
==============================>
...
Try to connect to https://sites.google.com/site/fredvsbinaries/guit_kungs.opus
error: op_test_open: 0
OK open
OpusTag.comments = 6
ALBUM=Summer 2016
ARTIST=Kungs
DATE=2016
DESCRIPTION=Loop Guitar.
GENRE=Dance
TITLE=guit_kungs.opus
op_bitrate = 2147483647
Data.Channels 2

===========>

Yep, yep, yep, connected and retrieving the tag + infos ok.

The trick was to do a loop with InPipe.Read and add buffadd to BufferURL.

So, good, there is one step more.

Sadly, op_read_float(HandleOP,@Buffer[0], Wantframesl) does not work.
I suspect that some read callback must be define for reading.

There is a method for opening steams with custom callback:

OP_WARN_UNUSED_RESULT OggOpusFile *op_test_callbacks(void *_source,
 const OpusFileCallbacks *_cb,const unsigned char *_initial_data,
 size_t _initial_bytes,int *_error) OP_ARG_NONNULL(2);

Translated with this:

op_test_callbacks: function(const source; const cb: TOpusFileCallbacks; const initial_data; initial_bytes: cuint;
  out error: cint): TOpusFile;

But sadly I do not understand what is source and not sure what is initial_data.
Also I do not understand how to deal with cb and assign read (to InPipe.Read ?)

TOpusFileCallbacks = record
    read: op_read_func;
    seek: op_seek_func;
    tell: op_tell_func;
    close: op_close_func;
  end;

This does not work:

HandleOP := op_test_callbacks(pointer(InPipe),op_callbacks, BufferURL[0], PipeBufferSize, err);  
 
Huh, once again, your advices will be extremely appreciated.

Thanks.

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

noreply
In reply to this post by fredvs
On Fri, January 27, 2017 7:54 am, fredvs wrote:
>
> I am not sure that a array of byte is what op_test_memory want.
> It is not a pipe (like lot of other audio lib use), if it is not a array
> of byte, what does he want...?
>


C doesn't even really have arrays since it is so low level (well this is
disputed). Often an API function that accepts a buffer could allow
multiple types of buffers, that you allocate yourself... Make sure the
function doesn't modify the buffer size (bad api practice) and make you
allocate the memory

It just seems like this is similar to windows API programming where the
windows C api would allow a fixed array of byte (or similar), or your own
allocated heap array.... i.e. whatever you want.

Advantage of heap array is you can change it; fixed static stack array is
fixed and not modifiable on heap... But make sure the api doesn't muck
with the buffer on it's end or try to allocate memory for it, as that
should be your job. But you never know with some api's. This should all be
in their docs? They don't have proper docs?

But I know nothing of Opus, sorry. (and can you look into the actual
source code of opus or is this closed source?)
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
Hello Lars.

Thanks for your brilliant light.

>  (and can you look into the actual source code of opus or is this closed source?)

==> https://github.com/xiph/opus

And translation into fpc-Pascal:

==> https://github.com/fredvs/uos/blob/master/src/uos_opusfile.pas

Fre;D
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

José Mejuto
In reply to this post by fredvs
El 28/01/2017 a las 13:32, fredvs escribió:

> TOpusFileCallbacks = record
>     read: op_read_func;
>     seek: op_seek_func;
>     tell: op_tell_func;
>     close: op_close_func;
>   end;
>
> This does not work:
>
> HandleOP := op_test_callbacks(pointer(InPipe),op_callbacks, BufferURL[0],
> PipeBufferSize, err);
>

Hello,

As you need to use records you should specify {$PACKRECORDS C} or you
could get a different record layout than C expects.

--

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
>  you should specify {$PACKRECORDS C}

Nice tip !

Added in each unit that deal with c libraries.

Many hanks.

Fre;D

PS: But this did not help ;-(
Many thanks ;-)
Reply | Threaded
Open this post in threaded view
|

Re: Pipe vs Memory buffer.

fredvs
In reply to this post by fredvs
@ Lars:

The library used for op_memory_open and op_seek is libopusfile.so:
https://github.com/xiph/opusfile

Translation into fpc-Pascal of the header:
https://github.com/fredvs/uos/blob/master/src/uos_opusfile.pas

And Opusfile.so  depends on libopus.so :
https://github.com/xiph/opus

Translation into fpc-Pascal of the header:
https://github.com/fredvs/uos/blob/master/src/uos_opus.pas

Fre;D


Many thanks ;-)
12