Prorammatically sending file using HTTP POST?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Prorammatically sending file using HTTP POST?

Bo Berglund
I have written a Lazarus/FPC program to configure an IoT device via a
tcp server I have implemented on the device. It all works well for
setting the parameters etc.
Now I want to also add a firmware update function to this configurator
program, but I could use some advice regarding the way to send the
file data...

The firmware updater service is running on the IoT device on port 80
as a web server and when one connects to that URL with a browser the
following happens:

1) A login dialog is shown where a username and a password needs to be
entered.

2) Next a very simple web form is shown with the following source:
<html>
<body>
<form method='POST' action='' enctype='multipart/form-data'>
   <input type='file' name='update'>
   <input type='submit' value='Update'>
</form>
</body>
</html>

3) When the "Browse..." button is clicked one gets to navigate the
file system and select a file to upload. The name is then displayed in
a box.

4) Upon hitting the submit button "Update" the file is sent to the IoT
device and used to update the firmware.

Now I want to get rid of the web browser step and integrate the
functionality of the form into the configuration program, but I don't
know how to actually send the POST data to the IoT device so it will
work...

The instructions for the WebUpdater service I have added to the IoT
device has a second alternative for uploading the firmware using curl:

curl -u user:pwd -F "image=@firmware.bin" <ip-address>:80/firmware

I don't want to require that curl is present on the PC where my
program will run, though, so I want to implement the actual post
message in my fpc code.

Does someone have a suggestion on how to do this?

By the way, I am using the Indy10 components via the indylaz package.
Lazarus is 1.8.0 and FPC 3.0.4 on Windows7.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Luca Olivetti-2
El 04/04/18 a les 13:50, Bo Berglund ha escrit:

>
> Does someone have a suggestion on how to do this?
>
> By the way, I am using the Indy10 components via the indylaz package.
> Lazarus is 1.8.0 and FPC 3.0.4 on Windows7.

I never used indy, but you can either use TFPHTTPClient (unit
fphttpclient) or synapse THHPSend.
Both allow to send form data.
Disregard my suggestion if you prefer to use indy.

Bye
--
Luca

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Bo Berglund
On Wed, 4 Apr 2018 15:29:36 +0200, Luca Olivetti
<[hidden email]> wrote:

>> By the way, I am using the Indy10 components via the indylaz package.
>> Lazarus is 1.8.0 and FPC 3.0.4 on Windows7.
>
>I never used indy, but you can either use TFPHTTPClient (unit
>fphttpclient) or synapse THHPSend.
>Both allow to send form data.
>Disregard my suggestion if you prefer to use indy.

Yes, I use Indy since the Delphi days but I have not used it to send
files before.
Indy has a TIdHTTP object/component, which presumably can be used but
there are no good examples I have found that show how to use it for
simulating POST from a form like the one I showed...


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Michael Van Canneyt


On Wed, 4 Apr 2018, Bo Berglund wrote:

> On Wed, 4 Apr 2018 15:29:36 +0200, Luca Olivetti
> <[hidden email]> wrote:
>
>>> By the way, I am using the Indy10 components via the indylaz package.
>>> Lazarus is 1.8.0 and FPC 3.0.4 on Windows7.
>>
>> I never used indy, but you can either use TFPHTTPClient (unit
>> fphttpclient) or synapse THHPSend.
>> Both allow to send form data.
>> Disregard my suggestion if you prefer to use indy.
>
> Yes, I use Indy since the Delphi days but I have not used it to send
> files before.
> Indy has a TIdHTTP object/component, which presumably can be used but
> there are no good examples I have found that show how to use it for
> simulating POST from a form like the one I showed...

I don't know what a 'good example' is for you, but the code in

https://stackoverflow.com/questions/27257577/indy-mime-decoding-of-multipart-form-data-requests-returns-trailing-cr-lf

Is quite understandable.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Bo Berglund
On Wed, 4 Apr 2018 17:09:49 +0200 (CEST), Michael Van Canneyt
<[hidden email]> wrote:

>I don't know what a 'good example' is for you, but the code in
>
>https://stackoverflow.com/questions/27257577/indy-mime-decoding-of-multipart-form-data-requests-returns-trailing-cr-lf
>
>Is quite understandable.

I found this example too, but it seems to be dealing with *server*
side stuff...

I am trying to create a *client* function which will send a POST
command like a browser would when I click the submit button on the
simple form I showed the source for.

It works fine if I use the web browser to connect and load the form,
then select the file and finally hit the submit button.
But since I already have all of the other device config items inside
an FPC program I wanted the firmware upload to also be part of it...

This is how far I have gotten with the UploadFirmware method:

function TConfigCommHandler.UploadFirmware(FileName: string; URL:
string): boolean;
var
  HTTP: TIdHTTP;
  Src: TIdMultipartFormDataStream;
begin
  Result := false;
  if not FileExists(FileName) then exit;
  HTTP := TIdHTTP.Create;
  Src := TIdMultipartFormDataStream.Create;
  try
    Src.AddFormField('?',?);
    Src.AddFile(FileName,'?');
    try
      HTTP.Post(URL, Src, ?);
      Result := true;
    except
      on E: Exception do
        FLastError := 'Exception: ' + E.Message;
    end;
  finally
    Src.Free;;
    HTTP.Free;
  end;
end;

It is all about formulating the correct parameters towards the
HTTP.Post function.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Anthony Walter-3
HTTP is quite simple actually. You just build a post message (a string), and send it over a socket. I've implemented HTTP client functions in many computer languages. Here are the key take aways if you want to implement it yourself.

Build this string filling out the blanks using your own values

POST {location} HTTP 1.1{linefeed}
Host: {domain}{linefeed}
Content-Length: {length}{linefeed}
{linefeed}
{content}

So to post the word 'hello' to http://example.com/yourpage?say, where the domain is example.com, and location is yourpage?say, the string would look like this in pascal.

Message := 
  'POST /yourpage?say HTTP 1.1'#10 +
  'Host: example.com'#10 +
  'Content-Length: 5'#10 +
  #10 +
  'hello';

Then you use some socket component, set the Host to 'example.com' so that it can resolve the ip address, and call Send(Message). 

This assumes you have a tcp client socket component with properties Host, and a method called Send, but most any/every socket component does.

If you want to post form named arguments in content, then just format them like:

Content := 'words=hello&priority=high&account=123';
ContentLength := IntToStr(Length(Content));

Message := 
  'POST /yourpage?say HTTP 1.1'#10 +
  'Host: example.com'#10 +
  'Content-Length: ' + ContentLength + #10 +
  #10 +
  Content;

Socket.Host := 'example.com';
Socket.Send(Message);

It's that easy.





_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Anthony Walter-3
Just a follow up, if care you get a response back from your POST, as in the server says okay, you just use the same socket to read. Also, you might need to call connect after you set the host:

var
  Message: string;
  Response: string;
...
// port 80 normally used for http
Socket.Connect('example.com', 80);
Socket.Send(Message);
Socket.Read(Response);
WriteLn(Response);

If the server accepted your POST, the response will start with 'HTTP 1.1 200 OK'#10, followed by some other headers. Socket Read might be named Receive or Recv, depending on your client socket implementation of choice.

If you want to learn may, you may want to example my THttpClient object:


And also, here is an example of how to write an object oriented socket class that wraps the traditional socket apis on all platforms:


If you look at that source you can also see how simple it is to add https support to your socket.

Socket.Secure := True;
Socket.Connect('example.com', 443);

And finally, you don't need to use my github code. anyAclient socket class for Pascal should work approximately the same.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Graeme Geldenhuys-6
In reply to this post by Bo Berglund
On 2018-04-04 16:05, Bo Berglund wrote:
> Yes, I use Indy since the Delphi days but I have not used it to send
> files before.

Yes, there is no problem in using Indy with FPC - I've done it for
years. So no need to switch components.

> Indy has a TIdHTTP object/component, which presumably can be used but
> there are no good examples

Have you actually tried asking the question in the official Indy support
newsgroup?

  NNTP Server:  news.atozed.com
  Groups:  atozedsoftware.indy.general
           atozedsoftware.indy.fpc
           atozedsoftware.indy.protocol.*
           atozedsoftware.indy.non-tech
           ... and many others....

I've always gotten very quick responses from them.

Have you tried this?

    http://lmgtfy.com/?q=delphi+indy+sending+file+via+http+post

I see many answers listed there. ;-)


Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Graeme Geldenhuys-6
On 2018-04-07 10:13, Graeme Geldenhuys wrote:
> Have you tried this?
>
>     http://lmgtfy.com/?q=delphi+indy+sending+file+via+http+post
>
> I see many answers listed there. ;-)


And if you are really lazy... ;-)




Indy has TIdMultipartFormDataStream for this purpose:

procedure TForm1.SendPostData;
var
  Stream: TStringStream;
  Params: TIdMultipartFormDataStream;
begin
  Stream := TStringStream.Create('');
  try
   Params := TIdMultipartFormDataStream.Create;
   try
    Params.AddFile('File1', 'C:\test.txt','application/octet-stream');
    try
     HTTP.Post('http://posttestserver.com/post.php', Params, Stream);
    except
     on E: Exception do
       ShowMessage('Error encountered during POST: ' + E.Message);
    end;
    ShowMessage(Stream.DataString);
   finally
    Params.Free;
   end;
  finally
   Stream.Free;
  end;
end;


That will sent a file to the PHP script on the server.



Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Graeme Geldenhuys-6
On 2018-04-07 10:20, Graeme Geldenhuys wrote:
> And if you are really lazy... ;-)

And here's an even simpler version.


-------- [ from Remy Lebeau ]-------------------------


TIdHTTP has two overloaded versions of Post() that take a filename as input:

var
  Response: String;

Response :=
IdHTTP1.Post('https://www.example.com/ex/exampleAPI.asmx/Process',
'c:\filename.txt');

....or....

var
  Response: TStream;

Response := TMemoryStream.Create;
IdHTTP1.Post('https://www.example.com/ex/exampleAPI.asmx/Process',
'c:\filename.txt', Response);
...
Response.Free;

Note that you are posting to an HTTPS URL, so you need to first assign
an SSL-enabled IOHandler, such as TIdSSLIOHandlerSocketOpenSSL, to the
TIdHTTP.IOHandler property beforehand.

-------------------[ end ]------------------------

Can't get simpler than that!

Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Bo Berglund
In reply to this post by Graeme Geldenhuys-6
On Sat, 7 Apr 2018 10:13:06 +0100, Graeme Geldenhuys
<[hidden email]> wrote:

>Have you actually tried asking the question in the official Indy support
>newsgroup?
>
>I've always gotten very quick responses from them.
>
Well, I asked in the Embarcadero forum (using XanaNews nttp
interface...) in the internet.winsock subforum and I got a reply from
Remy Lebeau (as always helpful).

So I think I am good on the POST item, but I have not been able to
test yet due to travels. Will try my hand on that shortly.

I can post the final solution here after I have completed the Lazarus
config program with the new function and verified that it works.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Alexander Grotewohl
In reply to this post by Graeme Geldenhuys-6
Those newsgroups are a horrid suggestion. The most recent messages are
from 2016. It seems they only leave it up for posterity.


On 04/07/2018 05:13 AM, Graeme Geldenhuys wrote:

> On 2018-04-04 16:05, Bo Berglund wrote:
>> Yes, I use Indy since the Delphi days but I have not used it to send
>> files before.
> Yes, there is no problem in using Indy with FPC - I've done it for
> years. So no need to switch components.
>
>> Indy has a TIdHTTP object/component, which presumably can be used but
>> there are no good examples
> Have you actually tried asking the question in the official Indy support
> newsgroup?
>
>    NNTP Server:  news.atozed.com
>    Groups:  atozedsoftware.indy.general
>             atozedsoftware.indy.fpc
>             atozedsoftware.indy.protocol.*
>             atozedsoftware.indy.non-tech
>             ... and many others....
>
> I've always gotten very quick responses from them.
>
> Have you tried this?
>
>      http://lmgtfy.com/?q=delphi+indy+sending+file+via+http+post
>
> I see many answers listed there. ;-)
>
>
> Regards,
>    Graeme
>

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Graeme Geldenhuys-6
On 2018-04-07 15:32, Alexander Grotewohl wrote:
> Those newsgroups are a horrid suggestion. The most recent messages are
> from 2016. It seems they only leave it up for posterity.

I got personally assured by Remy Lebeau that the NNTP newsgroups are
checked just as frequently as the Web Forums. The two are not synced,
but both are active options for Indy support.

ps:
  My last question in the Indy newsgroup (*.protocol.imap4 group) was in
  2017 and if you looked closer, you will see those got answered the
  same day.

ps #2:
  I much prefer NNTP newsgroups to the much slower and often ad-driven
  web forums, or mailing lists. Plus I can easily download all messages
  (dating back to 2006) for offline viewing and searching with my
  favourite NNTP client of choice.


Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Bo Berglund
On Sat, 7 Apr 2018 16:09:29 +0100, Graeme Geldenhuys
<[hidden email]> wrote:

>  I much prefer NNTP newsgroups to the much slower and often ad-driven
>  web forums, or mailing lists. Plus I can easily download all messages
>  (dating back to 2006) for offline viewing and searching with my
>  favourite NNTP client of choice.

Same here!
(btw I dropped off this topic because I had a fall from the roof of my
house and am currently hospitalized. Got my laptop yesterday
though...)


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

Klaus Hartnegg-3
In reply to this post by Bo Berglund
Am 04.04.2018 um 13:50 schrieb Bo Berglund:
> By the way, I am using the Indy10 components via the indylaz package.
> Lazarus is 1.8.0 and FPC 3.0.4 on Windows7.

If anybody wants to do this just with winsock:


const
   dstport = 80;
   http_path = '/post.php';

type
   Int32 = Longint;


function wordstr (value:word) : string;
var
   c : string;
begin
   str (value,c);
   wordstr := c;
end;


procedure SendHttpPost (c:ansistring;
                         var msg:string);

var
   wsadata : TWSAData;  // winsock

var
   sock : TSocket;
   addr : sockets.sockaddr_in;
   numbytes : Int32;
   sendstring : ansistring;
   rc : longint;
   ok : boolean;

begin
   // initialize winsock
   if (WSAStartup($0202, wsadata) <> 0) then begin
      msg := 'cannot initialize winsock';
      exit;
   end;

   // get socket
   sock := fpSocket(AF_INET, SOCK_STREAM, 0);
   if (sock = -1) then begin
      msg := 'cannot retrieve socket';
      exit;
   end;


   addr.sin_family := AF_INET;
   addr.sin_port := htons(dstport);
   GetServerIP (addr.sin_addr, ok);

   rc := fpConnect (sock, @addr, sizeof(addr));
   if rc <> 0 then begin
      msg := 'cannot connect to host';
      exit;
   end;

   // c := stringreplace (c,' ','&',[rfReplaceAll]);

   // send data
   sendstring := 'POST '+http_path+' HTTP/1.1'+#13+#10
               + 'host: '+NetAddrToStr(addr.sin_addr)+#13+#10
               + 'Content-Type: application/x-www-form-urlencoded'+#13+#10
               + 'Content-Length: '+wordstr(length(c))+#13+#10
               + #13+#10
               + c;

   numbytes := fpSend(sock, @sendstring[1], length(sendstring), 0);
   if (numbytes = SOCKET_ERROR) then begin
      msg := 'cannot send data to host';
   end else begin
      msg := 'Sent '+wordstr(numbytes)+' bytes to
'+NetAddrToStr(addr.sin_addr));
   end;

   // clean up
   closesocket(sock);

   // shut down winsock
   WSACleanup;
end;
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Prorammatically sending file using HTTP POST?

R0b0t1
In reply to this post by Bo Berglund
On Wed, Apr 4, 2018 at 6:50 AM, Bo Berglund <[hidden email]> wrote:

> I have written a Lazarus/FPC program to configure an IoT device via a
> tcp server I have implemented on the device. It all works well for
> setting the parameters etc.
> Now I want to also add a firmware update function to this configurator
> program, but I could use some advice regarding the way to send the
> file data...
>
> The firmware updater service is running on the IoT device on port 80
> as a web server and when one connects to that URL with a browser the
> following happens:
>
> 1) A login dialog is shown where a username and a password needs to be
> entered.
>
> 2) Next a very simple web form is shown with the following source:
> <html>
> <body>
> <form method='POST' action='' enctype='multipart/form-data'>
>    <input type='file' name='update'>
>    <input type='submit' value='Update'>
> </form>
> </body>
> </html>
>
> 3) When the "Browse..." button is clicked one gets to navigate the
> file system and select a file to upload. The name is then displayed in
> a box.
>
> 4) Upon hitting the submit button "Update" the file is sent to the IoT
> device and used to update the firmware.
>
> Now I want to get rid of the web browser step and integrate the
> functionality of the form into the configuration program, but I don't
> know how to actually send the POST data to the IoT device so it will
> work...
>
> The instructions for the WebUpdater service I have added to the IoT
> device has a second alternative for uploading the firmware using curl:
>
> curl -u user:pwd -F "image=@firmware.bin" <ip-address>:80/firmware
>

I'd strongly recommend using SSH/SCP instead, or at least HTTPS.
Certificates for HTTPS can be more difficult to set up on nontechnical
user's computers.

Cheers,
     R0b0t1

> I don't want to require that curl is present on the PC where my
> program will run, though, so I want to implement the actual post
> message in my fpc code.
>
> Does someone have a suggestion on how to do this?
>
> By the way, I am using the Indy10 components via the indylaz package.
> Lazarus is 1.8.0 and FPC 3.0.4 on Windows7.
>
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal