Using fphttpclient to simulate web action sequence

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Using fphttpclient to simulate web action sequence

leledumbo
Administrator
I need to create some kind of bot for automatic testing against our web app. The following app target real online web application:

{$mode objfpc}{$H+}

uses
  {$ifdef unix}
  cthreads,
  {$endif}
  Classes,SysUtils,fphttpclient;

procedure Log(http: TFPHTTPClient);
begin
  WriteLn('BEGIN **********************************************************');
  WriteLn(http.RequestHeaders.Text);
  WriteLn(http.ResponseStatusCode);
  WriteLn(http.ResponseHeaders.Text);  
  WriteLn('END **********************************************************');
end;

var
  http: TFPHTTPClient;
  formdata: TStrings;
begin
  http := TFPHTTPClient.Create(nil);
  try
    // initial GET request to initialize cookies
    http.Get('http://lelelagi.mysirclo.com');
    Log(http);

    // POST: add item to cart
    formdata := TStringList.Create;
    formdata.Values['cmd'] := 'add';
    formdata.Values['item_id'] := 'montauk-tote';
    formdata.Values['member_email'] := 'leledumbo_cool@yahoo.co.id ';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata); // #1
    Log(http);

    // POST: set country, city and shipping
    formdata.Clear;
    formdata.Values['cmd'] := 'shipping_country';
    formdata.Values['shipping_value'] := 'ID';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata); // #2
    Log(http);

    formdata.Clear;
    formdata.Values['cmd'] := 'shipping_city';
    formdata.Values['shipping_value'] := 'Kota Depok - Depok';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata);
    Log(http);

    formdata.Clear;
    formdata.Values['cmd'] := 'shipping';
    formdata.Values['shipping_value'] := 'JNE REG';
    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata);
    Log(http);

    // POST: place order
    formdata.Clear;
    formdata.Values['first_name'] := 'Test Name';
    formdata.Values['phone'] := '088712342345';
    formdata.Values['address_line1'] := 'Certain Street';
    formdata.Values['country'] := 'ID';
    formdata.Values['state'] := 'DKI Jakarta';
    formdata.Values['city'] := 'DKI Jakarta - Jakarta';
    formdata.Values['postal_code'] := '12424';
    formdata.Values['email'] := 'mail@testmail.co.id';
    formdata.Values['delivery_first_name'] := 'Test Name';
    formdata.Values['delivery_phone'] := '088712342345';
    formdata.Values['delivery_address_line1'] := 'Certain Street';
    formdata.Values['delivery_country'] := 'ID';
    formdata.Values['delivery_state'] := 'DKI Jakarta';
    formdata.Values['delivery_city'] := 'DKI Jakarta - Jakarta';
    formdata.Values['delivery_postal_code'] := '12424';
    formdata.Values['delivery_email'] := 'mail@testmail.co.id';
    formdata.Values['shipping_value'] := 'JNE REG';
    formdata.Values['payment_method'] := 'bank-transfer';
    formdata.Values['message'] := '';
    formdata.Values['agreement'] := 'on';
    http.FormPost('http://lelelagi.mysirclo.com/cart/place_order/guest',formdata);
    Log(http);
  finally
    http.Free;
  end;
end.

The app always dies with "EHTTPClient: Error reading data from socket" after the first FormPost, any of them (e.g. if you comment #1, it will die after #2). Did I miss something? Is this the correct way to make such a simulation?
Reply | Threaded
Open this post in threaded view
|

Re: Using fphttpclient to simulate web action sequence

Michael Van Canneyt


On Wed, 29 Apr 2015, leledumbo wrote:

> I need to create some kind of bot for automatic testing against our web app.
> The following app target real online web application:
>
> {$mode objfpc}{$H+}
>
> uses
>  {$ifdef unix}
>  cthreads,
>  {$endif}
>  Classes,SysUtils,fphttpclient;
>
> procedure Log(http: TFPHTTPClient);
> begin
>  WriteLn('BEGIN
> **********************************************************');
>  WriteLn(http.RequestHeaders.Text);
>  WriteLn(http.ResponseStatusCode);
>  WriteLn(http.ResponseHeaders.Text);
>  WriteLn('END **********************************************************');
> end;
>
> var
>  http: TFPHTTPClient;
>  formdata: TStrings;
> begin
>  http := TFPHTTPClient.Create(nil);
>  try
>    // initial GET request to initialize cookies
>    http.Get('http://lelelagi.mysirclo.com');
>    Log(http);
>
>    // POST: add item to cart
>    formdata := TStringList.Create;
>    formdata.Values['cmd'] := 'add';
>    formdata.Values['item_id'] := 'montauk-tote';
>    formdata.Values['member_email'] := '[hidden email] ';
>    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata); // #1
>    Log(http);
>
>    // POST: set country, city and shipping
>    formdata.Clear;
>    formdata.Values['cmd'] := 'shipping_country';
>    formdata.Values['shipping_value'] := 'ID';
>    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata); // #2
>    Log(http);
>
>    formdata.Clear;
>    formdata.Values['cmd'] := 'shipping_city';
>    formdata.Values['shipping_value'] := 'Kota Depok - Depok';
>    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata);
>    Log(http);
>
>    formdata.Clear;
>    formdata.Values['cmd'] := 'shipping';
>    formdata.Values['shipping_value'] := 'JNE REG';
>    http.FormPost('http://lelelagi.mysirclo.com/cart',formdata);
>    Log(http);
>
>    // POST: place order
>    formdata.Clear;
>    formdata.Values['first_name'] := 'Test Name';
>    formdata.Values['phone'] := '088712342345';
>    formdata.Values['address_line1'] := 'Certain Street';
>    formdata.Values['country'] := 'ID';
>    formdata.Values['state'] := 'DKI Jakarta';
>    formdata.Values['city'] := 'DKI Jakarta - Jakarta';
>    formdata.Values['postal_code'] := '12424';
>    formdata.Values['email'] := '[hidden email]';
>    formdata.Values['delivery_first_name'] := 'Test Name';
>    formdata.Values['delivery_phone'] := '088712342345';
>    formdata.Values['delivery_address_line1'] := 'Certain Street';
>    formdata.Values['delivery_country'] := 'ID';
>    formdata.Values['delivery_state'] := 'DKI Jakarta';
>    formdata.Values['delivery_city'] := 'DKI Jakarta - Jakarta';
>    formdata.Values['delivery_postal_code'] := '12424';
>    formdata.Values['delivery_email'] := '[hidden email]';
>    formdata.Values['shipping_value'] := 'JNE REG';
>    formdata.Values['payment_method'] := 'bank-transfer';
>    formdata.Values['message'] := '';
>    formdata.Values['agreement'] := 'on';
>
> http.FormPost('http://lelelagi.mysirclo.com/cart/place_order/guest',formdata);
>    Log(http);
>  finally
>    http.Free;
>  end;
> end.
>
> The app always dies with "EHTTPClient: Error reading data from socket" after
> the first FormPost, any of them (e.g. if you comment #1, it will die after
> #2). Did I miss something? Is this the correct way to make such a
> simulation?

Yes it is.
If the problem disappears after you do a Free and Create of the http class,
then it means something is left hanging after the call.
In that case I would need to check what is happening.

If the problem persists, then it means there is some mechanism on the server that
is causing this.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using fphttpclient to simulate web action sequence

leledumbo
Administrator
> If the problem disappears after you do a Free and Create of the http class,
then it means something is left hanging after the call.

If I Free and re-Create the class, how can I ensure the request header of subsequent requests contain the same cookies? I need to maintain 2 cookie values (which is given from first GET request in the response header) throughout all requests.

This is a sample output of the program:
BEGIN **********************************************************
[Request Headers]

[Response Status Code]
200
[Response Headers]
Server: nginx
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Vary: Accept-Encoding
Set-Cookie: sirclo_session_id=71e58d13b4124c71a54c0eb6438b8007; path=/
Set-Cookie: cart_code=f654373c175ac6e86da22c7400a146f0; expires=Fri, 29-Apr-2016 16:40:05 GMT; path=/
Pragma: no-cache
Date: Wed, 29 Apr 2015 16:40:05 GMT
X-Page-Speed: ngx_pagespeed 1.8
Cache-Control: max-age=0, no-cache, no-store
Connection: Keep-alive
Via: 1.1 ID-0002262071501332 uproxy-3

END **********************************************************
BEGIN **********************************************************
[Request Headers]
Content-Type: application/x-www-form-urlencoded
Content-Length: 79

[Response Status Code]
302
[Response Headers]
Server: nginx
Date: Wed, 29 Apr 2015 16:40:07 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Set-Cookie: sirclo_session_id=0c9f312bc97c0332105c2accc0feede5; path=/
Location: http://lelelagi.mysirclo.com/cart
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0
Connection: Keep-alive
Via: 1.1 ID-0002262071501332 uproxy-6

END **********************************************************
An unhandled exception occurred at $000000000046DCFB:
EHTTPClient: Error reading data from socket
  $000000000046DCFB
  $000000000046DA9B
  $000000000046F344
  $000000000046FAA8
  $000000000047027E

Each request log is delimited by BEGIN-END line. As you can see, in the first request, the response indicates 2 Set-Cookie:, which must be preserved for subsequent requests. But in the next request, the request header contains no Cookie: line, which is why the server issues another Set-Cookie:, though this time it's only sirclo_session_id issued, the cart_code is not issued again. Still, this is wrong as both must appear instead of just one of them.

P.S.:
1. I guess RequestHeaders.Text doesn't contain Cookie: line, does it?
2. 302 response status code is expected, should not be a problem
Reply | Threaded
Open this post in threaded view
|

Re: Using fphttpclient to simulate web action sequence

Michael Van Canneyt


On Wed, 29 Apr 2015, leledumbo wrote:

>> If the problem disappears after you do a Free and Create of the http class,
> then it means something is left hanging after the call.
>
> If I Free and re-Create the class, how can I ensure the request header of
> subsequent requests contain the same cookies? I need to maintain 2 cookie
> values (which is given from first GET request in the response header)
> throughout all requests.

Just extract the and save the value of the Set-Cookie header ?
I always store the cookies explicitly.

The class doesn't automatically handle cookies.
It doesn't store and retrieve the cookies.
Specifically, if it gets a set-cookie, it will not apply it to the next request.

We can think of a "TCookieJar" class implementation as an option, but for now
you must handle cookies yourself.

> Each request log is delimited by BEGIN-END line. As you can see, in the
> first request, the response indicates 2 Set-Cookie:, which must be preserved
> for subsequent requests. But in the next request, the request header
> contains no Cookie: line, which is why the server issues another
> Set-Cookie:, though this time it's only sirclo_session_id issued, the
> cart_code is not issued again. Still, this is wrong as both must appear
> instead of just one of them.
>
> P.S.:
> 1. I guess RequestHeaders.Text doesn't contain Cookie: line, does it?

It must

> 2. 302 response status code is expected, should not be a problem

I have plans to handle this in TFPHTTPClient.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Using fphttpclient to simulate web action sequence

leledumbo
Administrator
> Just extract the and save the value of the Set-Cookie header ?
> I always store the cookies explicitly.

If that's so, the current Cookies property implementation seems broken. I try to debug and indeed I found that each Set-Cookie line overrides the previous if there are more than one Set-Cookie (procedure DoCookies, FCookies gets cleared at the start).

I try to log Cookies.Text, and for instance:

Set-Cookie: cart_code=f654373c175ac6e86da22c7400a146f0; expires=Fri, 29-Apr-2016 16:40:05 GMT; path=/

results in:
cart_code=f654373c175ac6e86da22c7400a146f0
expires=Fri, 29-Apr-2016 16:40:05 GMT
path=/

which is not right since expires and path are cookie attributes instead of cookie value (I even wonder if TStrings is the right choice to store cookies). Guess I'll have to use Free + re-Create + search and parse Set-Cookie manually then.

> We can think of a "TCookieJar" class implementation as an option

I guess so, but it's better integrated in fphttpclient so it can act as a true browser session instead of just a one time request caller.

> It must

It actually does, but actually one can't prove that by inspecting RequestHeaders.Text BEFORE doing http method, because Cookies gets merged there DURING the http method.

> I have plans to handle this in TFPHTTPClient

I didn't ask if 302 is a problem, it's a statement that 302 is expected and is OK. Does what you say indicates that probably the cause of the stuck error?
Reply | Threaded
Open this post in threaded view
|

Re: Using fphttpclient to simulate web action sequence

Michael Van Canneyt


On Thu, 30 Apr 2015, leledumbo wrote:

>> Just extract the and save the value of the Set-Cookie header ?
>> I always store the cookies explicitly.
>
> If that's so, the current Cookies property implementation seems broken. I
> try to debug and indeed I found that each Set-Cookie line overrides the
> previous if there are more than one Set-Cookie (procedure DoCookies,
> FCookies gets cleared at the start).

I will check this.

>
> I try to log Cookies.Text, and for instance:
>
> Set-Cookie: cart_code=f654373c175ac6e86da22c7400a146f0; expires=Fri,
> 29-Apr-2016 16:40:05 GMT; path=/
>
> results in:
> cart_code=f654373c175ac6e86da22c7400a146f0
> expires=Fri, 29-Apr-2016 16:40:05 GMT
> path=/

I will check this. Sounds like something in the cookie parsing is wrong.

>
> which is not right since expires and path are cookie attributes instead of
> cookie value (I even wonder if TStrings is the right choice to store
> cookies). Guess I'll have to use Free + re-Create + search and parse
> Set-Cookie manually then.
>
>> We can think of a "TCookieJar" class implementation as an option
>
> I guess so, but it's better integrated in fphttpclient so it can act as a
> true browser session instead of just a one time request caller.

That I will definitely not do. I prefer the component to be stateless as much as possible.
Which is why I would separate out the hypothetical TCookieJar.

If you want that you'll need to make a separate class which combines the two into one class.
It you can then also handle saving cookies for later retrieval, which will be
pretty much application specific anyway.

>
>> It must
>
> It actually does, but actually one can't prove that by inspecting
> RequestHeaders.Text BEFORE doing http method, because Cookies gets merged
> there DURING the http method.
>
>> I have plans to handle this in TFPHTTPClient
>
> I didn't ask if 302 is a problem, it's a statement that 302 is expected and
> is OK. Does what you say indicates that probably the cause of the stuck
> error?

I meant that I want to have automated handling of 302 (based on a property to enable/disable this, obviously);
i.e. resubmit the request at the 302 location.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal