JSON Test Suite

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

JSON Test Suite

Graeme Geldenhuys-6
Hi,

I just stubled acros this JSON test suite. There graph shows plenty of
different JSON parsers and how they compare with this test suite. The
Free Pascal JSON parser was not in that list.

Has anybody tried the FPC JSON parser with these tests?

  https://github.com/nst/JSONTestSuite

This should catch a lot of corner cases.

Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Michael Van Canneyt


On Sun, 30 Oct 2016, Graeme Geldenhuys wrote:

> Hi,
>
> I just stubled acros this JSON test suite. There graph shows plenty of
> different JSON parsers and how they compare with this test suite. The
> Free Pascal JSON parser was not in that list.
>
> Has anybody tried the FPC JSON parser with these tests?
>
>  https://github.com/nst/JSONTestSuite
>
> This should catch a lot of corner cases.

I have not tested it, but patches pertaining to this will surely be
accepted.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

leledumbo
Administrator
In reply to this post by Graeme Geldenhuys-6
> Has anybody tried the FPC JSON parser with these tests?

I just did with the following program:

{$mode objfpc}
uses
  Classes,SysUtils,fpjson,jsonparser;
var
  Input: TStringList;
  JSON: TJSONData;
begin
  try
    try
      Input := TStringList.Create;
      Input.LoadFromFile(ParamStr(1));
      JSON := GetJSON('[""],');
      ExitCode := 0;
    finally
      JSON.Free;
      Input.Free;
    end;
  except
    on e: EParserError do begin
      ExitCode := 1;
    end;
    on e: Exception do begin
      ExitCode := 2;
    end;
  end;
end.

Result is not very convincing:
parsing.html

I test some of the bad cases, indeed fpjson parser is too permissive.
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

leledumbo
Administrator
Whoops, it looks like a CSS is missing: style.css
Plus, I compare it with Go 1.7's  encoding/json package.
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Graeme Geldenhuys-6
In reply to this post by leledumbo
On 2016-10-30 13:48, leledumbo wrote:
>
> I just did with the following program:

Awesome, thanks.


> Result is not very convincing:

In comparison to Go, no it isn't. I guess we have some work to do.


Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Michael Van Canneyt
In reply to this post by leledumbo


On Sun, 30 Oct 2016, leledumbo wrote:

>> Has anybody tried the FPC JSON parser with these tests?
>
> I just did with the following program:
>
> {$mode objfpc}
> uses
>  Classes,SysUtils,fpjson,jsonparser;
> var
>  Input: TStringList;
>  JSON: TJSONData;
> begin
>  try
>    try
>      Input := TStringList.Create;
>      Input.LoadFromFile(ParamStr(1));
>      JSON := GetJSON('[""],');

Ehm. This is not parsing anything parametrizable?


I had a look at some of the constructs.
Some of them are not meant to be supported.

I did some random test, for instance

n_array_1_true_without_comma.json : [1 true]

FPC reports an error, as expected.
In your HTML, it is in color for 'parsing should have failed but succeeded'

So I have some doubts as to your results...

Michael.

This is the program I used:

{$mode objfpc}
uses
   Classes,SysUtils,fpjson,jsonparser;

var
   Input: TStream;
   JSON: TJSONData;

begin
   JSON:=Nil;
   INPUT:=Nil;
   ExitCode:=0;
   try
     try
       Input := TFileStream.Create(ParamStr(1),fmOpenRead or fmShareDenyWrite);
       JSON := GetJSON(Input);
     finally
       JSON.Free;
       Input.Free;
     end;
   except
     on e: EParserError do begin
       ExitCode := 1;
     end;
     on e: Exception do begin
       ExitCode := 2;
     end;
   end;
end.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Michael Van Canneyt
In reply to this post by Graeme Geldenhuys-6


On Sun, 30 Oct 2016, Graeme Geldenhuys wrote:

> On 2016-10-30 13:48, leledumbo wrote:
>>
>> I just did with the following program:
>
> Awesome, thanks.
>
>
>> Result is not very convincing:
>
> In comparison to Go, no it isn't. I guess we have some work to do.

Well, things may not be so bad...

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

leledumbo
Administrator
In reply to this post by Graeme Geldenhuys-6
> In comparison to Go, no it isn't. I guess we have some work to do.

I mean, if fcl-json wants to be on top. In comparison with others, we're among the second top lines (good cases mostly correct, bad cases a bit behind). At least fcl-json is way above 3 of Perl's parsers and Haskell which crashes even on good cases. Here's the complete comparison:
parsing.html

> Ehm. This is not parsing anything parametrizable?

My bad, I copy-paste-d the modified program for ensuring that it's the parser that really accepts the broken JSON. I think you could easily understand that the hardcoded parameter should be Input.Text :p
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

leledumbo
Administrator
> I mean, if fcl-json wants to be on top. In comparison with others, we're among the second top lines (good cases mostly correct, bad cases a bit behind).

To make us proud, for the good cases, there are only 2 parsers that fully accept the test suite: java's nanojson & fcl-json
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Michael Van Canneyt
In reply to this post by leledumbo


On Sun, 30 Oct 2016, leledumbo wrote:

>> In comparison to Go, no it isn't. I guess we have some work to do.
>
> I mean, if fcl-json wants to be on top. In comparison with others, we're
> among the second top lines (good cases mostly correct, bad cases a bit
> behind). At least fcl-json is way above 3 of Perl's parsers and Haskell
> which crashes even on good cases. Here's the complete comparison:
> parsing.html
> <http://free-pascal-general.1045716.n5.nabble.com/file/n5726718/parsing.html>
>
>> Ehm. This is not parsing anything parametrizable?
>
> My bad, I copy-paste-d the modified program for ensuring that it's the
> parser that really accepts the broken JSON. I think you could easily
> understand that the hardcoded parameter should be Input.Text :p

No problem. What about the false positive I demonstrated ?

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

leledumbo
Administrator
> No problem. What about the false positive I demonstrated ?

I guess we'll need to talk (open issue?) to the author for that. I have no idea if it's really should be supported or just a misunderstanding from the author.

Anyway, FPC fcl-json is now merged into the repository:
https://github.com/nst/JSONTestSuite/commit/c9f84db0f130c9eedb87218e34a2a4a1298097cd
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Michael Van Canneyt


On Mon, 31 Oct 2016, leledumbo wrote:

>> No problem. What about the false positive I demonstrated ?
>
> I guess we'll need to talk (open issue?) to the author for that. I have no
> idea if it's really should be supported or just a misunderstanding from the
> author.

Nono: the test is correct.
fcl-json gives an error, but in your test result it shows up as accepting it ?

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

leledumbo
Administrator
> Nono: the test is correct.
> fcl-json gives an error, but in your test result it shows up as accepting it ?

Eh? Will need to recheck. The requirement is clear:

- 0 for accept
- 1 for reject
- >1 for crash

Could it be one of the two except block is wrongly executed?
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Michael Van Canneyt


On Tue, 1 Nov 2016, leledumbo wrote:

>> Nono: the test is correct.
>> fcl-json gives an error, but in your test result it shows up as accepting
>> it ?
>
> Eh? Will need to recheck. The requirement is clear:
>
> - 0 for accept
> - 1 for reject
> - >1 for crash

I understood that. I got 1 for the indicated test.

Which is correct, it must be rejected.

So I don't think except blocks are to blame.

Michael.


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Marco van de Voort
In reply to this post by leledumbo
In our previous episode, leledumbo said:

> > Nono: the test is correct.
> > fcl-json gives an error, but in your test result it shows up as accepting
> > it ?
>
> Eh? Will need to recheck. The requirement is clear:
>
> - 0 for accept
> - 1 for reject
> - >1 for crash
>
> Could it be one of the two except block is wrongly executed?

Too late, it already is in the news :-)

http://www.theregister.co.uk/2016/11/01/json_parsers_tested/
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Benito van der Zander
In reply to this post by Graeme Geldenhuys-6
Hi,

it is better to use TJSONParser than GetJSON.

GetJSON is defined as:

procedure DefJSONParserHandler(AStream: TStream; const AUseUTF8: Boolean; out
  Data: TJSONData);

Var
  P : TJSONParser;

begin
  Data:=Nil;
  P:=TJSONParser.Create(AStream,[joUTF8]);
  try
    Data:=P.Parse;
  finally
    P.Free;
  end;
end;

with

procedure DefJSONParserHandler(AStream: TStream; const AUseUTF8: Boolean; out
  Data: TJSONData);

Var
  P : TJSONParser;

begin
  Data:=Nil;
  P:=TJSONParser.Create(AStream,[joStrict,joUTF8]);
  try
    Data:=P.Parse;
  finally
    P.Free;
  end;
end;

it should pass more


Best,

Benito 




_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: JSON Test Suite

Benito van der Zander

Hi,
huh, why does the build suddenly fail?

Well, FPC 3.0.0 does not have the joStrict option.
It is only there in 3.1.1.

Perhaps there should be two fpc parser entries.

Best,
Benito 


On 06.11.2016 17:34, Benito van der Zander wrote:
Hi,

it is better to use TJSONParser than GetJSON.

GetJSON is defined as:

procedure DefJSONParserHandler(AStream: TStream; const AUseUTF8: Boolean; out
  Data: TJSONData);

Var
  P : TJSONParser;

begin
  Data:=Nil;
  P:=TJSONParser.Create(AStream,[joUTF8]);
  try
    Data:=P.Parse;
  finally
    P.Free;
  end;
end;

with

procedure DefJSONParserHandler(AStream: TStream; const AUseUTF8: Boolean; out
  Data: TJSONData);

Var
  P : TJSONParser;

begin
  Data:=Nil;
  P:=TJSONParser.Create(AStream,[joStrict,joUTF8]);
  try
    Data:=P.Parse;
  finally
    P.Free;
  end;
end;

it should pass more


Best,

Benito 





_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal