presenting fpunit test results

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

presenting fpunit test results

Vincent Snijders
Hi,

I have created some tests with fpunit and now I want to present the test
results on a html page like
http://hccweb1.bai.ne.jp/tsune-1/NUnitTestResult.html

Graeme Geldenhuys provided me with some sample xsl files and css files,
but what seemed to be missing in those is timing results and a complete
listing of all executed tests and results per suite. I don't know how to
program stylesheets, so my question is:

* Has anybody else converted the xml results to html?
* Can somebody help with the creation of the html page?

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Ewald Horn
Hi Vincent,

most of what I do is programs that output HTML results. I'm sure I can be of
some assistance to you. Can you zip and send the unit test results to me so
I can make you a small sample of what I have in mind ?

Kind regards
Ewald Horn
Web: www.nofuss.co.za


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
Ewald Horn wrote:
> Hi Vincent,
>
> most of what I do is programs that output HTML results. I'm sure I can
> be of some assistance to you. Can you zip and send the unit test results
> to me so I can make you a small sample of what I have in mind ?
>

Thanks for your offer.

In the coming days, I will send you the xml files in private.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
In reply to this post by Vincent Snijders
Hi Vincent,

> Graeme Geldenhuys provided me with some sample xsl files and css files,
> but what seemed to be missing in those is timing results and a complete
> listing of all executed tests and results per suite. I don't know how to

Oh, I left out the processing of the executed test result.... That
could explain why they are missing.  :-)
The sample xml file I used didn't have much to report on, so I left it out.

If you can give me a snippet of the XML file (node names and
hierarchy) I will amend the XSLT file for you.

Regards,
  - Graeme -
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
In reply to this post by Vincent Snijders
Hi Vincent,

Is is easy to add the list of executed tests (5 min job), but the
missing feature I believe you are looking for is the timing results.
Currently none of the fpcunit results contain timing values.  This is
something I wanted to add long ago.  The GUI TestRunner does show the
total time taken, but that is only displayed, not saved anywere.

I will see how far I get with adding timing result today... if done, I
will submit a patch in fpc-devel.

Regards,
  - Graeme -


On 9/7/06, Vincent Snijders <[hidden email]> wrote:
> Graeme Geldenhuys provided me with some sample xsl files and css files,
> but what seemed to be missing in those is timing results and a complete
> listing of all executed tests and results per suite. I don't know how to
> program stylesheets, so my question is:


--
There's no place like 127.0.0.1
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Dean Zobec
Graeme Geldenhuys ha scritto:

> Hi Vincent,
>
> Is is easy to add the list of executed tests (5 min job), but the
> missing feature I believe you are looking for is the timing results.
> Currently none of the fpcunit results contain timing values.  This is
> something I wanted to add long ago.  The GUI TestRunner does show the
> total time taken, but that is only displayed, not saved anywere.
>
> I will see how far I get with adding timing result today... if done, I
> will submit a patch in fpc-devel.
>
> Regards,
>  - Graeme -
At first sight, it seems that the methods StartTest and EndTest of the
listeners are the right place where to place the code to do the timings.
I guess the right approach would be to add an option at command line or
otherwise to add the timing result. Same for the list of the run tests.

Successive steps that are natural is the automation of fpcunit testing:
a daemon that checks some svn directories periodically to find if there
are changes in the tests, does the update, compiles the tests and
publishes the tests (as an html page or sends the results to some
registered users via  e-mail, irc or othere means). I'm investigating
how to best achieve this, I would use lnet for the networking part.

Ciao, Dean




_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
Dean Zobec schreef:

> Graeme Geldenhuys ha scritto:
>> Hi Vincent,
>>
>> Is is easy to add the list of executed tests (5 min job), but the
>> missing feature I believe you are looking for is the timing results.
>> Currently none of the fpcunit results contain timing values.  This is
>> something I wanted to add long ago.  The GUI TestRunner does show the
>> total time taken, but that is only displayed, not saved anywere.
>>
>> I will see how far I get with adding timing result today... if done, I
>> will submit a patch in fpc-devel.
>>
>> Regards,
>>  - Graeme -
> At first sight, it seems that the methods StartTest and EndTest of the
> listeners are the right place where to place the code to do the timings.
> I guess the right approach would be to add an option at command line or
> otherwise to add the timing result. Same for the list of the run tests.
>
> Successive steps that are natural is the automation of fpcunit testing:
> a daemon that checks some svn directories periodically to find if there
> are changes in the tests, does the update, compiles the tests and
> publishes the tests (as an html page or sends the results to some
> registered users via  e-mail, irc or othere means). I'm investigating
> how to best achieve this, I would use lnet for the networking part.

I would use command line tools like svn and scp to do the network part.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
In reply to this post by Dean Zobec
On 9/8/06, Dean Zobec <[hidden email]> wrote:
> At first sight, it seems that the methods StartTest and EndTest of the
> listeners are the right place where to place the code to do the timings.
> I guess the right approach would be to add an option at command line or
> otherwise to add the timing result. Same for the list of the run tests.


Thanks for the pointers Dean.   I didn't get time today, but should
over the weekend.


Regards,
  - Graeme -
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
Graeme Geldenhuys wrote:

> On 9/8/06, Dean Zobec <[hidden email]> wrote:
>
>> At first sight, it seems that the methods StartTest and EndTest of the
>> listeners are the right place where to place the code to do the timings.
>> I guess the right approach would be to add an option at command line or
>> otherwise to add the timing result. Same for the list of the run tests.
>
>
>
> Thanks for the pointers Dean.   I didn't get time today, but should
> over the weekend.

The tests are in Lazarus svn in the test subdirectory.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
In reply to this post by Graeme Geldenhuys-2
Graeme Geldenhuys schreef:
> On 9/8/06, Dean Zobec <[hidden email]> wrote:
>> At first sight, it seems that the methods StartTest and EndTest of the
>> listeners are the right place where to place the code to do the timings.
>> I guess the right approach would be to add an option at command line or
>> otherwise to add the timing result. Same for the list of the run tests.
>
>
> Thanks for the pointers Dean.   I didn't get time today, but should
> over the weekend.

Any progres?

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
Sorry, I got side tracked with work requirements. Who decided there
must only be 24 hours in a day! :-)

I already have most of the timing results implemented, just need to
rework the way the Text TestRunner uses the XML output. I will finish
it in this week.

Regards,
  - Graeme -


On 19/09/06, Vincent Snijders <[hidden email]> wrote:

> Graeme Geldenhuys schreef:
> > On 9/8/06, Dean Zobec <[hidden email]> wrote:
> >> At first sight, it seems that the methods StartTest and EndTest of the
> >> listeners are the right place where to place the code to do the timings.
> >> I guess the right approach would be to add an option at command line or
> >> otherwise to add the timing result. Same for the list of the run tests.
> >
> >
> > Thanks for the pointers Dean.   I didn't get time today, but should
> > over the weekend.
>
> Any progres?
>
> Vincent
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>


--
There's no place like 127.0.0.1
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
In reply to this post by Vincent Snijders
Vincent Snijders wrote:

> Ewald Horn wrote:
>
>> Hi Vincent,
>>
>> most of what I do is programs that output HTML results. I'm sure I can
>> be of some assistance to you. Can you zip and send the unit test
>> results to me so I can make you a small sample of what I have in mind ?
>>
>
> Thanks for your offer.
>

A sample xml file with test results can be found on
http://www.hu.freepascal.org/lazarus/testresults/results.xml

What is missing in these files in the test hierarchy:
A future version of the results file will probably include it, so it
results.html can show first totals for testsuite, then total for
examples, etc.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
Vincent Snijders wrote:

> A sample xml file with test results can be found on
> http://www.hu.freepascal.org/lazarus/testresults/results.xml
>
> What is missing in these files in the test hierarchy:

forgot to paste the link :-(
http://www.hu.freepascal.org/lazarus/testresults/tree.xml

> A future version of the results file will probably include it, so it
> results.html can show first totals for testsuite, then total for
> examples, etc.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
In reply to this post by Vincent Snijders
Hi Vincent,

I haven't forgotten about your updated .xslt file to take into account
the timing results. I have been really busy at work, and have a
deadline approaching this Friday. After that, I should have a bit more
time...

In the mean time, I have been updating my Linux build server for the
tiOPF project, which also uses fpcUnit.  I am modifying the .xslt file
I gave you for my own usage as well. The tiOPF project has over 1500
tests, so I have a lot of data in the results.xml file to play with.
As soon as I have the .xslt & .css files complete, I will send you a
copy.  Maybe we can even add it to the fpcUnit examples in SubVersion.

Does FPC come with a unit that can process xslt files and save the
generated result to a file?

Regards,
  - Graeme -


On 24/09/06, Vincent Snijders <[hidden email]> wrote:

> Vincent Snijders wrote:
> > Ewald Horn wrote:
> >
> >> Hi Vincent,
> >>
> >> most of what I do is programs that output HTML results. I'm sure I can
> >> be of some assistance to you. Can you zip and send the unit test
> >> results to me so I can make you a small sample of what I have in mind ?
> >>
> >
> > Thanks for your offer.
> >
>
> A sample xml file with test results can be found on
> http://www.hu.freepascal.org/lazarus/testresults/results.xml
>
> What is missing in these files in the test hierarchy:
> A future version of the results file will probably include it, so it
> results.html can show first totals for testsuite, then total for
> examples, etc.
>
> Vincent
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>


--
There's no place like 127.0.0.1
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
In reply to this post by Vincent Snijders
Hi Vincent,

Attached is my latest efforts on fpcUnit Test Reports using XSLT.
Before, I sent you two versions of the .xsl file.  One created a
report based on CSS only and the other one based on HTML Tables only.
I tested both against my results (around 900 tests) and the CSS
version was dog slow!!

I then created from scratch a .xsl file generating HTML Tables (small
ones) and uses a small CSS file to format (colors, fonts, etc) the
generated HTML.  The speed is *much* better.

Attached is a archive containing the CSS, XSL and sample results
files.  I also included a small script file (fixup_xml.sh) which
inserts a single line into the xml results file, to specify  the xslt
stylesheet to use.

The reason for the script file, is because I didn't what to modify
fpcUnit to insert that line automaticaly, as I  like to generate
different formatted results from the original xml file and need to
insert different XSLT stylesheets for that.  For example, I generate
two different outputs from the same results.xml file. One for HTML
results going to my web server and the other, a plain text email,
which gets emailed to a newsgroup showing only the summary of the
results.

This version is much better than the ones I sent you before.  A few
things are still missing though, but that is due to the data being
missing in the xml results.

* The test listings need to be grouped by test suites which will allow
for a better summaries to be generated.
* More attributes need to be added to the test listings to give
information like: Did the test pass, fail or give a error.  This will
allow me to link the summary test listing to the detailed results
using the HTML version of the XSLT.
* Ignored/Skipped tests still need support - My version of fpcUnit has
not been patched yet.

Enjoy, and please give me some feedback. Good or bad!

Regards,
  - Graeme -



On 19/09/06, Vincent Snijders <[hidden email]> wrote:

> Graeme Geldenhuys schreef:
> > On 9/8/06, Dean Zobec <[hidden email]> wrote:
> >> At first sight, it seems that the methods StartTest and EndTest of the
> >> listeners are the right place where to place the code to do the timings.
> >> I guess the right approach would be to add an option at command line or
> >> otherwise to add the timing result. Same for the list of the run tests.
> >
> >
> > Thanks for the pointers Dean.   I didn't get time today, but should
> > over the weekend.
>
> Any progres?
>
> Vincent
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>

--
There's no place like 127.0.0.1

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

fpcUnit_Report.tar.gz (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
Graeme Geldenhuys schreef:

> Hi Vincent,
>
> Attached is my latest efforts on fpcUnit Test Reports using XSLT.
> Before, I sent you two versions of the .xsl file.  One created a
> report based on CSS only and the other one based on HTML Tables only.
> I tested both against my results (around 900 tests) and the CSS
> version was dog slow!!
>
> I then created from scratch a .xsl file generating HTML Tables (small
> ones) and uses a small CSS file to format (colors, fonts, etc) the
> generated HTML.  The speed is *much* better.
>
> Attached is a archive containing the CSS, XSL and sample results
> files.  I also included a small script file (fixup_xml.sh) which
> inserts a single line into the xml results file, to specify  the xslt
> stylesheet to use.

I like the result:
http://www.hu.freepascal.org/lazarus/testresults/results.xml

>
> The reason for the script file, is because I didn't what to modify
> fpcUnit to insert that line automaticaly, as I  like to generate
> different formatted results from the original xml file and need to
> insert different XSLT stylesheets for that.  For example, I generate
> two different outputs from the same results.xml file. One for HTML
> results going to my web server and the other, a plain text email,
> which gets emailed to a newsgroup showing only the summary of the
> results.

The console testrunner, that I use, will insert the stylesheet information.

>
> This version is much better than the ones I sent you before.  A few
> things are still missing though, but that is due to the data being
> missing in the xml results.
>
> * The test listings need to be grouped by test suites which will allow
> for a better summaries to be generated.

This would be my first feature request.

> * More attributes need to be added to the test listings to give
> information like: Did the test pass, fail or give a error.  This will
> allow me to link the summary test listing to the detailed results
> using the HTML version of the XSLT.
> * Ignored/Skipped tests still need support - My version of fpcUnit has
> not been patched yet.
>
> Enjoy, and please give me some feedback. Good or bad!

Looking forward to further improvements.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
On 02/10/06, Vincent Snijders <[hidden email]> wrote:

> I like the result:
> http://www.hu.freepascal.org/lazarus/testresults/results.xml

I couldn't view it. Got an error that the results.xsl file couldn't be
found. Note: I renamed the results.xsl file to fpcunit.xsl, so you
will have to update your 2nd line in the results.xml file to point to
the right xsl file.  Damn, that sounds confusing! :-)

> The console testrunner, that I use, will insert the stylesheet information.

I didn't want to amend mine, for in case I accidentally send it in a
patch. Also I needed two result output formats.


> > * The test listings need to be grouped by test suites which will allow
> > for a better summaries to be generated.
>
> This would be my first feature request.

Mine too.  Coming up shortly...


Regards,
  - Graeme -
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Vincent Snijders
Graeme Geldenhuys schreef:
> On 02/10/06, Vincent Snijders <[hidden email]> wrote:
>
>> I like the result:
>> http://www.hu.freepascal.org/lazarus/testresults/results.xml
>
> I couldn't view it. Got an error that the results.xsl file couldn't be
> found. Note: I renamed the results.xsl file to fpcunit.xsl, so you
> will have to update your 2nd line in the results.xml file to point to
> the right xsl file.  Damn, that sounds confusing! :-)

I already did that and tested that. Maybe the browser is showing you an old version?

>
>> The console testrunner, that I use, will insert the stylesheet
>> information.
>
> I didn't want to amend mine, for in case I accidentally send it in a
> patch. Also I needed two result output formats.

Make it a command line parameter.

>
>
>> > * The test listings need to be grouped by test suites which will allow
>> > for a better summaries to be generated.
>>
>> This would be my first feature request.
>
> Mine too.  Coming up shortly...
>

Great.

Vincent
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
On 02/10/06, Vincent Snijders <[hidden email]> wrote:
>
> I already did that and tested that. Maybe the browser is showing you an old version?

Ah yes, it view my browser cache. Refresh fixed it. :-)

> >> The console testrunner, that I use, will insert the stylesheet
> >> information.
> >
> > I didn't want to amend mine, for in case I accidentally send it in a
> > patch. Also I needed two result output formats.
>
> Make it a command line parameter.

Nope I didn't want to, because then I need to re-run all my tests
again, just to get another output format.  Well, I guess I could add
and just not use it. That way I could still insert my own stylesheets
on a single test run.

Oh, here is my results for tiOPF2. I haven't enabled all tests yet.
Those are only half of them.  See the reason why I want to fix the
Test Listing nodes.  Also notice that my summary is Red, not Green,
due to the Error count > 0.

http://opensoft.homeip.net/tiopf/fpcunit/results.xml

Graeme.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: presenting fpunit test results

Graeme Geldenhuys-2
In reply to this post by Graeme Geldenhuys-2
Hi Vincent,

Small bugfix in the fpcunit.xsl file.  On line 58, the $ sign was
missing infront of the "failureCount' variable.  This caused the
Summary line (when failures occured and no errors) to show up green
instead of magenta.

Line 58 should look as follows:

   <xsl:when test="$failureCount &gt; 0">


Regards,
  - Graeme -


On 01/10/06, Graeme Geldenhuys <[hidden email]> wrote:

> Hi Vincent,
>
> Attached is my latest efforts on fpcUnit Test Reports using XSLT.
> Before, I sent you two versions of the .xsl file.  One created a
> report based on CSS only and the other one based on HTML Tables only.
> I tested both against my results (around 900 tests) and the CSS
> version was dog slow!!
>
> I then created from scratch a .xsl file generating HTML Tables (small
> ones) and uses a small CSS file to format (colors, fonts, etc) the
> generated HTML.  The speed is *much* better.
>
> Attached is a archive containing the CSS, XSL and sample results
> files.  I also included a small script file (fixup_xml.sh) which
> inserts a single line into the xml results file, to specify  the xslt
> stylesheet to use.
>
> The reason for the script file, is because I didn't what to modify
> fpcUnit to insert that line automaticaly, as I  like to generate
> different formatted results from the original xml file and need to
> insert different XSLT stylesheets for that.  For example, I generate
> two different outputs from the same results.xml file. One for HTML
> results going to my web server and the other, a plain text email,
> which gets emailed to a newsgroup showing only the summary of the
> results.
>
> This version is much better than the ones I sent you before.  A few
> things are still missing though, but that is due to the data being
> missing in the xml results.
>
> * The test listings need to be grouped by test suites which will allow
> for a better summaries to be generated.
> * More attributes need to be added to the test listings to give
> information like: Did the test pass, fail or give a error.  This will
> allow me to link the summary test listing to the detailed results
> using the HTML version of the XSLT.
> * Ignored/Skipped tests still need support - My version of fpcUnit has
> not been patched yet.
>
> Enjoy, and please give me some feedback. Good or bad!
>
> Regards,
>   - Graeme -
>
>
>
> On 19/09/06, Vincent Snijders <[hidden email]> wrote:
> > Graeme Geldenhuys schreef:
> > > On 9/8/06, Dean Zobec <[hidden email]> wrote:
> > >> At first sight, it seems that the methods StartTest and EndTest of the
> > >> listeners are the right place where to place the code to do the timings.
> > >> I guess the right approach would be to add an option at command line or
> > >> otherwise to add the timing result. Same for the list of the run tests.
> > >
> > >
> > > Thanks for the pointers Dean.   I didn't get time today, but should
> > > over the weekend.
> >
> > Any progres?
> >
> > Vincent
> > _______________________________________________
> > fpc-pascal maillist  -  [hidden email]
> > http://lists.freepascal.org/mailman/listinfo/fpc-pascal
> >
>
>
> --
> There's no place like 127.0.0.1
>
>
>


--
There's no place like 127.0.0.1
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal