Add API to official documentation search

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Add API to official documentation search

Free Pascal - General mailing list
Could FPC make an API for the official documentation search at https://www.freepascal.org/docsearch/docsearch.var so we could make calls like:

https://www.freepascal.org/docsearch/docsearch.var?word=List

and get back a JSON object that had the search results? This is important for integration with 3rd party IDE's.


Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Graeme Geldenhuys-6
On 17/11/2019 4:29 pm, Ryan Joseph via fpc-pascal wrote:
> https://www.freepascal.org/docsearch/docsearch.var?word=List
>
> and get back a JSON object that had the search results? This is
> important for integration with 3rd party IDE's.

No such API exists (that I know of), but the desire to integrate the
FPC, LCL, fpGUI and any other API documentation into other IDE's and
Programming Editors has existed for over a decade (by me at least). I
also needed the requirement of having the help available offline,
because I often worked in locations with no internet access. I'm of the
rare breed of developers that actually reads documentation. ;-)

That's why I invested months of research in various help formats and
eventually spent over a year developing a help viewer that could easily
integrate into IDE's and Editors, and even be used from the command line
or via IPC calls. The help viewer is now the default help viewer of
fpGUI, and is called DocView.

It used the INF help format (initially developed by IBM), which is
extremely compact, really fast, supports full text search. DocView takes
full advantage of all that and can concatenate multiple help files and
their Tables of Content at runtime, supports advanced searching,
bookmarks, inline annotation and more.

Details on integrating Docview in various IDE's and editors:

   http://fpgui.sourceforge.net/docview_ide_integration.shtml



Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Graeme Geldenhuys-6
In reply to this post by Free Pascal - General mailing list
On 17/11/2019 4:29 pm, Ryan Joseph via fpc-pascal wrote:
> at https://www.freepascal.org/docsearch/docsearch.var 

Good to see Free Pascal now actually has searchable online docs (without
the help of Internet Search Engines). That's a big step up from a few
years ago. Well done to whomever was involved in that!

Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Michael Van Canneyt


On Sun, 17 Nov 2019, Graeme Geldenhuys wrote:

> On 17/11/2019 4:29 pm, Ryan Joseph via fpc-pascal wrote:
>> at https://www.freepascal.org/docsearch/docsearch.var 
>
> Good to see Free Pascal now actually has searchable online docs (without
> the help of Internet Search Engines). That's a big step up from a few
> years ago. Well done to whomever was involved in that!

That would be me.

And there already is an API. How else ? This is Free Pascal !


If you look in the browser developer console when using that page,
you'll see there are 2 calls available which do return JSON.

for dropdown completion:

https://www.freepascal.org/docsearch/docsearch.cgi/list?t=contains&m=1&q=as
https://www.freepascal.org/docsearch/docsearch.cgi/list?t=contains&m=1&q=al

q is the query parameter. m=1 requests metadata. m=0 omits metadata.

For actual search there is

https://www.freepascal.org/docsearch/docsearch.cgi/search?m=1&q=equals

again, search term is in 'q'. m has the same meaning.

The JSON is straightforward.

But I will document the API.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Michael Van Canneyt
In reply to this post by Free Pascal - General mailing list


On Sun, 17 Nov 2019, Ryan Joseph via fpc-pascal wrote:

> Could FPC make an API for the official documentation search at https://www.freepascal.org/docsearch/docsearch.var so we could make calls like:
>
> https://www.freepascal.org/docsearch/docsearch.var?word=List
>
> and get back a JSON object that had the search results? This is important for integration with 3rd party IDE's.

See my response to Graeme.

Even Microsoft asked official permission to use our website as pascal reference.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Graeme Geldenhuys-6
On 17/11/2019 10:22 pm, Michael Van Canneyt wrote:
> Even Microsoft asked official permission to use our website as pascal reference.

For what need? VS Code?

Regards,
  Graeme

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Graeme Geldenhuys-6
In reply to this post by Michael Van Canneyt
On 17/11/2019 10:20 pm, Michael Van Canneyt wrote:
>
> That would be me.

Thought that much. ;-)


> And there already is an API. How else ? This is Free Pascal !
>
>
> If you look in the browser developer console when using that page,
> you'll see there are 2 calls available which do return JSON.

My bad, I initially thought Ryan was simply quoting the FPC docs URL
with a "wish for a search URL". It's only after I sent my first reply
that I actually clicked on the URL and noticed the search box. Very nice
to see it available now.

How is the search data generated - via fpdoc? What storage format does
it use?
I remember years ago I found a JavaScript search engine that could be
integrated with fpdoc's HTML output, but that storage format was
JavaScript (if I remember correctly) and was huge.


Regards,
  Graeme

--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

My public PGP key:  http://tinyurl.com/graeme-pgp
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Michael Van Canneyt
In reply to this post by Graeme Geldenhuys-6


On Sun, 17 Nov 2019, Graeme Geldenhuys wrote:

> On 17/11/2019 10:22 pm, Michael Van Canneyt wrote:
>> Even Microsoft asked official permission to use our website as pascal reference.
>
> For what need? VS Code?

Stated purpose was special handling in the Bing search engine.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Michael Van Canneyt
In reply to this post by Graeme Geldenhuys-6


On Sun, 17 Nov 2019, Graeme Geldenhuys wrote:

> On 17/11/2019 10:20 pm, Michael Van Canneyt wrote:
>>
>> That would be me.
>
> Thought that much. ;-)
>
>
>> And there already is an API. How else ? This is Free Pascal !
>>
>>
>> If you look in the browser developer console when using that page,
>> you'll see there are 2 calls available which do return JSON.
>
> My bad, I initially thought Ryan was simply quoting the FPC docs URL
> with a "wish for a search URL". It's only after I sent my first reply
> that I actually clicked on the URL and noticed the search box. Very nice
> to see it available now.
>
> How is the search data generated - via fpdoc? What storage format does
> it use?

Postgres database. the data is generated once from the HTML using fpindex.
All is in SVN. packages/fpindex.

I will of course need to update the database when we release 3.2 and I update the
docs.

> I remember years ago I found a JavaScript search engine that could be
> integrated with fpdoc's HTML output, but that storage format was
> JavaScript (if I remember correctly) and was huge.

Well, that would also have been an option, but I thought it would be nice to
showcase that FPC has all the tools you need for whatever task you set it

Never miss a chance to show off what FPC can do ;-)

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Graeme Geldenhuys-6
In reply to this post by Michael Van Canneyt
On 17/11/2019 10:31 pm, Michael Van Canneyt wrote:
> Stated purpose was special handling in the Bing search engine.

Either way, that's a pie in the face for Embarcadero. :-)

Regards,
  Graeme

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Free Pascal - General mailing list
On 11/17/2019 2:41 PM, Graeme Geldenhuys wrote:
> On 17/11/2019 10:31 pm, Michael Van Canneyt wrote:
>> Stated purpose was special handling in the Bing search engine.
> Either way, that's a pie in the face for Embarcadero. :-)
>
They might rather get a chuckle out of it, after all, it is Bing. And as
you should know, friends don't let friends use Bing! :P

Ralf ;-)

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Free Pascal - General mailing list
In reply to this post by Graeme Geldenhuys-6
On Sunday, November 17, 2019, Graeme Geldenhuys <[hidden email]> wrote:
Either way, that's a pie in the face for Embarcadero. :-)

They probably asked Embarcadero the same question for Delphi language reference

thanks,
Dmitry 

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Free Pascal - General mailing list
In reply to this post by Michael Van Canneyt
On Sun, 17 Nov 2019 23:35:34 +0100 (CET), Michael Van Canneyt
<[hidden email]> wrote:

>I will of course need to update the database when we release 3.2 and I update the
>docs.
>

I had a look at it too and searched for baseunix fpopen etc.
Found this:
https://www.freepascal.org/docs-html/rtl/baseunix/fpopen.html

Look at the example, specifically the top comment of the example:

{ Program to demonstrate the fdOpen, fdwrite and fdCLose functions. }

Is this a typo or indication of early, later changed naming
convention?
Should be fp at the start of the names rather than fd, right?


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Michael Van Canneyt


On Mon, 18 Nov 2019, Bo Berglund via fpc-pascal wrote:

> On Sun, 17 Nov 2019 23:35:34 +0100 (CET), Michael Van Canneyt
> <[hidden email]> wrote:
>
>> I will of course need to update the database when we release 3.2 and I update the
>> docs.
>>
>
> I had a look at it too and searched for baseunix fpopen etc.
> Found this:
> https://www.freepascal.org/docs-html/rtl/baseunix/fpopen.html
>
> Look at the example, specifically the top comment of the example:
>
> { Program to demonstrate the fdOpen, fdwrite and fdCLose functions. }
>
> Is this a typo or indication of early, later changed naming
> convention?

Changed naming convention when switching from 'linux' unit to more general
*unix units, more than 15 years ago.

> Should be fp at the start of the names rather than fd, right?

For baseunix, yes. I have fixed it. Thanks for pointing it out.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Free Pascal - General mailing list
In reply to this post by Michael Van Canneyt


> On Nov 17, 2019, at 5:20 PM, Michael Van Canneyt <[hidden email]> wrote:
>
> That would be me.
>
> And there already is an API. How else ? This is Free Pascal !

Sorry for the late response.

Nice this is exactly what I wanted. However when I started to look at the actual results I'm confused. For my example of searching for "list" here is the first result:


FCL units reference tfpobjectlist.html
Return the first non-nil object in the list

What is this exactly? I expected to get results like from https://docs.getlazarus.org where it would say this is a class or a method etc... and a description of the class. I had been using their API (Anthony is responsible for the project I think) but its got bugs and I haven't been able to contact the developer to fix them.

What if I want to see the reference for TStringList? If I enter that term I get lots of results but all I have to go on is the "context" which doesn't really tell me what the result is how to filter it.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Michael Van Canneyt


On Mon, 18 Nov 2019, Ryan Joseph via fpc-pascal wrote:

>
>
>> On Nov 17, 2019, at 5:20 PM, Michael Van Canneyt <[hidden email]> wrote:
>>
>> That would be me.
>>
>> And there already is an API. How else ? This is Free Pascal !
>
> Sorry for the late response.
>
> Nice this is exactly what I wanted.  However when I started to look at the
> actual results I'm confused.  For my example of searching for "list" here
> is the first result:
>
>
> FCL units reference tfpobjectlist.html
> Return the first non-nil object in the list

> What is this exactly?  I expected to get results like from
> https://docs.getlazarus.org where it would say this is a class or a method
> etc...  and a description of the class.  I had been using their API
> (Anthony is responsible for the project I think) but its got bugs and I
> haven't been able to contact the developer to fix them.

You get a list of pages that contain the search term. They are listed in
order of amount of hits in the page.

>
> What if I want to see the reference for TStringList?  If I enter that term
> I get lots of results but all I have to go on is the "context" which
> doesn't really tell me what the result is how to filter it.

If you want exact matches on class names or identifiers, then there is no such API.

Such an API can be conceived, but does not exist at present.
Should be easy enough to do, I suppose.

As with all things FPC, the sources are available:

svn co https://svn.freepascal.org/svn/html/docsearch docsearch

Feel free to suggest improvements. It should be easy enough to add a first
element that contains an exact match on one or more identifiers by looking
for a file with the correct name, given an initial directory.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Free Pascal - General mailing list


> On Nov 18, 2019, at 1:15 PM, Michael Van Canneyt <[hidden email]> wrote:
>
> As with all things FPC, the sources are available:
>
> svn co https://svn.freepascal.org/svn/html/docsearch docsearch
>
> Feel free to suggest improvements. It should be easy enough to add a first
> element that contains an exact match on one or more identifiers by looking
> for a file with the correct name, given an initial directory.

Where are the doc pages generated from? For example https://www.freepascal.org/docs-html/rtl/classes/tstrings.html has a nice header "Class to manage arrays or collections of strings" and also at the top-right: Reference for unit "classes". There is even a "Description" section which is pretty good to know.

Can you not get those back from the search results? Sorry if it's obvious but I don't know how you did the search.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Free Pascal - General mailing list
In reply to this post by Free Pascal - General mailing list
You said, "but its got bugs and I haven't been able to contact the developer to fix them"

What are the bugs you want fixed? 

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Add API to official documentation search

Free Pascal - General mailing list


> On Nov 20, 2019, at 5:25 AM, Anthony Walter via fpc-pascal <[hidden email]> wrote:
>
> You said, "but its got bugs and I haven't been able to contact the developer to fix them"
>
> What are the bugs you want fixed?

There you are. ;) There are stray tags which are messing things up. For example:

https://docs.getlazarus.org/?method=codesearch&format=xml&phrase=List

notice all the <b> tags that are breaking up the titles.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal