BSON formatter

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

BSON formatter

Free Pascal - General mailing list
Working on the language server it's become clear that JSON is going to be a killer bottleneck, but that's what Microsoft decided to use. It's almost a deal breaker just because of the sheer size of the data due to JSON's verbose plain text formatting.

I've seen there is such thing as BSON (https://en.wikipedia.org/wiki/BSON) which could be implemented over JSON without disrupting the underlying architecture. Is there any BSON formatters for the FPC JSON libraries? It's just serializing in a  different text format so it's probably pretty easy to implement but I thought I would ask first.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: BSON formatter

Michael Van Canneyt


On Thu, 7 May 2020, Ryan Joseph via fpc-pascal wrote:

> Working on the language server it's become clear that JSON is going to be a killer bottleneck, but that's what Microsoft decided to use. It's almost a deal breaker just because of the sheer size of the data due to JSON's verbose plain text formatting.
>
> I've seen there is such thing as BSON (https://en.wikipedia.org/wiki/BSON)
> which could be implemented over JSON without disrupting the underlying
> architecture.  Is there any BSON formatters for the FPC JSON libraries?
> It's just serializing in a  different text format so it's probably pretty
> easy to implement but I thought I would ask first.

There currently isn't but it has been on my todo list for ages. Patches welcome :)

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: BSON formatter

Andreas Schneider-7
In reply to this post by Free Pascal - General mailing list
Am 2020-05-07 08:40, schrieb Ryan Joseph via fpc-pascal:

> Working on the language server it's become clear that JSON is going to
> be a killer bottleneck, but that's what Microsoft decided to use. It's
> almost a deal breaker just because of the sheer size of the data due
> to JSON's verbose plain text formatting.
>
> I've seen there is such thing as BSON
> (https://en.wikipedia.org/wiki/BSON) which could be implemented over
> JSON without disrupting the underlying architecture. Is there any BSON
> formatters for the FPC JSON libraries? It's just serializing in a
> different text format so it's probably pretty easy to implement but I
> thought I would ask first.

I would go with ProtoBuf for something like this. If you really want to
be flexibel in the datastructure, look at msgpack/messagepack. Both
(protobuf and msgpack) are pretty good in benchmarks.

--
Andreas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: BSON formatter

Michael Van Canneyt


On Thu, 7 May 2020, Andreas Schneider wrote:

> Am 2020-05-07 08:40, schrieb Ryan Joseph via fpc-pascal:
>> Working on the language server it's become clear that JSON is going to
>> be a killer bottleneck, but that's what Microsoft decided to use. It's
>> almost a deal breaker just because of the sheer size of the data due
>> to JSON's verbose plain text formatting.
>>
>> I've seen there is such thing as BSON
>> (https://en.wikipedia.org/wiki/BSON) which could be implemented over
>> JSON without disrupting the underlying architecture. Is there any BSON
>> formatters for the FPC JSON libraries? It's just serializing in a
>> different text format so it's probably pretty easy to implement but I
>> thought I would ask first.
>
> I would go with ProtoBuf for something like this. If you really want to
> be flexibel in the datastructure, look at msgpack/messagepack. Both
> (protobuf and msgpack) are pretty good in benchmarks.

The LSP protocol requires JSON, so it's not an option to use something else.
I'm not even sure BSON will be accepted for LSP.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: BSON formatter

Free Pascal - General mailing list


> On May 7, 2020, at 4:44 PM, Michael Van Canneyt <[hidden email]> wrote:
>
> I'm not even sure BSON will be accepted for LSP.

I talked to them already and they're not keen to add extensions to the plugin so I'll probably give up on this route. The spec from Microsoft doesn't say anything about this either so maybe it's a bottle neck others have hit. Sending plain text is horribly stupid and Microsoft doesn't seem to be taking performance seriously. If they want to improve performance there is other low hanging fruit but it requires Microsoft to extend the protocol.

The amount of redundant information being sent around is ridiculous. Mainly, the spec gives no way for the client to cache results and refer to them later in various requests. We're literally sending around mega bytes of data for no reason whatsoever, except lazy programmers I guess.

Computers just keep getting slower and slower sometimes it feels...

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: BSON formatter

Andreas Schneider-7
Am 2020-05-07 15:26, schrieb Ryan Joseph via fpc-pascal:
> The amount of redundant information being sent around is ridiculous.
> Mainly, the spec gives no way for the client to cache results and
> refer to them later in various requests. We're literally sending
> around mega bytes of data for no reason whatsoever, except lazy
> programmers I guess.
>
> Computers just keep getting slower and slower sometimes it feels...

That's the normal approach now. Why should you optimize anything if
you can simply buy better hardware? So I end up with a machine ten
times as fast as a few years ago and still cannot run much more in
parallel than back then, because even calendar apps are now written
with a full browser stack (looking at electron here).

As someone who still is happy when a GUI is a statically compiled
app that is < 10MB and doesn't take more than that as memory when
starting up, I am disgusted by the current "state of the art"
development. It's no longer an art.

(I am even more pissed off about this on my smartphone. Why does
a messenger app have to be 100 MB in size?! We had whole operating
systems in just 1% of that size. Even a full oracle database is
just around 30 MB if you strip away the java based management
console stuff ... and that thing is a feature monster for data
management.)

</rant>

--
Andreas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: BSON formatter

Michael Van Canneyt
In reply to this post by Free Pascal - General mailing list


On Thu, 7 May 2020, Ryan Joseph via fpc-pascal wrote:

>
>
>> On May 7, 2020, at 4:44 PM, Michael Van Canneyt <[hidden email]> wrote:
>>
>> I'm not even sure BSON will be accepted for LSP.
>
> I talked to them already and they're not keen to add extensions to the
> plugin so I'll probably give up on this route.  The spec from Microsoft
> doesn't say anything about this either so maybe it's a bottle neck others
> have hit.  Sending plain text is horribly stupid and Microsoft doesn't
> seem to be taking performance seriously.  If they want to improve
> performance there is other low hanging fruit but it requires Microsoft to
> extend the protocol.

At least partial updates to a file should be there.
Sending the whole file at essentially every few keystrokes is just braindead...

>
> The amount of redundant information being sent around is ridiculous.
> Mainly, the spec gives no way for the client to cache results and refer to
> them later in various requests.  We're literally sending around mega bytes
> of data for no reason whatsoever, except lazy programmers I guess.

Indeed.

I was doing some tests in VS Code with omnipascal; The Omnipascal LSP server process takes 100% cpu.
No surprises there....


>
> Computers just keep getting slower and slower sometimes it feels...

Welcome to IT in 2020:
waiting, staring at a wait cursor... And more waiting....  :-)

Like I said:
This is probably one of the reasons for the current trend towards huge amounts of really small files.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: BSON formatter

Free Pascal - General mailing list


> On May 7, 2020, at 9:11 PM, Michael Van Canneyt <[hidden email]> wrote:
>
> At least partial updates to a file should be there.
> Sending the whole file at essentially every few keystrokes is just braindead...
>

That's still happening but it's Sublime Texts fault for a bad API. It is fixed in the current development branch though (ST4).

The problem is workspace symbols. For my project there are 40k symbols (system SDKs I like to browse easily) which translates to 10MB of JSON text.

It's so stupid it makes my feel ill and I weep for my CPU. I'm going to submit an issue to Microsoft since they at least allow this on GitHub.

>>
>> The amount of redundant information being sent around is ridiculous. Mainly, the spec gives no way for the client to cache results and refer to
>> them later in various requests.  We're literally sending around mega bytes
>> of data for no reason whatsoever, except lazy programmers I guess.
>
> Indeed.
>
> I was doing some tests in VS Code with omnipascal; The Omnipascal LSP server process takes 100% cpu.
> No surprises there....

I've been talking to the LSP people at ST and they tell me it's really common for servers to be written in JavaScript because it's easily supported within VSCode's architecture (which Electron I'm sure you know). They may have written the thing in JavaScript for all I know. That's a problem right there as far as performance is concerned.

Regards,
        Ryan Joseph

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal