Any recommendations for a good in-memory storage type

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Any recommendations for a good in-memory storage type

Graeme Geldenhuys-2
Hi,

While implementing drag-n-drop support in fpGUI, I created a
TfpgMimeData class. This class will hold various data items, and a
mime-type describing each of the data items. Obviously the
TfpgMimeData class should be able to handle any data type. So my
question is, what is a good universal type to store all kinds of data
(strings, images (binary data), sound etc)?

The possible ideas I can think of is the following:

  TByteArray
  Variants
  TMemoryStream

So which one would be more fitting for any mime types, or is there
another option I didn't think of?  Here is a simple usage example of
the TfpgMimeData class - to help put this in perspective:


var
  m: TfpgMimeData;
  d: TfpgDrag;
  a: TfpgDropAction;
begin
      m := TfpgMimeData.Create;
      m.SetData('text/plain', 'My name is Earl');
      m.SetData('text/html', 'My name is <b>Earl</b>');  // text/plain
can actually be created from this automatically
      m.SetData('image/png', MyPNGImage);

      d := TfpgDrag.Create(self);
      d.MimeData := m;
      d.StartDrag([daCopy]);

      // d manages the lifespan of m
      // d will be freed automatically when drag action is complete or cancelled
end;


--
Regards,
  - Graeme -


_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net:8080/fpgui/
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Felipe Monteiro de Carvalho
Personally, I would go for TMemoryStream

--
Felipe Monteiro de Carvalho
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Marco van de Voort
In reply to this post by Graeme Geldenhuys-2
In our previous episode, Graeme Geldenhuys said:
> While implementing drag-n-drop support in fpGUI, I created a
> TfpgMimeData class. This class will hold various data items, and a
> mime-type describing each of the data items. Obviously the
> TfpgMimeData class should be able to handle any data type. So my
> question is, what is a good universal type to store all kinds of data
> (strings, images (binary data), sound etc)?
 
> The possible ideas I can think of is the following:
>
>   TByteArray
>   Variants
>   TMemoryStream

Different choices. The question is if you want some typing (if only runtime)
and easy conversions, or e.g. have the data in a directly writable format.

It depends on what you do most, operating on the data, or not.

e.g. You can easily assign a string to a variant, but for the rest you need to
more or less stream it. Same for e.g. classes.

At the same time, you can very quickly dump bytearrays to disc (and
tmemorystream is only a thin wrapper around one).

So what do you do more?

I would first try a variant.
 
(the example code doesn't perform any action on the data of the mimetype,
so it doesn't really add anything IMHO)

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Michael Van Canneyt
In reply to this post by Felipe Monteiro de Carvalho


On Wed, 22 Sep 2010, Felipe Monteiro de Carvalho wrote:

> Personally, I would go for TMemoryStream
>

That would be my option too.
It's the most flexible.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Graeme Geldenhuys-2
In reply to this post by Marco van de Voort
On 22 September 2010 12:52, Marco van de Voort wrote:
> Different choices. The question is if you want some typing (if only runtime)
> and easy conversions, or e.g. have the data in a directly writable format.

Well, the data needs to be passed via Clipboard or Drag-n-Drop API's
of the underlying system (when used from application to application),
or can bypass those API's when dragging and dropping inside the same
application (where a more efficient data format could be used).

Currently I'm only busy with Xlib (but GDI will follow soon). There
are some conversion support, like I suggested. for example: a HTML
data string can be converted to a Plain Text string, by simply
striping all tags. Images, Sound etc will be passed as a byte array
(at least under Xlib).

TfpgMimeData has the following convenience functions/properties to
automatically search for the mime type and return the associated data,
or if an exact mime type doesn't exist, try and do a conversion of a
similar mime type (as in the case of HTML -> Plain Text). Or when
setting data, use the property to determine what mime type must be
associated with the data being stored.

eg:  TfpgMimeData
         property Urls           (text with URI formatting
         property Text           (text)
         property HTML        (text with tags)
         function Data(mimetype)     (byte array)


> I would first try a variant.

Just for getting the initial DND support to work under Xlib, I limited
myself to text types only, so choose variant as my storage type, but
now that I'm moving to the next phase (multiple data types), I am
considering other storage types because I don't know if variant is
suitable for binary data like sound, images etc.. I'm leaning towards
TMemoryStream now. Just spotting the new incoming messages, Felipe and
Michael seem to share my thoughts on that.


--
Regards,
  - Graeme -


_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net:8080/fpgui/
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Marco van de Voort
In our previous episode, Graeme Geldenhuys said:
> > I would first try a variant.
>
> Just for getting the initial DND support to work under Xlib, I limited
> myself to text types only, so choose variant as my storage type, but
> now that I'm moving to the next phase (multiple data types), I am
> considering other storage types because I don't know if variant is
> suitable for binary data like sound, images etc..

No. But it allows you to keep text as text, not as streamed value. I don't
know if assigning a string to a variant is copy-on-write though.

If you don't operate much on it, it doesn't matter. But the above sounds
like you do most of the work on text, not binary blobs.

> I'm leaning towards
> TMemoryStream now. Just spotting the new incoming messages, Felipe and
> Michael seem to share my thoughts on that.

They didn't ask what you wanted to do exactly either :-)

variants support most base Delphi types, afaik including class , interface
and dyn array.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Michael Van Canneyt


On Wed, 22 Sep 2010, Marco van de Voort wrote:

> In our previous episode, Graeme Geldenhuys said:
>>> I would first try a variant.
>>
>> Just for getting the initial DND support to work under Xlib, I limited
>> myself to text types only, so choose variant as my storage type, but
>> now that I'm moving to the next phase (multiple data types), I am
>> considering other storage types because I don't know if variant is
>> suitable for binary data like sound, images etc..
>
> No. But it allows you to keep text as text, not as streamed value. I don't
> know if assigning a string to a variant is copy-on-write though.
>
> If you don't operate much on it, it doesn't matter. But the above sounds
> like you do most of the work on text, not binary blobs.
>
>> I'm leaning towards
>> TMemoryStream now. Just spotting the new incoming messages, Felipe and
>> Michael seem to share my thoughts on that.
>
> They didn't ask what you wanted to do exactly either :-)
>
> variants support most base Delphi types, afaik including class , interface
> and dyn array.

For the kind of data he is most likely to receive, a stream still seems best.

For most data formats, maybe a TStringSTream is better suited. You'll have
the text as data available at once, which is the most likely use anyway, and
has all semantics of a stream.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Graeme Geldenhuys-2
In reply to this post by Marco van de Voort
On 22 September 2010 13:31, Marco van de Voort wrote:
>
> No. But it allows you to keep text as text, not as streamed value.

I'll see how it goes, maybe I could store text as text, and binary as
binary - this will allow for the least amount of conversions. Plus the
storage variable not being used will not be initialized - to reduce
memory consumption. I'll experiment further.


> If you don't operate much on it, it doesn't matter. But the above sounds
> like you do most of the work on text, not binary blobs.

Well, that will depend on the application the developer writes, not on
what I do or use.


> variants support most base Delphi types, afaik including class , interface
> and dyn array.

Supporting delphi data types is pointless when drag and drop occurs
between two applications - hence the reason I am opting for the
mime-type standards. This is why I can't believe the stupid
drag-n-drop support implemented in Delphi and Kylix. There, the data
enters the event handler as a TObject instance, so the VCL and CLX
frameworks only support drag-n-drop inside the same application. For
external DND, you must revert to external OS API's - using Win32 API
under Windows, and no idea what under Kylix.


--
Regards,
  - Graeme -


_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net:8080/fpgui/
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

Matt Emson-2
In reply to this post by Graeme Geldenhuys-2
Graeme Geldenhuys wrote:
> So which one would be more fitting for any mime types, or is there
> another option I didn't think of?  Here is a simple usage example of
> the TfpgMimeData class - to help put this in perspective:
>  

This seems a lot like the BMessage class used with in the BeAPI. There
is an API called MUSCLE, which originates from a project created by
Jeremey Freisner for BeOS, but is now very "cross platform". It uses a
Message class to implement the format you mention below. It is designed
to be used for comms, so it has the concept of "flattening" data to a
binary format and then "unflattening" the data at the destination.

What does all this have to do with your class? I wrote an implementation
for Delphi a good 5 years ago. It might be worth looking at? It was in
the original MUSCLE distro, but if not, I can find you a copy.

https://public.msli.com/lcs/muscle/

Messages are not the "fastest" way of storing data, but the idea of an
extensible generic data storage mechanism is extremely powerful,
especially for distributed comms. MUSCLE implements a whole lot more and
really is worth looking at if you want to create C/S over a WAN. The
MUSCLE server becomes a middle man dispatcher on to which the clients
publish resources and subscribe to events. The "server" becomes another
node. Really love it, would love it more if it wasn't written in C++.

M


>
> var
>   m: TfpgMimeData;
>   d: TfpgDrag;
>   a: TfpgDropAction;
> begin
>       m := TfpgMimeData.Create;
>       m.SetData('text/plain', 'My name is Earl');
>       m.SetData('text/html', 'My name is <b>Earl</b>');  // text/plain
> can actually be created from this automatically
>       m.SetData('image/png', MyPNGImage);
>
>       d := TfpgDrag.Create(self);
>       d.MimeData := m;
>       d.StartDrag([daCopy]);
>
>       // d manages the lifespan of m
>       // d will be freed automatically when drag action is complete or cancelled
> end;
>
>
>  

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Any recommendations for a good in-memory storage type

zaher dirkey
In reply to this post by Graeme Geldenhuys-2
On Wed, Sep 22, 2010 at 1:31 PM, Graeme Geldenhuys <[hidden email]> wrote:
Hi,

While implementing drag-n-drop support in fpGUI, I created a
TfpgMimeData class. This class will hold various data items, and a
mime-type describing each of the data items. Obviously the
TfpgMimeData class should be able to handle any data type. So my
question is, what is a good universal type to store all kinds of data
(strings, images (binary data), sound etc)?

The possible ideas I can think of is the following:

 TByteArray
 Variants
 TMemoryStream

So which one would be more fitting for any mime types, or is there
another option I didn't think of?  Here is a simple usage example of
the TfpgMimeData class - to help put this in perspective:


var
 m: TfpgMimeData;
 d: TfpgDrag;
 a: TfpgDropAction;
begin
     m := TfpgMimeData.Create;
     m.SetData('text/plain', 'My name is Earl');
     m.SetData('text/html', 'My name is <b>Earl</b>');  // text/plain
can actually be created from this automatically
     m.SetData('image/png', MyPNGImage);

     d := TfpgDrag.Create(self);
     d.MimeData := m;
     d.StartDrag([daCopy]);

     // d manages the lifespan of m
     // d will be freed automatically when drag action is complete or cancelled
end;

 
Just initial idea.

TfpgMimeData class(TfpgDragData);
Picture:TPic ...
end;

var
 m: TfpgDragData;
 d: TfpgDrag;
 a: TfpgDropAction;
begin
     m := TfpgMimeData.Create;
     m.SetData('text/plain', 'My name is Earl');
     m.SetData('text/html', 'My name is <b>Earl</b>');  // text/plain
can actually be created from this automatically

   m.SetData('image/png'); 
     (m as TfpgMimeData).Picture.Assign(MyPNGImage);

     d := TfpgDrag.Create(self);
     d.DragData := m;
     d.StartDrag([daCopy]);

     // d manages the lifespan of m
     // d will be freed automatically when drag action is complete or cancelled
end;

--
Zaher Dirkey

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal