How to save a huge XML?

classic Classic list List threaded Threaded
40 messages Options
12
Reply | Threaded
Open this post in threaded view
|

How to save a huge XML?

Marcos Douglas B. Santos
Hi,

I need to save some huge XML files in HD.
Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G.. 30G...

What the best way to save "line by line"?

Regards,
Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Antonio Fortuny

Le 18/06/2013 14:35, Marcos Douglas a écrit :
Hi,

I need to save some huge XML files in HD.
Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G.. 30G...

What the best way to save "line by line"?
Append chuncks of lines to a TFileStream until source XML file is exhausted

Regards,
Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

--
Sita
                Software
Antonio Fortuny
Senior Software engineer

220, avenue de la Liberté
L-4602 Niederkorn
Tel.: +352 58 00 93 - 93
www.sitasoftware.lu
Your IT Partner

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
On Tue, Jun 18, 2013 at 9:39 AM, Antonio Fortuny
<[hidden email]> wrote:

>
>
> Le 18/06/2013 14:35, Marcos Douglas a écrit :
>
> Hi,
>
> I need to save some huge XML files in HD.
> Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G.. 30G...
>
> What the best way to save "line by line"?
>
> Append chuncks of lines to a TFileStream until source XML file is exhausted

But I'm using TXMLDocument class to make the XML.

For each line (log) I do this:
(doesn't matter the information in Par variable)

procedure TXMLLogger.Log;
var
  I: Integer;
  No: TDOMElement;
  Par: TParam;
begin
  No := FDoc.CreateElement('line');
  for I := 0 to FParams.Count-1 do
  begin
    Par := FParams.Items[I];
    TDOMElement(No).SetAttribute(Par.Name, Par.AsString);
  end;
  FRootNode.AppendChild(No);

  FParams.Clear;
end;

At the end, I do some like:
procedure TXMLLogger.Finish;
var
  No: TDOMElement;
begin
  No := FDoc.CreateElement('finish');
  TDOMElement(No).SetAttribute('dh', FmtDateTime(Now));
  FRootNode.AppendChild(No);
end;

And the application call:
procedure TXMLLogger.SaveToFile(const AFileName: string);
begin
  WriteXMLFile(FDoc, AFileName);
end;

So, there is a way to use TXMLDocument but save line by line more faster?

Regards,
Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Michael Van Canneyt


On Tue, 18 Jun 2013, Marcos Douglas wrote:

> On Tue, Jun 18, 2013 at 9:39 AM, Antonio Fortuny
> <[hidden email]> wrote:
>>
>>
>> Le 18/06/2013 14:35, Marcos Douglas a écrit :
>>
>> Hi,
>>
>> I need to save some huge XML files in HD.
>> Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G.. 30G...
>>
>> What the best way to save "line by line"?
>>
>> Append chuncks of lines to a TFileStream until source XML file is exhausted
>
> But I'm using TXMLDocument class to make the XML.
>
> For each line (log) I do this:
> (doesn't matter the information in Par variable)
>
> procedure TXMLLogger.Log;
> var
>  I: Integer;
>  No: TDOMElement;
>  Par: TParam;
> begin
>  No := FDoc.CreateElement('line');
>  for I := 0 to FParams.Count-1 do
>  begin
>    Par := FParams.Items[I];
>    TDOMElement(No).SetAttribute(Par.Name, Par.AsString);
>  end;
>  FRootNode.AppendChild(No);
>
>  FParams.Clear;
> end;
>
> At the end, I do some like:
> procedure TXMLLogger.Finish;
> var
>  No: TDOMElement;
> begin
>  No := FDoc.CreateElement('finish');
>  TDOMElement(No).SetAttribute('dh', FmtDateTime(Now));
>  FRootNode.AppendChild(No);
> end;
>
> And the application call:
> procedure TXMLLogger.SaveToFile(const AFileName: string);
> begin
>  WriteXMLFile(FDoc, AFileName);
> end;
>
> So, there is a way to use TXMLDocument but save line by line more faster?
>
No.
The writer already uses a fixed buffer. Your problem is the use of TXMLDocument.

Simply put: logging to XML (worse: using DOM) is a VERY bad idea.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
On Tue, Jun 18, 2013 at 10:32 AM, Michael Van Canneyt
<[hidden email]> wrote:

>
>
> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>
>> On Tue, Jun 18, 2013 at 9:39 AM, Antonio Fortuny
>> <[hidden email]> wrote:
>>>
>>>
>>>
>>> Le 18/06/2013 14:35, Marcos Douglas a écrit :
>>>
>>> Hi,
>>>
>>> I need to save some huge XML files in HD.
>>> Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G..
>>> 30G...
>>>
>>> What the best way to save "line by line"?
>>>
>>> Append chuncks of lines to a TFileStream until source XML file is
>>> exhausted
>>
>>
>> But I'm using TXMLDocument class to make the XML.
>>
>> For each line (log) I do this:
>> (doesn't matter the information in Par variable)
>>
>> procedure TXMLLogger.Log;
>> var
>>  I: Integer;
>>  No: TDOMElement;
>>  Par: TParam;
>> begin
>>  No := FDoc.CreateElement('line');
>>  for I := 0 to FParams.Count-1 do
>>  begin
>>    Par := FParams.Items[I];
>>    TDOMElement(No).SetAttribute(Par.Name, Par.AsString);
>>  end;
>>  FRootNode.AppendChild(No);
>>
>>  FParams.Clear;
>> end;
>>
>> At the end, I do some like:
>> procedure TXMLLogger.Finish;
>> var
>>  No: TDOMElement;
>> begin
>>  No := FDoc.CreateElement('finish');
>>  TDOMElement(No).SetAttribute('dh', FmtDateTime(Now));
>>  FRootNode.AppendChild(No);
>> end;
>>
>> And the application call:
>> procedure TXMLLogger.SaveToFile(const AFileName: string);
>> begin
>>  WriteXMLFile(FDoc, AFileName);
>> end;
>>
>> So, there is a way to use TXMLDocument but save line by line more faster?
>>
>
> No. The writer already uses a fixed buffer. Your problem is the use of
> TXMLDocument.

Ideas? Maybe do what Antonio Fortuny said before?

>
> Simply put: logging to XML (worse: using DOM) is a VERY bad idea.

Well, this is a request of my client. Other process will read this XML.

Thanks,
Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Michael Van Canneyt


On Tue, 18 Jun 2013, Marcos Douglas wrote:

>> No. The writer already uses a fixed buffer. Your problem is the use of
>> TXMLDocument.
>
> Ideas? Maybe do what Antonio Fortuny said before?
>
>>
>> Simply put: logging to XML (worse: using DOM) is a VERY bad idea.
>
> Well, this is a request of my client. Other process will read this XML.

Under no circumstances should you use TXMLDocument for this.

Better write the XML manually then: 'fire and forget'.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Antonio Fortuny
In reply to this post by Marcos Douglas B. Santos

Le 18/06/2013 15:42, Marcos Douglas a écrit :
On Tue, Jun 18, 2013 at 10:32 AM, Michael Van Canneyt
[hidden email] wrote:

On Tue, 18 Jun 2013, Marcos Douglas wrote:

On Tue, Jun 18, 2013 at 9:39 AM, Antonio Fortuny
[hidden email] wrote:


Le 18/06/2013 14:35, Marcos Douglas a écrit :

Hi,

I need to save some huge XML files in HD.
Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G..
30G...

What the best way to save "line by line"?

Append chuncks of lines to a TFileStream until source XML file is
exhausted

But I'm using TXMLDocument class to make the XML.

For each line (log) I do this:
(doesn't matter the information in Par variable)

procedure TXMLLogger.Log;
var
 I: Integer;
 No: TDOMElement;
 Par: TParam;
begin
 No := FDoc.CreateElement('line');
 for I := 0 to FParams.Count-1 do
 begin
   Par := FParams.Items[I];
   TDOMElement(No).SetAttribute(Par.Name, Par.AsString);
 end;
 FRootNode.AppendChild(No);

 FParams.Clear;
end;

At the end, I do some like:
procedure TXMLLogger.Finish;
var
 No: TDOMElement;
begin
 No := FDoc.CreateElement('finish');
 TDOMElement(No).SetAttribute('dh', FmtDateTime(Now));
 FRootNode.AppendChild(No);
end;

And the application call:
procedure TXMLLogger.SaveToFile(const AFileName: string);
begin
 WriteXMLFile(FDoc, AFileName);
end;

So, there is a way to use TXMLDocument but save line by line more faster?

No. The writer already uses a fixed buffer. Your problem is the use of
TXMLDocument.
Ideas? Maybe do what Antonio Fortuny said before?
Yep, a text file: assign, reset, append, close. I use an own build logger facility (for historical reasons) but I wonder whether Lazarus (or FPC) does not have such a logger component.

Simply put: logging to XML (worse: using DOM) is a VERY bad idea.
Well, this is a request of my client. Other process will read this XML.

Thanks,
Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal

--
Sita
                Software
Antonio Fortuny
Senior Software engineer

220, avenue de la Liberté
L-4602 Niederkorn
Tel.: +352 58 00 93 - 93
www.sitasoftware.lu
Your IT Partner

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
In reply to this post by Michael Van Canneyt
On Tue, Jun 18, 2013 at 10:44 AM, Michael Van Canneyt
<[hidden email]> wrote:

>
>
> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>
>>> No. The writer already uses a fixed buffer. Your problem is the use of
>>> TXMLDocument.
>>
>>
>> Ideas? Maybe do what Antonio Fortuny said before?
>>
>>>
>>> Simply put: logging to XML (worse: using DOM) is a VERY bad idea.
>>
>>
>> Well, this is a request of my client. Other process will read this XML.
>
>
> Under no circumstances should you use TXMLDocument for this.

So, where is recommended to use TXMLDocument class?

>
> Better write the XML manually then: 'fire and forget'.

Ok, I'll do it.

Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
In reply to this post by Antonio Fortuny
On Tue, Jun 18, 2013 at 10:49 AM, Antonio Fortuny
<[hidden email]> wrote:

>
>
> Le 18/06/2013 15:42, Marcos Douglas a écrit :
>
> On Tue, Jun 18, 2013 at 10:32 AM, Michael Van Canneyt
> <[hidden email]> wrote:
>
> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>
> On Tue, Jun 18, 2013 at 9:39 AM, Antonio Fortuny
> <[hidden email]> wrote:
>
> Le 18/06/2013 14:35, Marcos Douglas a écrit :
>
> Hi,
>
> I need to save some huge XML files in HD.
> Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G..
> 30G...
>
> What the best way to save "line by line"?
>
> Append chuncks of lines to a TFileStream until source XML file is
> exhausted
>
> But I'm using TXMLDocument class to make the XML.
>
> For each line (log) I do this:
> (doesn't matter the information in Par variable)
>
> procedure TXMLLogger.Log;
> var
>  I: Integer;
>  No: TDOMElement;
>  Par: TParam;
> begin
>  No := FDoc.CreateElement('line');
>  for I := 0 to FParams.Count-1 do
>  begin
>    Par := FParams.Items[I];
>    TDOMElement(No).SetAttribute(Par.Name, Par.AsString);
>  end;
>  FRootNode.AppendChild(No);
>
>  FParams.Clear;
> end;
>
> At the end, I do some like:
> procedure TXMLLogger.Finish;
> var
>  No: TDOMElement;
> begin
>  No := FDoc.CreateElement('finish');
>  TDOMElement(No).SetAttribute('dh', FmtDateTime(Now));
>  FRootNode.AppendChild(No);
> end;
>
> And the application call:
> procedure TXMLLogger.SaveToFile(const AFileName: string);
> begin
>  WriteXMLFile(FDoc, AFileName);
> end;
>
> So, there is a way to use TXMLDocument but save line by line more faster?
>
> No. The writer already uses a fixed buffer. Your problem is the use of
> TXMLDocument.
>
> Ideas? Maybe do what Antonio Fortuny said before?
>
> Yep, a text file: assign, reset, append, close. I use an own build logger
> facility (for historical reasons) but I wonder whether Lazarus (or FPC)
> does not have such a logger component.

Ok, thank you.

Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Michael Van Canneyt
In reply to this post by Marcos Douglas B. Santos


On Tue, 18 Jun 2013, Marcos Douglas wrote:

> On Tue, Jun 18, 2013 at 10:44 AM, Michael Van Canneyt
> <[hidden email]> wrote:
>>
>>
>> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>>
>>>> No. The writer already uses a fixed buffer. Your problem is the use of
>>>> TXMLDocument.
>>>
>>>
>>> Ideas? Maybe do what Antonio Fortuny said before?
>>>
>>>>
>>>> Simply put: logging to XML (worse: using DOM) is a VERY bad idea.
>>>
>>>
>>> Well, this is a request of my client. Other process will read this XML.
>>
>>
>> Under no circumstances should you use TXMLDocument for this.
>
> So, where is recommended to use TXMLDocument class?

General XML processing. Small documents.

You should be aware that the DOM always has the whole XML document in memory,
plus a substantial amount of overhead. This is not something FPC specific, but is
inherent in the DOM model. (a W3 spec)

Regardles of XML or not: keeping logs in memory is simply a bad idea.
You can buffer them for performance reasons, but after some time you must write to file.

The DOM model does not allow this.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Michael Van Canneyt
In reply to this post by Marcos Douglas B. Santos


On Tue, 18 Jun 2013, Marcos Douglas wrote:

>> Ideas? Maybe do what Antonio Fortuny said before?
>>
>> Yep, a text file: assign, reset, append, close. I use an own build logger
>> facility (for historical reasons) but I wonder whether Lazarus (or FPC)
>> does not have such a logger component.
>
> Ok, thank you.

Keep the file open between writes.
Otherwise you'll notice a serious performance penalty.
You can always call Flush() to make sure the content of
a buffer was written to disc.

Thus, in case of a crash, the XML file will contain all log entries,
but will not be well-formed XML.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
In reply to this post by Michael Van Canneyt
On Tue, Jun 18, 2013 at 11:01 AM, Michael Van Canneyt
<[hidden email]> wrote:

>
>
> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>
>> On Tue, Jun 18, 2013 at 10:44 AM, Michael Van Canneyt
>> <[hidden email]> wrote:
>>>
>>>
>>>
>>> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>>>
>>>>> No. The writer already uses a fixed buffer. Your problem is the use of
>>>>> TXMLDocument.
>>>>
>>>>
>>>>
>>>> Ideas? Maybe do what Antonio Fortuny said before?
>>>>
>>>>>
>>>>> Simply put: logging to XML (worse: using DOM) is a VERY bad idea.
>>>>
>>>>
>>>>
>>>> Well, this is a request of my client. Other process will read this XML.
>>>
>>>
>>>
>>> Under no circumstances should you use TXMLDocument for this.
>>
>>
>> So, where is recommended to use TXMLDocument class?
>
>
> General XML processing. Small documents.
>
> You should be aware that the DOM always has the whole XML document in
> memory, plus a substantial amount of overhead. This is not something FPC
> specific, but is inherent in the DOM model. (a W3 spec)
>
> Regardles of XML or not: keeping logs in memory is simply a bad idea. You
> can buffer them for performance reasons, but after some time you must write
> to file.
>
> The DOM model does not allow this.

Yes, I see now.
I'll change my app.

Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

zaher dirkey
In reply to this post by Michael Van Canneyt

On Tue, Jun 18, 2013 at 5:01 PM, Michael Van Canneyt <[hidden email]> wrote:
Regardles of XML or not: keeping logs in memory is simply a bad idea.

​Log it to a csv file, and at the end you can convert it to xml for your client, you can do that without loading it into memory, process it line by line.

Best Regards
Zaher Dirkey

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
On Tue, Jun 18, 2013 at 11:42 AM, Zaher Dirkey <[hidden email]> wrote:

>
> On Tue, Jun 18, 2013 at 5:01 PM, Michael Van Canneyt
> <[hidden email]> wrote:
>>
>> Regardles of XML or not: keeping logs in memory is simply a bad idea.
>
>
> Log it to a csv file, and at the end you can convert it to xml for your
> client, you can do that without loading it into memory, process it line by
> line.

But in that way I will have two process to run.
I'll write a XML using strings... but thank you.

Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

zaher dirkey

On Tue, Jun 18, 2013 at 6:00 PM, Marcos Douglas <[hidden email]> wrote:
But in that way I will have two process to run.
I'll write a XML using strings... but thank you.

one process, but in function like TXMLLogger.Finish you can convert it,
​using strings ​in memory, it will eat your memory.


Best Regards
Zaher Dirkey

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
On Tue, Jun 18, 2013 at 12:04 PM, Zaher Dirkey <[hidden email]> wrote:
>
> On Tue, Jun 18, 2013 at 6:00 PM, Marcos Douglas <[hidden email]> wrote:
>>
>> But in that way I will have two process to run.
>> I'll write a XML using strings... but thank you.
>
>
> one process, but in function like TXMLLogger.Finish you can convert it,
> using strings in memory, it will eat your memory.

But I continue to keep the XML in memory...
The easy and fast way as I can see is change the TXMLLogger class to
write a simple file text but not using DOM.

Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Michael Van Canneyt


On Tue, 18 Jun 2013, Marcos Douglas wrote:

> On Tue, Jun 18, 2013 at 12:04 PM, Zaher Dirkey <[hidden email]> wrote:
>>
>> On Tue, Jun 18, 2013 at 6:00 PM, Marcos Douglas <[hidden email]> wrote:
>>>
>>> But in that way I will have two process to run.
>>> I'll write a XML using strings... but thank you.
>>
>>
>> one process, but in function like TXMLLogger.Finish you can convert it,
>> using strings in memory, it will eat your memory.
>
> But I continue to keep the XML in memory...
> The easy and fast way as I can see is change the TXMLLogger class to
> write a simple file text but not using DOM.

Yes.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

zaher dirkey
In reply to this post by Marcos Douglas B. Santos

On Tue, Jun 18, 2013 at 6:09 PM, Marcos Douglas <[hidden email]> wrote:
But I continue to keep the XML in memory...
The easy and fast way as I can see is change the TXMLLogger class to
write a simple file text but not using DOM.

Hmm, give us a small example of that xml.​

Best Regards
Zaher Dirkey

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Marcos Douglas B. Santos
On Tue, Jun 18, 2013 at 12:21 PM, Zaher Dirkey <[hidden email]> wrote:

>
>
> On Tue, Jun 18, 2013 at 6:09 PM, Marcos Douglas <[hidden email]> wrote:
>>
>> But I continue to keep the XML in memory...
>> The easy and fast way as I can see is change the TXMLLogger class to
>> write a simple file text but not using DOM.
>
>
> Hmm, give us a small example of that xml.

<?xml version="1.0" encoding="utf-8"?>
<root>
  <connection login="uref" database="DB_REF"/>
  <start dh="2013-06-18T13:54:57"/>
  <line idDoc="912" status="ok"/>
  <line idDoc="913" status="ok"/>
  <line idDoc="914" status="ok"/>
  <line...>
  <finish dh="2013-06-18T13:54:58"/>
</root>

Marcos Douglas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: How to save a huge XML?

Sven Barth-2
In reply to this post by Michael Van Canneyt
On 18.06.2013 16:01, Michael Van Canneyt wrote:

>
>
> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>
>> On Tue, Jun 18, 2013 at 10:44 AM, Michael Van Canneyt
>> <[hidden email]> wrote:
>>>
>>>
>>> On Tue, 18 Jun 2013, Marcos Douglas wrote:
>>>
>>>>> No. The writer already uses a fixed buffer. Your problem is the use of
>>>>> TXMLDocument.
>>>>
>>>>
>>>> Ideas? Maybe do what Antonio Fortuny said before?
>>>>
>>>>>
>>>>> Simply put: logging to XML (worse: using DOM) is a VERY bad idea.
>>>>
>>>>
>>>> Well, this is a request of my client. Other process will read this XML.
>>>
>>>
>>> Under no circumstances should you use TXMLDocument for this.
>>
>> So, where is recommended to use TXMLDocument class?
>
> General XML processing. Small documents.
>
> You should be aware that the DOM always has the whole XML document in
> memory, plus a substantial amount of overhead. This is not something FPC
> specific, but is inherent in the DOM model. (a W3 spec)
>
> Regardles of XML or not: keeping logs in memory is simply a bad idea.
> You can buffer them for performance reasons, but after some time you
> must write to file.
>
> The DOM model does not allow this.

Maybe we should implement a simple API for just writing an XML to a
stream. Something like SAX only the other way round. E.g.

=== example code begin ===

xml.StartNode('outer');
xml.AddAttribute('foo', 42);
xml.StartNode('blubb');
xml.AddAttribute('bar', 'test');
xml.AddAttribute('bar2', 'xyz');
xml.SetValue('Hello');
xml.EndNode;
xml.StartNode('blubb');
xml.AddAttribute('bar2', 'alpha');
xml.AddAttribute('bar', 'beta');
xml.SetValue('World');
xml.EndNode;
xml.EndNode;

=== example code end ===

would the result in

=== xml begin ===

<outer foo="42">
   <blubb bar="test" bar2="xyz">
     Hello
   </blubb>
   <blubb bar2="alpha" bar="beta">
     Hello
   </blubb>
</outer>

=== xml end ===

Regards,
Sven
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
12