XML files and the ext4 bug

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

XML files and the ext4 bug

Gustavo Enrique Jimenez
Hi

Seems like the ext4 file system has a bug or some issue with files
updated often :

https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781

I use XML files heavily. It is great, XML units save me a lot of work
! My programs update de xml files quite often.

I am concerned about this particular ext4 bug. Could you say if
freepascal's XML units are affected by this bug ? ext4 developers
recomend this :

https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54

Thanks in advance

Gustavo

ps: excuse my english
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: XML files and the ext4 bug

David Emerson
I can't comment on pascal's XML handling, but perhaps I can shed some
light on the situation. I belive this new filesystem thing will bite
you if you do something like this:

assign (my_file, '/some/file');

rewrite (my_file);
// ... write out the contents of the file ...
close (my_file);

reset (my_file); <-- right here is where you get screwed.

ext4 and other modern filesystems have not committed your previous
rewrite to disk yet! So when you try to reset and read the file, you
get an EMPTY file from the OS. If you then read in the (empty) data,
make a few changes and write it out again, you're going to end up with
a patchy file.

Also, if some kind of crash occurs before the file has been sync'd then
the file would be completely gone.

When I write a config file out, I usually do so by writing out program
variables by name, hardcoded in my program -- rather than by retrieving
data from a config file and then writing out the same data that I just
read.

If you must do it by reading and then writing, then call (the pascal
equivalent of) fsync after close, keeping in mind that you will have to
wait for the (very slow) disk whenever that sync gets called.

Disclaimer: I don't REALLY know what I'm talking about: I am really
going out on a limb with some guesswork here. But I think I understood
Ts'o's message, and I doubt fpc calls sync on close ;)

Thanks, by the way, for including that message from Theodore Ts'o, which
I found to be quite enlightening!

~David.

On Wed 11 Mar 2009, Gustavo Enrique Jimenez wrote:

> Hi
>
> Seems like the ext4 file system has a bug or some issue with files
> updated often :
>
> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781
>
> I use XML files heavily. It is great, XML units save me a lot of work
> ! My programs update de xml files quite often.
>
> I am concerned about this particular ext4 bug. Could you say if
> freepascal's XML units are affected by this bug ? ext4 developers
> recomend this :
>
>
https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54
>
> Thanks in advance
>
> Gustavo
>
> ps: excuse my english
> ____________________________



_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: XML files and the ext4 bug

Florian Klaempfl
In reply to this post by Gustavo Enrique Jimenez
Gustavo Enrique Jimenez schrieb:
> Hi
>
> Seems like the ext4 file system has a bug or some issue with files
> updated often :
>
> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781

This is only a problem when your machine crashes!

>
> I use XML files heavily. It is great, XML units save me a lot of work
> ! My programs update de xml files quite often.
>
> I am concerned about this particular ext4 bug. Could you say if
> freepascal's XML units are affected by this bug ? ext4 developers
> recomend this :

Syncing and renaming is not done by the XML units. This is something the
programmer has to do.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: XML files and the ext4 bug

Michael Van Canneyt
In reply to this post by Gustavo Enrique Jimenez


On Wed, 11 Mar 2009, Gustavo Enrique Jimenez wrote:

> Hi
>
> Seems like the ext4 file system has a bug or some issue with files
> updated often :
>
> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781
>
> I use XML files heavily. It is great, XML units save me a lot of work
> ! My programs update de xml files quite often.
>
> I am concerned about this particular ext4 bug. Could you say if
> freepascal's XML units are affected by this bug ? ext4 developers
> recomend this :

Well, there is no 'bug', there is a misunderstanding of how things work.
That said, XML algorithms use method 1.

You can do method 3 yourself (it's just renaming files and syncing), but
I wouldn't recommend it as standard practice, as it uses 3 times more
diskspace and is quite slow.

And if methods 1 & 2 are "unsafe", then I think that there is a fundamental
bug in the filesystem (I never believed "journaling" filesystems can work
100% correctly in the first place), despite what Theodore T'so says.

The filesystem should just work. The standard application programmer should
never have to take into account the eventualities described there.

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: XML files and the ext4 bug

Florian Klaempfl
Michael Van Canneyt schrieb:
>
> And if methods 1 & 2 are "unsafe", then I think that there is a fundamental
> bug in the filesystem (I never believed "journaling" filesystems can work
> 100% correctly in the first place),

Journaling means only: the file system is always in an internally
consistent state It does not mean that it doesn't lose data!

> despite what Theodore T'so says.
>
> The filesystem should just work. The standard application programmer should
> never have to take into account the eventualities described there.

The filesystem works. It failes only if power is turned off or the
system crashes. But those are eventualities the programmer should not
take into account ;)
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: XML files and the ext4 bug

Michael Van Canneyt


On Thu, 12 Mar 2009, Florian Klaempfl wrote:

> Michael Van Canneyt schrieb:
> >
> > And if methods 1 & 2 are "unsafe", then I think that there is a fundamental
> > bug in the filesystem (I never believed "journaling" filesystems can work
> > 100% correctly in the first place),
>
> Journaling means only: the file system is always in an internally
> consistent state It does not mean that it doesn't lose data!

In this definition, yes. But I find that rather limited :-)

>
> > despite what Theodore T'so says.
> >
> > The filesystem should just work. The standard application programmer should
> > never have to take into account the eventualities described there.
>
> The filesystem works. It failes only if power is turned off or the
> system crashes. But those are eventualities the programmer should not
> take into account ;)

That is exactly what I meant...

Michael.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: XML files and the ext4 bug

Jonas Maebe-2
Michael Van Canneyt wrote on do, 12 mrt 2009:

> On Thu, 12 Mar 2009, Florian Klaempfl wrote:
>
>> Journaling means only: the file system is always in an internally
>> consistent state It does not mean that it doesn't lose data!
>
> In this definition, yes.

Well, that is the only definition of file system journaling I know of.

> But I find that rather limited :-)

It is much better than the previous situation where losing power could  
mean that your entire partition would become unreadable (due to a  
corruption of the inode indexes/file allocation tables/b-trees/...).

It's of course not a substitute for backups or versioned file systems.


Jonas

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: XML files and the ext4 bug

Gustavo Enrique Jimenez
Ok, thanks for all your replys !

I will take more care with my xml files.

Gustavo



2009/3/12 Jonas Maebe <[hidden email]>:

> Michael Van Canneyt wrote on do, 12 mrt 2009:
>
>> On Thu, 12 Mar 2009, Florian Klaempfl wrote:
>>
>>> Journaling means only: the file system is always in an internally
>>> consistent state It does not mean that it doesn't lose data!
>>
>> In this definition, yes.
>
> Well, that is the only definition of file system journaling I know of.
>
>> But I find that rather limited :-)
>
> It is much better than the previous situation where losing power could mean
> that your entire partition would become unreadable (due to a corruption of
> the inode indexes/file allocation tables/b-trees/...).
>
> It's of course not a substitute for backups or versioned file systems.
>
>
> Jonas
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> http://lists.freepascal.org/mailman/listinfo/fpc-pascal
>
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/mailman/listinfo/fpc-pascal