FreePascal Windows - Force files to write to disk

classic Classic list List threaded Threaded
42 messages Options
123
Reply | Threaded
Open this post in threaded view
|

FreePascal Windows - Force files to write to disk

Zaaphod

I have a freepascal Windows console application that I recently ported over from DOS Turbo Pascal.   I am have a small settings file that I write to disk that keeps getting corrupted.  It happens only occasionally, but it’s always this one file, no others.    Do I need to do something specific to force windows to write the file to disk and not have it in some kind of cache or buffer?   I am using the following code to write the file:

 

 

   Assign(BitFile,'BitSave.pax');

   ReWrite(BitFile);

   WriteLn(BitFile,XADJ:1:8);

   WriteLn(BitFile,YADJ:1:8);

   WriteLn(BitFile,ZADJ:1:8);

   WriteLn(BitFile,WADJ:1:8);

   WriteLn(BitFile,AADJ:1:8);

   WriteLn(BitFile,TADJ:1:8);

   WriteLn(BitFile,VADJ:1:8);

   WriteLn(BitFile,UADJ:1:8);

   WriteLn(BitFile,CurrentTool);

   WriteLn(Bitfile,P_Value[4]);

   Close(BitFile);

 

The above only happens when I make a change to one of the settings, and the system is never having any issue at that time, yet upon occasion I will have an error opening the file and when I inspect it, it’s just a long string of [nul].   I am not leaving the file open, I’m assigning it, writing to it, then closing it, so there should be no opportunity for data to be lost.  I am having the same issue on about 8 different computers, most of which are running solid state hard drives. It’s always just this one file that is affected, nothing else.  No other process ever uses this file, so it must be my freepascal program that is leaving the file in a vulnerable state.

My understanding was the Close(file); would save the file all the way to disk and flush any buffers that were holding it, but I’m wondering if that was only true for DOS and there is some other windows cache or buffer that also needs to be instructed to flush to truly save the file all the way to disk and close it.

 

Thanks for any advice on this

 

James


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Tomas Hajny-2
On Tue, February 21, 2017 22:12, James Richters wrote:


Hello James,

> I have a freepascal Windows console application that I recently ported
> over from DOS Turbo Pascal.   I am have a small settings file that I write
> to disk that keeps getting corrupted.  It happens only occasionally, but
> it’s always this one file, no others.    Do I need to do something
> specific to force windows to write the file to disk and not have it in
> some kind of cache or buffer?   I am using the following code to write the
> file:
>
>    Assign(BitFile,'BitSave.pax');
>    ReWrite(BitFile);
>    WriteLn(BitFile,XADJ:1:8);
>    WriteLn(BitFile,YADJ:1:8);
>    WriteLn(BitFile,ZADJ:1:8);
>    WriteLn(BitFile,WADJ:1:8);
>    WriteLn(BitFile,AADJ:1:8);
>    WriteLn(BitFile,TADJ:1:8);
>    WriteLn(BitFile,VADJ:1:8);
>    WriteLn(BitFile,UADJ:1:8);
>    WriteLn(BitFile,CurrentTool);
>    WriteLn(Bitfile,P_Value[4]);
>    Close(BitFile);
>
> The above only happens when I make a change to one of the settings, and
> the system is never having any issue at that time, yet upon occasion I
> will have an error opening the file and when I inspect it, it’s just a
> long string of [nul].   I am not leaving the file open, I’m assigning it,
> writing to it, then closing it, so there should be no opportunity for data
> to be lost.  I am having the same issue on about 8 different computers,
> most of which are running solid state hard drives. It’s always just this
> one file that is affected, nothing else.  No other process ever uses this
> file, so it must be my freepascal program that is leaving the file in a
> vulnerable state.

It's difficult to find out possible reasons without fully compilable
source. If I should guess, I would check the types of the variables used
for writing and also make sure that the content of those variables isn't
overwritten in memory (i.e. even before they're written to the file). In
particular, if your variables are strings and you compiled the source
(originally written for Turbo Pascal, i.e. using shortstrings) e.g. with
{$H+}, or in a compiler mode using ansistrings or
widestrings/unicodestrings by default _and_ the original source modified
the strings in a non-standard way, you might overwrite the content of
those variables.

BTW, what exactly is the error you get when opening the file?


> My understanding was the Close(file); would save the file all the way to
> disk and flush any buffers that were holding it, but I’m wondering if that
> was only true for DOS and there is some other windows cache or buffer that
> also needs to be instructed to flush to truly save the file all the way to
> disk and close it.

Your understanding is correct, nothing else should be necessary.

Tomas


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Jonas Maebe-3
In reply to this post by Zaaphod
James Richters wrote:
> I have a freepascal Windows console application that I recently ported
> over from DOS Turbo Pascal.   I am have a small settings file that I
> write to disk that keeps getting corrupted.  It happens only
> occasionally, but it’s always this one file, no others.    Do I need to
> do something specific to force windows to write the file to disk and not
> have it in some kind of cache or buffer?   I am using the following code
> to write the file:

This reminds me of a bug report from a couple of years back, where
someone else had a somewhat similar weird issue under Windows:
http://bugs.freepascal.org/view.php?id=27670

Do you happen to use Flexraid as well?


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Santiago A.
In reply to this post by Zaaphod
El 21/02/2017 a las 22:12, James Richters escribió:

 

Thanks for any advice on this


My first action would be to guarantee that the file is closed with a try...finally.

AssignFile(BitFile,'BitSave.pax');

ReWrite(BitFile);

try

....

finally

  CloseFile(BitFile);

end;


Second:

Do you have the file opened with an editor, or things like that, while running the program?.

Third:

Check that all variables have valid values and that values are converted to string properly to be printable.

Fourth:

Do you use somewhere {$I+} {$I-}? I also had some problems with that with old programs.

Fourth:

What's the type BitFile?
BitFile: TextFile;
BitFile: File;

Maybe using writeln with non-textfile files may cause problems.

-- 
Saludos

Santiago A.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Jürgen Hestermann
In reply to this post by Zaaphod
Am 2017-02-21 um 22:12 schrieb James Richters:
 > I have a freepascal Windows console application that I recently ported over from DOS Turbo Pascal.   I am have a small settings file that I write to disk that keeps getting corrupted.  It happens only occasionally, but it’s always this one file, no others.    Do I need to do something specific to force windows to write the file to disk and not have it in some kind of cache or buffer?   I am using the following code to write the file:
 >    Assign(BitFile,'BitSave.pax');
 >    ReWrite(BitFile);
 >    WriteLn(BitFile,XADJ:1:8);
 >    WriteLn(BitFile,YADJ:1:8);
 >    WriteLn(BitFile,ZADJ:1:8);
 >    WriteLn(BitFile,WADJ:1:8);
 >    WriteLn(BitFile,AADJ:1:8);
 >    WriteLn(BitFile,TADJ:1:8);
 >    WriteLn(BitFile,VADJ:1:8);
 >    WriteLn(BitFile,UADJ:1:8);
 >    WriteLn(BitFile,CurrentTool);
 >    WriteLn(Bitfile,P_Value[4]);
 >    Close(BitFile);
 > The above only happens when I make a change to one of the settings, and the system is never having any issue at that time, yet upon occasion I will have an error opening the file and when I inspect it, it’s just a long string of [nul].   I am not leaving the file open, I’m assigning it, writing to it, then closing it, so there should be no opportunity for data to be lost.  I am having the same issue on about 8 different computers, most of which are running solid state hard drives. It’s always just this one file that is affected, nothing else.  No other process ever uses this file, so it must be my freepascal program that is leaving the file in a vulnerable state.
 > My understanding was the Close(file); would save the file all the way to disk and flush any buffers that were holding it, but I’m wondering if that was only true for DOS and there is some other windows cache or buffer that also needs to be instructed to flush to truly save the file all the way to disk and close it.

How is "Bitfile" declared?
According to http://wiki.freepascal.org/Text
it should be either

var BitFile : TextFile;

or

var BitFile : Text;

Also, I am not sure whether "Close" is equivalent to "CloseFile" and  "Assign" is equivalent to "AssignFile".
I remenber that I had some issues using the old (TP7) notations but I am don't remember the details.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Zaaphod
In reply to this post by Santiago A.

Thanks for the advice on things to look at.

 

Here is some more information.  First of all, this happens extremely rarely, about once a month or two on a pc here and another there, but it’s odd that it’s always this one file, and it should not be happening at all.    I’m using turbo pascal compatibility mode, and it’s a console application, there is no multi threading going on, it’s straight inline code.  No other programs on the system ever access this file and there is no reason to open it with a text editor or anything like that.    The error I get is when trying to open the file to read it.  It’s something like read past end of file, because the corrupted file is just one long line, once I read that, the second read is past the end of file.  I can do checking to get the file length and avoid the error, but that doesn’t solve the real issue, which is that the data that is supposed to be in the file is just gone. 

 

Bitfile is type TEXT like this:

 

Bitfile: Text;

 

and my variables are all doubles. 

Normally what happens is, everything is fine for a month or two, but then one day someone will come in and notice the error on the screen, it was fine the night before, the program running overnight, in the morning the program is not running,  and when the attempt is made to run it,  The error is found and the cause of the error is BitSave.pax has been corrupted and is now a long string of [nul]   the computers this runs on are all windows 10 with Samsung solid state hard drives.    I suspect that the majority of the time, this has happened due to winders doing an automatic update and restarting without permission as windows 10 annoyingly likes to do, thus crashing my program in order to do this restart that is SO important.  I have shut off windows update via group policy, but still had the problem since, however I cannot be certain there was not a power failure.   The computers are set to re-power up after power loss in BIOS and the program is set to run on startup, in fact the computers have no other function than to run this one program, they are never used for anything else.  The procedure that writes out this file is only ever used when one of the variables is changed, and they can only be changed by manual input, which is not happening anymore when the issue occurs.   The thing is, even if windows forced a restart while my program was running this file should have been closed at the time, because if someone was standing there editing the variables, they would see the restart notification and close the program first, or be able to tell me they had a power failure.

 

As a temporary measure, I’m just writing out the file twice so at least I have a backup, but I don’t see why the backup would not be corrupted by this same issue, so that’s probably pretty useless.

Unfortunately this is difficult to troubleshoot because it happens so rarely. I have tried to force it to happen by terminating my program with task manager and cannot cause the file to be corrupted.   I have not yet tried to duplicate the error by shutting down the system while the program is running.  I could try that.     If it does turn out to be an issue with my program running during shutdown,  is there some way I can detect a pending shutdown and exit normally before the shutdown happens?   It wouldn’t really solve the cause of the issue, but it would possibly help prevent some of the occurrences related to shutdowns, but would not help with power failures.

 

I am unable to use Try in turbo pascal compatible mode.  If I use it, I get identifier not found ‘Try’   If I attempt to compile in some mode other than turbo pascal compatible, I end up with thousands of other errors, so I would pretty much have to do a complete re-write.   I can’t see how the close could not be run, there is no way out of my procedure without running it since it’s a single thread console application.

 

James

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Santiago A.
Sent: Wednesday, February 22, 2017 3:48 AM
To: FPC-Pascal users discussions <[hidden email]>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

 

El 21/02/2017 a las 22:12, James Richters escribió:

 

Thanks for any advice on this


My first action would be to guarantee that the file is closed with a try...finally.

AssignFile(BitFile,'BitSave.pax');

ReWrite(BitFile);

try

....

finally

  CloseFile(BitFile);

end;


Second:

Do you have the file opened with an editor, or things like that, while running the program?.

Third:

Check that all variables have valid values and that values are converted to string properly to be printable.

Fourth:

Do you use somewhere {$I+} {$I-}? I also had some problems with that with old programs.

Fourth:

What's the type BitFile?
BitFile: TextFile;
BitFile: File;

Maybe using writeln with non-textfile files may cause problems.


-- 
Saludos
 
Santiago A.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Peter
In reply to this post by Zaaphod
On 21/02/17 21:12, James Richters wrote:


> My understanding was the Close(file); would save the file all the way to
> disk and flush any buffers that were holding it,

I would try adding a Flush(File) before the Close, even though that
should not really be needed.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Sven Barth-2
In reply to this post by Jürgen Hestermann

Am 22.02.2017 13:12 schrieb "Jürgen Hestermann" <[hidden email]>:
> Also, I am not sure whether "Close" is equivalent to "CloseFile" and  "Assign" is equivalent to "AssignFile".
> I remenber that I had some issues using the old (TP7) notations but I am don't remember the details.

They are the same. The ones without "File" are from TP times and were renamed in Delphi to avoid confusing them with the Assign and Close methods of the component hierarchy.

Regards,
Sven


_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Santiago A.
In reply to this post by Zaaphod
El 22/02/2017 a las 13:17, James Richters escribió:

The error I get is when trying to open the file to read it.  It’s something like read past end of file, because the corrupted file is just one long line, once I read that, the second read is past the end of file.  I can do checking to get the file length and avoid the error, but that doesn’t solve the real issue, which is that the data that is supposed to be in the file is just gone. 


Just two lines of research ;-)

A) While you are reading the file you call the procedure that writes it.

B) The is a bug, not in writing the part, but in the reading code: Somewhere when you read the file, you don't close it, so the file remains open. Usually it's not a big issue, you can open the file several times, and when you close the program everything is closed. But some times, when a parameter is changed you rewrite it while it is still open and everything gets messed. So this two events must happen, the program has executed the part of code that lets the file open and next a parameter is changed.

 

The thing is, even if windows forced a restart while my program was running this file should have been closed at the time, because if someone was standing there editing the variables, they would see the restart notification and close the program first, or be able to tell me they had a power failure.

 

As a temporary measure, I’m just writing out the file twice so at least I have a backup, but I don’t see why the backup would not be corrupted by this same issue, so that’s probably pretty useless.

Yes, I think that's the way to go.

1) Before writing, read it and if it's ok make a backup of the file
2) Write the file.
3) Read again what you have written to check whether it's ok.
4) Make a second backup.

That way, when you load parameters you have two backups to recover. And maybe a clue of what's going wrong.
Beside, I would add a timestamp inside the file.

In addition, you could use flush and {$I-}


-- 
Saludos

Santiago A.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Zaaphod

Here is the procedure that reads the file. It is only read once when the program is started, then written to when changes are made.

I don’t have a good reason for having a different file variable name in my read procedure than my write procedure, but those are local varaibles to the procedures anyway so it shouldn’t matter what I call them.

 

Var

   locfile : text;

If FileExists('Bitsave.pax') Then

Begin

   Writeln('BitSave.pax Found, Loading... ');

   Assign(locfile,'BitSave.pax');

   reset(locfile);

   ReadLn(locfile,XAdj);

   ReadLn(locfile,YAdj);

   ReadLn(locfile,ZAdj);

   ReadLn(locfile,WAdj);

   ReadLn(locfile,AAdj);

   ReadLn(locfile,TAdj);

   ReadLn(locfile,VAdj);

   ReadLn(locfile,UAdj);

   ReadLn(locfile,CurrentTool);

   ReadLn(locfile,P_Value[4]);

   Close(locfile);

End

 

I will do as you suggest and read what I just thought I wrote back to verify it.   I may also incorporate some kind of checksum as well and it wouldn’t be a bad idea to check how many lines are in the file before I try to read it, just to make sure I don’t get read past end of file error.  It seems like a lot just to save just 10 numbers, but it’s a pain if those numbers are gone!    I will also try flush and {$i-}  as well. 

 

Thanks to everyone for your advice and help

 

Jim

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Santiago A.
Sent: Wednesday, February 22, 2017 11:07 AM
To: FPC-Pascal users discussions <[hidden email]>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

 

El 22/02/2017 a las 13:17, James Richters escribió:

The error I get is when trying to open the file to read it.  It’s something like read past end of file, because the corrupted file is just one long line, once I read that, the second read is past the end of file.  I can do checking to get the file length and avoid the error, but that doesn’t solve the real issue, which is that the data that is supposed to be in the file is just gone. 


Just two lines of research ;-)

A) While you are reading the file you call the procedure that writes it.

B) The is a bug, not in writing the part, but in the reading code: Somewhere when you read the file, you don't close it, so the file remains open. Usually it's not a big issue, you can open the file several times, and when you close the program everything is closed. But some times, when a parameter is changed you rewrite it while it is still open and everything gets messed. So this two events must happen, the program has executed the part of code that lets the file open and next a parameter is changed.


 

The thing is, even if windows forced a restart while my program was running this file should have been closed at the time, because if someone was standing there editing the variables, they would see the restart notification and close the program first, or be able to tell me they had a power failure.

 

As a temporary measure, I’m just writing out the file twice so at least I have a backup, but I don’t see why the backup would not be corrupted by this same issue, so that’s probably pretty useless.

Yes, I think that's the way to go.

1) Before writing, read it and if it's ok make a backup of the file
2) Write the file.
3) Read again what you have written to check whether it's ok.
4) Make a second backup.

That way, when you load parameters you have two backups to recover. And maybe a clue of what's going wrong.
Beside, I would add a timestamp inside the file.

In addition, you could use flush and {$I-}



-- 
Saludos
 
Santiago A.

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Jonas Maebe-3
In reply to this post by Zaaphod
On 22/02/17 13:17, James Richters wrote:
> No other programs on the system ever access this file

Not even virus scanners?


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Zaaphod
I suppose a virus scanner could access it, I didn't think of that.   I guess
I should have said no other program intentionally access the file.   It's
just a way for me to store some variables and get them back, it serves no
other purpose.  

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jonas Maebe
Sent: Wednesday, February 22, 2017 3:47 PM
To: FPC-Pascal users discussions <[hidden email]>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

On 22/02/17 13:17, James Richters wrote:
> No other programs on the system ever access this file

Not even virus scanners?


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Mark Morgan Lloyd-5
On 22/02/17 21:00, James Richters wrote:
> I suppose a virus scanner could access it, I didn't think of that.   I guess
> I should have said no other program intentionally access the file.   It's
> just a way for me to store some variables and get them back, it serves no
> other purpose.

A virus scanner might disagree. .pax is a known type which might contain
an archive hence a whole lot of nasty stuff, and it probably isn't wise
to use it casually... call it .txt or use the standard .ini format.

--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Klaus Hartnegg-3
In reply to this post by Zaaphod
Am 21.02.2017 um 22:12 schrieb James Richters:
>  Assign(BitFile,'BitSave.pax');
    {$I-}
>  ReWrite(BitFile);
    if ioresult <> 0 then
       writeln ('error opening pax file');

>  WriteLn(BitFile,XADJ:1:8);
>  WriteLn(BitFile,YADJ:1:8);
>  WriteLn(BitFile,ZADJ:1:8);
>  WriteLn(BitFile,WADJ:1:8);
>  WriteLn(BitFile,AADJ:1:8);
>  WriteLn(BitFile,TADJ:1:8);
>  WriteLn(BitFile,VADJ:1:8);
>  WriteLn(BitFile,UADJ:1:8);
>  WriteLn(BitFile,CurrentTool);
>  WriteLn(Bitfile,P_Value[4]);
    writeln (bitfile, 'EOF');
    if ioresult <> 0 then
       writeln ('error writing pax file');
>  Close(BitFile);
    if ioresult <> 0 then
       writeln ('error closing pax file');
    {$I+}

This writes something at the end of the file that does not come from a
variable. Next time the error happens, you can check if that is present.
Then your variables were overwritten in memory. Also it checks for
errors. And I would disable antivirus or at least tell it to not scan
this file.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Klaus Hartnegg-3
In reply to this post by Zaaphod
Am 22.02.2017 um 21:27 schrieb James Richters:
> Here is the procedure that reads the file.

Also you could always try to immediately read the file each time it has
been modified. Append a log entry to some other file, telling that the
pax file was modified and whether it was still correct. This would tell
whether it happens when the file is written, or at some other time.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Zaaphod
In reply to this post by Klaus Hartnegg-3
Thanks for the advice everyone.   It is very much appreciated.   I have been
working on a combination of ideas here.  Currently I am doing an MD5sum of a
string created from all my variables concatenated together and writing that
as the second to the last line of the file, I also am writing some fixed
text at the beginning and end of the file as suggested so I can check for
variables being overwritten in memory, great idea!   I am then reading the
file back to verify it has written properly, making a backup, checking that,
and also using ioresults as below. I will probably rename the file as well,
and exclude it from virus checkers.  When I read the file I can verify the
MD5sum and if that test fails, I'll just automatically try the backup, If
that also fails the MD5sum I'll notify the user.  

Thanks again for all the help

James
-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Klaus Hartnegg
Sent: Thursday, February 23, 2017 6:33 AM
To: [hidden email]
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Am 21.02.2017 um 22:12 schrieb James Richters:
>  Assign(BitFile,'BitSave.pax');
    {$I-}
>  ReWrite(BitFile);
    if ioresult <> 0 then
       writeln ('error opening pax file');

>  WriteLn(BitFile,XADJ:1:8);
>  WriteLn(BitFile,YADJ:1:8);
>  WriteLn(BitFile,ZADJ:1:8);
>  WriteLn(BitFile,WADJ:1:8);
>  WriteLn(BitFile,AADJ:1:8);
>  WriteLn(BitFile,TADJ:1:8);
>  WriteLn(BitFile,VADJ:1:8);
>  WriteLn(BitFile,UADJ:1:8);
>  WriteLn(BitFile,CurrentTool);
>  WriteLn(Bitfile,P_Value[4]);
    writeln (bitfile, 'EOF');
    if ioresult <> 0 then
       writeln ('error writing pax file');
>  Close(BitFile);
    if ioresult <> 0 then
       writeln ('error closing pax file');
    {$I+}

This writes something at the end of the file that does not come from a
variable. Next time the error happens, you can check if that is present.
Then your variables were overwritten in memory. Also it checks for errors.
And I would disable antivirus or at least tell it to not scan this file.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Zaaphod
I am still having this issue.  I've managed to narrow down what is happening
some.   The problem is my data is not actually being written all the way to
disk.   I now have a repeatable proven method to reproduce the issue.  Here
is the sequence.

In my freepascal console application, I create the file
Assign file
Rewrite file
Writeln file
Writeln file
Writeln file ...
Flush file
Close file

I then immediately read back the file, and all is good.  I even look at the
file with notepad++ to see that the data is correct.... and it is fine.

If I then power off the test computer (not doing a correct windows shutdown,
this simulates an unattended power failure) when it boots back up, my file
has the correct date and time and the correct length, hovever the contents
of the file is a string of $00 with no other characters, no carriage returns
and no linefeeds, however the number of $00s is EXACTLY how many bytes the
file should have been.  It's like the file was allocated but data not
actually written to disk yet... but I thought Close (file) was supposed to
do that.    I think I am dealing with some kind of disk caching going on
here, but I'm not sure how to force my files to commit all the way to disk.
I should note that these systems are all using Samsung SSDs in them, perhaps
there is some SSD weirdness going on?     I also notice it's not just this
one file that is affected, but EVERY SINGLE FILE I create with freepascal,
this is the only one I notice because the other ones are all work files that
end up being re-created every time I run my program, but I checked them all
and they are all exactly the same..  correct lengths but all data is a
string of 00s  

I'm thinking of just disabling all write behind caching in windows and on
the SSD if I can figure it out,  it's not needed anyway anymore because SSDs
are way more than fast enough, and I would rather wait for my data to be
written NOW instead of it NOT getting written later!!    The problem with
solving this by disabling write caching is that I can never be sure that
this was done.  So I would like to find a real solution that would force my
files all the way to disk on my own without depending on the system being
configured a certain way.   I really need these tiny files to survive power
failures.

Thanks for any advice on how to do this

James



-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of James Richters
Sent: Friday, February 24, 2017 1:53 PM
To: 'FPC-Pascal users discussions' <[hidden email]>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Thanks for the advice everyone.   It is very much appreciated.   I have been
working on a combination of ideas here.  Currently I am doing an MD5sum of a
string created from all my variables concatenated together and writing that
as the second to the last line of the file, I also am writing some fixed
text at the beginning and end of the file as suggested so I can check for
variables being overwritten in memory, great idea!   I am then reading the
file back to verify it has written properly, making a backup, checking that,
and also using ioresults as below. I will probably rename the file as well,
and exclude it from virus checkers.  When I read the file I can verify the
MD5sum and if that test fails, I'll just automatically try the backup, If
that also fails the MD5sum I'll notify the user.  

Thanks again for all the help

James
-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Klaus Hartnegg
Sent: Thursday, February 23, 2017 6:33 AM
To: [hidden email]
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Am 21.02.2017 um 22:12 schrieb James Richters:
>  Assign(BitFile,'BitSave.pax');
    {$I-}
>  ReWrite(BitFile);
    if ioresult <> 0 then
       writeln ('error opening pax file');

>  WriteLn(BitFile,XADJ:1:8);
>  WriteLn(BitFile,YADJ:1:8);
>  WriteLn(BitFile,ZADJ:1:8);
>  WriteLn(BitFile,WADJ:1:8);
>  WriteLn(BitFile,AADJ:1:8);
>  WriteLn(BitFile,TADJ:1:8);
>  WriteLn(BitFile,VADJ:1:8);
>  WriteLn(BitFile,UADJ:1:8);
>  WriteLn(BitFile,CurrentTool);
>  WriteLn(Bitfile,P_Value[4]);
    writeln (bitfile, 'EOF');
    if ioresult <> 0 then
       writeln ('error writing pax file');
>  Close(BitFile);
    if ioresult <> 0 then
       writeln ('error closing pax file');
    {$I+}

This writes something at the end of the file that does not come from a
variable. Next time the error happens, you can check if that is present.
Then your variables were overwritten in memory. Also it checks for errors.
And I would disable antivirus or at least tell it to not scan this file.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Henry Vermaak
On Tue, Mar 21, 2017 at 10:39:23AM -0400, James Richters wrote:

> I am still having this issue.  I've managed to narrow down what is happening
> some.   The problem is my data is not actually being written all the way to
> disk.   I now have a repeatable proven method to reproduce the issue.  Here
> is the sequence.
>
> In my freepascal console application, I create the file
> Assign file
> Rewrite file
> Writeln file
> Writeln file
> Writeln file ...
> Flush file

Have you tried using the FlushFileBuffers() Windows API?  Something like
this:

FlushFileBuffers(TextRec(AFile).Handle);

Add "windows" to the uses clause, obviously.

Henry
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Jonas Maebe-3
In reply to this post by Zaaphod
On 21/03/17 15:39, James Richters wrote:
> If I then power off the test computer (not doing a correct windows shutdown,
> this simulates an unattended power failure) when it boots back up, my file
> has the correct date and time and the correct length, hovever the contents
> of the file is a string of $00 with no other characters, no carriage returns
> and no linefeeds, however the number of $00s is EXACTLY how many bytes the
> file should have been.  It's like the file was allocated but data not
> actually written to disk yet... but I thought Close (file) was supposed to
> do that.

No, it does not do that. It flushes everything from the internal program
buffers to the operating system, but the operating system will perform
write-back caching. This is not specific to FPC programs, it does this
almost all writes to a regular hard drive or SSD (Windows does not do it
for USB sticks, afaik). Not doing so would slow down you system
significantly.


Jonas
_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: FreePascal Windows - Force files to write to disk

Zaaphod
In reply to this post by Henry Vermaak
I have not tried FlushFileBuffers() yet, so I just tried it now,  I am getting the same results.   I have also tried disabling write caching in disk management and STILL have the same results.   I am now thinking this is some caching being done on the SSD that is ignoring FlushFileBuffers()  Probably a 'feature' designed to help extend the life of the SSD, but making it useless to store data that needs to survive a power failure.

I have tried running my freepascal program on a normal mechanical hard disk and flushfilebuffers() seems to be working as expected, Data is getting actually written to the disk and my 'power failure' is not affecting the data at all, but on the SSD, it's still just a correct length file of 00s

Does anyone know of a way to force critical data to be written to a SSD so it's not lost during a power failure?  

James

-----Original Message-----
From: fpc-pascal [mailto:[hidden email]] On Behalf Of Henry Vermaak
Sent: Tuesday, March 21, 2017 11:08 AM
To: FPC-Pascal users discussions <[hidden email]>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

On Tue, Mar 21, 2017 at 10:39:23AM -0400, James Richters wrote:
> I am still having this issue.  I've managed to narrow down what is happening
> some.   The problem is my data is not actually being written all the way to
> disk.   I now have a repeatable proven method to reproduce the issue.  Here
> is the sequence.
>
> In my freepascal console application, I create the file Assign file
> Rewrite file Writeln file Writeln file Writeln file ...
> Flush file

Have you tried using the FlushFileBuffers() Windows API?  Something like
this:

FlushFileBuffers(TextRec(AFile).Handle);

Add "windows" to the uses clause, obviously.

Henry
_______________________________________________
fpc-pascal maillist  -  [hidden email] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

_______________________________________________
fpc-pascal maillist  -  [hidden email]
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
123