Writing to a drive which may be spun down

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Writing to a drive which may be spun down

Brian-4
Hi all,

Running Ubuntu 18 LTS, I have a console-mode number-cruncher which
writes occasional output files. It works just fine if the output is
directed to a drive which is permanently spun up, but can fail if the
output is directed to a USB drive (by which I mean a USB-connected 8TB
external drive, not one of the keydrives). I've followed things
through with the debugger, and it seems that the problem occurs when
the drive has powered itself down due to lack of accesses. The assign
and rewrite work OK, but then the first attempt to write to the file
fails with a 'No such file' error. Putting a large (30 second!) delay
before the first write means everything works OK, presumably because
the drive gets time to spin up again.

My question: Is there a standard method for handling this situation,
i.e. making sure that a drive has not spun down, or is it just a case
of writing a wrapper round the write function and handling the 'No
such file' error with a wait and a retry?

Thanks,

Brian.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Free Pascal - General mailing list
On Thu, 21 Nov 2019 01:16:16 -0500, Brian
<[hidden email]> wrote:

>Hi all,
>
>Running Ubuntu 18 LTS, I have a console-mode number-cruncher which
>writes occasional output files. It works just fine if the output is
>directed to a drive which is permanently spun up, but can fail if the
>output is directed to a USB drive (by which I mean a USB-connected 8TB
>external drive, not one of the keydrives). I've followed things
>through with the debugger, and it seems that the problem occurs when
>the drive has powered itself down due to lack of accesses. The assign
>and rewrite work OK, but then the first attempt to write to the file
>fails with a 'No such file' error. Putting a large (30 second!) delay
>before the first write means everything works OK, presumably because
>the drive gets time to spin up again.
>
>My question: Is there a standard method for handling this situation,
>i.e. making sure that a drive has not spun down, or is it just a case
>of writing a wrapper round the write function and handling the 'No
>such file' error with a wait and a retry?
>
 Maybe not a solution here but I have experienced the same kind of
problem on my Windows 7 laptop where I have a 2TB data drive where all
my work is being done. Compilations and other similar tasks were
annoyingly slow to start when I discovered that Windows7 had spun down
the drive and no matter what I did to change settings it would not
stay running.

So I created a very small FPC program that writes and reads back one
byte to a file on the drive every 10 seconds and measures the latency
(using gettickcount) to use to display the max/min and average delays.
When this is running (it is in my startup) I no longer have these
startup problems of the drive.

Right now it has accumulated 35510 access time values which average 15
ms with a max of 327 ms. All since the last reboot.


--
Bo Berglund
Developer in Sweden

_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

wkitty42
In reply to this post by Brian-4
On 11/21/19 1:16 AM, Brian wrote:
> My question: Is there a standard method for handling this situation,
> i.e. making sure that a drive has not spun down, or is it just a case
> of writing a wrapper round the write function and handling the 'No
> such file' error with a wait and a retry?


why not just turn off or adjust your power management so that drive doesn't spin
down when it is connected? i do similar here but for my monitors which sometimes
don't want to wake back up after being put to sleep...


--
  NOTE: No off-list assistance is given without prior approval.
        *Please keep mailing list traffic on the list unless*
        *a signed and pre-paid contract is in effect with us.*
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Free Pascal - General mailing list
In reply to this post by Brian-4
On Thu, 21 Nov 2019 01:16:16 -0500
Brian <[hidden email]> wrote:

> Hi all,
>
> Running Ubuntu 18 LTS, I have a console-mode number-cruncher which
> writes occasional output files. It works just fine if the output is
> directed to a drive which is permanently spun up, but can fail if the
> output is directed to a USB drive (by which I mean a USB-connected 8TB
> external drive, not one of the keydrives).

Maybe Ubuntu has the support for USB external devices in the kernel
compiled as a module?
If so the solution is to recompile the kernel.
Greetings, Marco
--
Simplex sigillum veri
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Brian-4
In reply to this post by wkitty42
On 11/21/19 7:12 AM, [hidden email] wrote:

> On 11/21/19 1:16 AM, Brian wrote:
>> My question: Is there a standard method for handling this situation,
>> i.e. making sure that a drive has not spun down, or is it just a case
>> of writing a wrapper round the write function and handling the 'No
>> such file' error with a wait and a retry?
>
>
> why not just turn off or adjust your power management so that drive
> doesn't spin down when it is connected? i do similar here but for my
> monitors which sometimes don't want to wake back up after being put to
> sleep...
>

Because on the *overwhelming* majority of occasions, I'm happy for the
drives to sleep if not being accessed. Most things that I do don't
need the USB drives. Bo's idea of a program which kept the drives spun
up had occurred to me if I couldn't find a setting somewhere, but I
was hoping for a solution which didn't require me to either disable
the drive's power management or do constant accesses.

Brian.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Winfried Bartnick
Am 21.11.19 um 13:52 schrieb Brian:
On 11/21/19 7:12 AM, [hidden email] wrote:
On 11/21/19 1:16 AM, Brian wrote:
My question: Is there a standard method for handling this situation,
i.e. making sure that a drive has not spun down, or is it just a case
of writing a wrapper round the write function and handling the 'No
such file' error with a wait and a retry?

why not just turn off or adjust your power management so that drive
doesn't spin down when it is connected? i do similar here but for my
monitors which sometimes don't want to wake back up after being put to
sleep...

Because on the *overwhelming* majority of occasions, I'm happy for the
drives to sleep if not being accessed. Most things that I do don't
need the USB drives. Bo's idea of a program which kept the drives spun
up had occurred to me if I couldn't find a setting somewhere, but I
was hoping for a solution which didn't require me to either disable
the drive's power management or do constant accesses.

Brian.


The old trick from DOS times with fileExists does not work anymore because the directory is cached.

What I do with "sleeping" USB disks is this:

{$I-}
repeat
application.processmessages;
assignFile(txt,'/mnt/test.txt');
rewrite (txt);
writeln (txt,'I''m so tired ...');
closeFile(txt);
io := ioResult;
// showMessage ('--> '+IntToStr(io));
until io=0;
{$I+}


If your disk is not corrupt or full then after this loop the drive is ready.

Tested with USB drives from Seagate and Toshiba.

This only tested with Linux!

Winni


_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Brian-4
On 11/21/19 2:38 PM, Winfried Bartnick wrote:

<...>

>
> writeln (txt,'I''m so tired ...');
> closeFile(txt);
> io := ioResult;

Thanks Winni, that sounds the sort of idea I'm looking for...

Brian.
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Zaaphod
I thought there was a way to figure out if the drive was spun down, perhaps with a S.M.A.R.T command?

I like the idea of using the ioResult, but I would put in a way to exit the loop for ioReults of errors not caused by a spun down drive would report right away and a timeout in case some other condition causes the same error as being spun down, so it's not caught in an infinite loop for errors not caused by the spindown.  Something like this... (Untested):

const
           A_Day = 1;
         An_Hour = A_Day/24;
        A_Minute = An_Hour/60;
        A_Second = A_Minute/60;

StartTime:=now;
assignFile(txt,'/mnt/test.txt');
{$I-}
rewrite (txt);
repeat
application.processmessages;
writeln (txt,'I''m so tired ...');
io := ioResult;
// showMessage ('--> '+IntToStr(io));
until (io=NoSuchFileErrorCode) or (StartTime+(30*A_Second)<Now);
{$I+}
closeFile(txt);
If io<>0 Then
   DoSomethingAboutError;

It seems strange that this would be an issue at all,  I would think the drive itself would cache any data written to it for long enough to get the drive spun up... seems very short sighted on hard drive design for this to be an issue at all, and even if the drive didn't do this,  it seems like the operating system should take care of it for you.  


James

-----Original Message-----
From: fpc-pascal <[hidden email]> On Behalf Of Brian
Sent: Thursday, November 21, 2019 8:55 PM
To: FPC-Pascal users discussions <[hidden email]>
Subject: Re: [fpc-pascal] Writing to a drive which may be spun down

On 11/21/19 2:38 PM, Winfried Bartnick wrote:

<...>

>
> writeln (txt,'I''m so tired ...');
> closeFile(txt);
> io := ioResult;

Thanks Winni, that sounds the sort of idea I'm looking for...

Brian.
_______________________________________________
fpc-pascal maillist  -  [hidden email] https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Winfried Bartnick
James dragged the solution a little bit away from the
"quick-and-dirty-corner".

This way it is usable also in critical situations.

SMART is a never ending story and you jump into the jungle:

* A lot of noname disk have no entry in the SMART DB

* Even a half year old Maxtor/Seagate was not found  in an updated DB

* On external disks SMART is often disabled by default

* I don't know a Pascal interface to smart

But what will you exspect when the price for 1 TB is now down to 40.- €???

It would be a good idea if reality had not so many obstacles.

Winni




Am 22.11.19 um 14:38 schrieb James Richters:

> I thought there was a way to figure out if the drive was spun down, perhaps with a S.M.A.R.T command?
>
> I like the idea of using the ioResult, but I would put in a way to exit the loop for ioReults of errors not caused by a spun down drive would report right away and a timeout in case some other condition causes the same error as being spun down, so it's not caught in an infinite loop for errors not caused by the spindown.  Something like this... (Untested):
>
> const
>             A_Day = 1;
>           An_Hour = A_Day/24;
>          A_Minute = An_Hour/60;
>          A_Second = A_Minute/60;
>
> StartTime:=now;
> assignFile(txt,'/mnt/test.txt');
> {$I-}
> rewrite (txt);
> repeat
> application.processmessages;
> writeln (txt,'I''m so tired ...');
> io := ioResult;
> // showMessage ('--> '+IntToStr(io));
> until (io=NoSuchFileErrorCode) or (StartTime+(30*A_Second)<Now);
> {$I+}
> closeFile(txt);
> If io<>0 Then
>     DoSomethingAboutError;
>
> It seems strange that this would be an issue at all,  I would think the drive itself would cache any data written to it for long enough to get the drive spun up... seems very short sighted on hard drive design for this to be an issue at all, and even if the drive didn't do this,  it seems like the operating system should take care of it for you.
>
>
> James
>
> -----Original Message-----
> From: fpc-pascal <[hidden email]> On Behalf Of Brian
> Sent: Thursday, November 21, 2019 8:55 PM
> To: FPC-Pascal users discussions <[hidden email]>
> Subject: Re: [fpc-pascal] Writing to a drive which may be spun down
>
> On 11/21/19 2:38 PM, Winfried Bartnick wrote:
>
> <...>
>
>> writeln (txt,'I''m so tired ...');
>> closeFile(txt);
>> io := ioResult;
> Thanks Winni, that sounds the sort of idea I'm looking for...
>
> Brian.
> _______________________________________________
> fpc-pascal maillist  -  [hidden email] https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Reply | Threaded
Open this post in threaded view
|

Re: Writing to a drive which may be spun down

Zaaphod
Ooops made a mistake in the until...  also maybe a short delay between retries would be helpful


const
            A_Day = 1;
          An_Hour = A_Day/24;
         A_Minute = An_Hour/60;
         A_Second = A_Minute/60;
StartTime:=now;
assignFile(txt,'/mnt/test.txt');
{$I-}
rewrite (txt);
repeat
application.processmessages;
writeln (txt,'I''m so tired ...');
io := ioResult;
// showMessage ('--> '+IntToStr(io));
If io=NoSuchFileErrorCode then
   Sleep(100);
until (io<>NoSuchFileErrorCode) or (StartTime+(30*A_Second)<Now); {$I+}
closeFile(txt); If io<>0 Then
    DoSomethingAboutError;



James  

-----Original Message-----
From: fpc-pascal <[hidden email]> On Behalf Of Winfried Bartnick
Sent: Friday, November 22, 2019 9:35 AM
To: [hidden email]
Subject: Re: [fpc-pascal] Writing to a drive which may be spun down

James dragged the solution a little bit away from the "quick-and-dirty-corner".

This way it is usable also in critical situations.

SMART is a never ending story and you jump into the jungle:

* A lot of noname disk have no entry in the SMART DB

* Even a half year old Maxtor/Seagate was not found  in an updated DB

* On external disks SMART is often disabled by default

* I don't know a Pascal interface to smart

But what will you exspect when the price for 1 TB is now down to 40.- €???

It would be a good idea if reality had not so many obstacles.

Winni




Am 22.11.19 um 14:38 schrieb James Richters:

> I thought there was a way to figure out if the drive was spun down, perhaps with a S.M.A.R.T command?
>
> I like the idea of using the ioResult, but I would put in a way to exit the loop for ioReults of errors not caused by a spun down drive would report right away and a timeout in case some other condition causes the same error as being spun down, so it's not caught in an infinite loop for errors not caused by the spindown.  Something like this... (Untested):
>
> const
>             A_Day = 1;
>           An_Hour = A_Day/24;
>          A_Minute = An_Hour/60;
>          A_Second = A_Minute/60;
>
> StartTime:=now;
> assignFile(txt,'/mnt/test.txt');
> {$I-}
> rewrite (txt);
> repeat
> application.processmessages;
> writeln (txt,'I''m so tired ...');
> io := ioResult;
> // showMessage ('--> '+IntToStr(io));
> until (io=NoSuchFileErrorCode) or (StartTime+(30*A_Second)<Now); {$I+}
> closeFile(txt); If io<>0 Then
>     DoSomethingAboutError;
>
> It seems strange that this would be an issue at all,  I would think the drive itself would cache any data written to it for long enough to get the drive spun up... seems very short sighted on hard drive design for this to be an issue at all, and even if the drive didn't do this,  it seems like the operating system should take care of it for you.
>
>
> James
>
> -----Original Message-----
> From: fpc-pascal <[hidden email]> On Behalf
> Of Brian
> Sent: Thursday, November 21, 2019 8:55 PM
> To: FPC-Pascal users discussions <[hidden email]>
> Subject: Re: [fpc-pascal] Writing to a drive which may be spun down
>
> On 11/21/19 2:38 PM, Winfried Bartnick wrote:
>
> <...>
>
>> writeln (txt,'I''m so tired ...');
>> closeFile(txt);
>> io := ioResult;
> Thanks Winni, that sounds the sort of idea I'm looking for...
>
> Brian.
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
> _______________________________________________
> fpc-pascal maillist  -  [hidden email]
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
_______________________________________________
fpc-pascal maillist  -  [hidden email] https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
_______________________________________________
fpc-pascal maillist  -  [hidden email]
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal