What happens when writing to a file during a power failure?

My program is essentially a data acquisition solution using the usb/serial bus. My program reads a data-stream from the serial port and appends that data to a string. When the user press “stop record” that data is then written to a file with a name, date & time stamp.

In this scenario if the laptop battery runs out or some other power failure occurs, then that precious data will never be saved. My software will most likely be used off-site or ‘in the field’ ie. outside and therefore used on a laptop.

So, my thoughts are is to have a timer that opens and closes a file periodically (maybe every 1-5 seconds), writes the data, closes the file, opens the file, writes, closes etc. etc.

But what happens to that file if it’s interrupted during a write - Does the file become corrupt? and “never” able to be opened because an EOF was never written?

Or are there other solutions that may be worth exploring?

Cheers.

  1. If the power outage occurs before file writing, that data is gone.

  2. I would imagine with a modern file system that if theres a power outage while writing, you’ll either get partial data, or none at all. I don’t expect the data to become corrupted, unless of course your reading program is looking a specific EOF marker, which your application wasn’t able to write in time.

If you use the textOutStream object, you can open the file using the “append” mode and write the text, then close it again. If you use a binarystream you’d need to skip to the end and then write the data.

Thanks Sam,

Yep! agreed that if the outage occurs before file writing - it is gone forever.

Getting partial data is ok, but none at all is not. If that CAN happen, then I need to implement a scenario where it CANNOT happen. It can’t be left up to possibilities.

I would be using textOutputStream and appending as suggested.

Maybe saving “incremental” files would be safer. ie recoveryfile01, recoveryfile02, etc. That way the user could open/import the last recovery file?

If you do changes to existing files, some FS will already do them atomic and there’s no “need” to do it by yourself.

But if we talk about bringing volatile memory to non-volatile memory, there’s afaik no 100% safe way to prevent data loss in case of emergency.

When writing to critical files and need to retain data, I typically work with temporary files which keeps a copy of the data in the event of a power outage. Here is an example of the steps:

  1. copy the original file and have a .tmp suffix on the temp file
  2. open the original file and append data
  3. confirm the file has been updated and delete the tmp file.

There are a few variations on the save and open methods. Some purists even suggest only making modifications to a second backup file and then overwrite the original file.

There are several ways to ensure the data is not lost:

  1. Use a modern file system. (I assume you’re using Windows? Please tag this post likewise!) E.g, prefer NTFS over FAT, as that has journaling mechanisms to make sure nothing is lost.
  2. Use an old file system such as FAT on an external disk, and configure the disk to be ejectable at all times, because then Windows would be flushing the data more often, ensuring it’s not just resting in the memory cache.
  3. Issue flush calls for the file and the volume you write to. Google that.
  4. After writing, close the file and unmount and remount the volume. That’ll ensure perfectly nothing is lost until that point, though it only works with external disks no one else writes to.

Good luck.

[quote=356238:@Thomas Tempelmann]There are several ways to ensure the data is not lost:

  1. Use a modern file system. (I assume you’re using Windows? Please tag this post likewise!) E.g, prefer NTFS over FAT, as that has journaling mechanisms to make sure nothing is lost.
  2. Use an old file system such as FAT on an external disk, and configure the disk to be ejectable at all times, because then Windows would be flushing the data more often, ensuring it’s not just resting in the memory cache.
  3. Issue flush calls for the file and the volume you write to. Google that.
  4. After writing, close the file and unmount and remount the volume. That’ll ensure perfectly nothing is lost until that point, though it only works with external disks no one else writes to.

Good luck.[/quote]

And work entirely on non-volatile memory. Store nothing important in any case in volatile memory. This makes things much slower but reduces the danger of data loss dramatically.
If you can’t choose to work on a FS with journaling mechanisms, write everything twice or more (backup, backup, backup,…) :slight_smile:

[quote=356237:@Eugene Dakin]copy the original file and have a .tmp suffix on the temp file
open the original file and append data
confirm the file has been updated and delete the tmp file.[/quote]
Good idea. I would abstract this in my libraries in order to make it common practice without paying attention every time.

ok a few words from the local storage person… There is several levels of caching when writing to a “disk”. If we are talking about local/direct attached disk (not talking about NAS/SAN/IP based storage). There is Application layer caching. There is cache at the filesystem level. There is a cache at the “disk” layer (think SCSI subsystem or IDE/SAS subsystem). Depending what O/S and Application will depend on if you can “flush” the IOs to the physical media (disk/SSD) or not.

during a power outage all those cache are lost and everything that is “written” but not flushed all the way to the physical media. Most O/S filesystems’ and disk subsystems write the IOs in a FIFO model (first in, first out). Application level writes will depend on the Application. Some arent FIFO based. Yeah that sucks when they aren’t.

best thing to do is has some sort of “battery” backup that will hold the system up long enough to flush the IOs through all the caches down to the physical media.

A quick question; how long does the recording usual take? Perhaps you could simply write the data the moment it comes in?

How much of the precious data will be entirely missed while the power is off? Perhaps a UPS is the best solution.

Thanks for everyone’s input.

I always knew this was going to be an issue - I’ve spent time on other parts of the program first, and now is the time to get it finally sorted.

Although I must confess that it’s an issue that may never arise, so perhaps I’m being a bit “dramatic”. Nevertheless, it should be implemented because the user could after doing a successful test recording, become distracted celebrating their own magnificence and forget to press the stop record button, battery goes flat… nothing is saved… their jubilation would be short lived.

The recordings would typically be around 60 secs or less. The data isn’t coming in at a high sample rate. I have a test file that was recorded for 10 minutes that weighs in at around 200kb - very small files.

So yes, writing the data as it comes in, to a temporary file is the best solution. This is exactly the approach I took when writing a previous version of my software some years ago in quickbasic. The reason I haven’t implemented it yet is because at the time of writing the code for capturing the data and drawing a graph etc. I didn’t know how to code file I/O operations using Xojo - I do now!

However, I’m still not clear about what happens during a file write and someone pulls the plug. So with an NTFS system the file should still be able to be read?

I remember something about “automatic” File Caching in Windows. So writing to a temp file may not elleviate any potential problem. I just found this:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa364218(v=vs.85).aspx

Can turn off file Write-caching on the users’ computer:

Thanks Cho, that option looks familiar from years back when I was dealing dealing with sound recording devices. I don’t think it applies in my case.

Let’s say that I’ve implemented the code for writing the data directly to a file and all is working as expected. An UPS (or is it A UPS?) does not resolve or answer the original question - because a UPS can also break down!

Here’s an analogy using an old analog tape recorder:
• I decide I’m going to record my latest greatest guitar composition
• I set the levels then press record
• I complete my composition (approx. 2 minutes)
• The phone rings and I get distracted, therefore forget to press stop record
• After ten minutes on the phone, there is a power blackout

When the power comes back on, I suddenly realise that I didn’t press stop record - but no big deal because I know the important recording is still on the tape somewhere. I then rewind the tape to that part and listen back to my wonderful performance. :slight_smile:

BUT
imagine if the situation was the same (as in the bullet points above) except, when the power came back on, I re-wound the tape to where the recording should have been, only to find, to my greatest alarm, there was nothing there!!

So then I do a bit of research and find out that this particular model of tape recorder has a peculiar habit of erasing the last recording on power up if there had been a power failure. Well, that situation would be utterly absurd and I would likely spend the rest of my life hunting down the perpetrator who created the said device and explaining to them my displeasure via the means of a clenched fist!

SO, this is where I’m at:

I will be implementing the “save as you go” approach. The file will be writing as it is reading from the usb port. But the issue still remains the same. If the user does not press stop, the file will not be closed and therefore a potential for corruption of the file?

Therefore, as I suggested earlier, would it not be better to do incremental saves, as in the temp file is incremented, ie. temp01, temp02, etc. that way it can at least be assured, that backup files are being created/written to/saved. This could be done say every 10 seconds. In the unlikely event of a power failure, the user could import a recovery file that is as the data file was, 10 seconds before the failure?

OR am I just over-thinking this?

With Write-caching, when your code write the data “directly” to a file, it actually write to the cache. The OS may report that done, written to file when it is not yet. Then Windows decide when to write “directly” to file. It can be very quick or may not be very quick. When the data is in transition in the cache and there is power failure, the data cannot be written “directly” to file any more.

With Write-caching, there is never a directly write to file. Even incremental temp files, the problem will still be there with the last batch or batches of data.

The incremental temp files when saved should be safe. I’m simply writing the “string” or memory to a file. The string is always appended to from the serial buffer. A new file can easily be saved by saving that string to a new file.

If this was done every ten seconds, then losing the last file is acceptable. I can’t see how more than one file would become corrupt?

As Sam suggested, instead of storing the data then writing it when completed, why not write the data as it arrives to an SQLite DB? The seems like it would be a perfect use-case for SQLite and writes to the DB are very fast. This way, you’ll have the data saved to the point until the power was lost. You could then create whatever file format you want from the raw data in the DB.

I would image there would be other benefits as well; store recordings/sessions by name, dates, etc.

Taken from:

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364439(v=vs.85).aspx?

I have no idea how TextOutputStream.Create works internally, nor what calls TextOutputStream.Flush makes.

If you want to ensure its on the disk, this is the only guaranteed method of writing the data to disk as I assume Xojo doesnt set these flags.

It might be worth putting in a feature request for unbuffered writes but I suspect it would be quicker just to use declares.

There wasn’t any better way of solving this issue other than through empirical testing.

So, after implementing code to write directly to a file (recoveryFile.temp) and then switching the power off during the recording just to see what would happen, I’m happy to report that after 10 or more of these “power-off” tests, the recoveryFile can be opened, and has not been corrupted. :slight_smile:

BUT, I discovered that my preferences file wasn’t being saved correctly. It didn’t make sense because the preferences file when written to is always saved immediately (or so I thought). After more testing and reading I realised that the preferences file needed to be flushed first.

writePrefs.Flush
writePrefs.Close

I would not have solved this issue without all the good advice, so thanks to everyone.

I chose Sams post:

[quote]Perhaps you could simply write the data the moment it comes in?[/quote] As “This answers my question”, but technically the question was “What happens when writing to a file during a power failure?” well, nothing bad as far as I’ve tested.

Cheers.

I’m really hoping you tested on a virtual machine rather than a physical one, to avoid potential damages (hardware or software)!