Data persistence error after sudden power cut

[crespum]

EDIT: Something I haven’t mentioned is that before cutting the power I print the file through ssh with cat and it’s definitely not empty and size increases as expected.

---

Thanks for the answer @thgreasi. I’m going to respond under your lines:

@thgreasi

I see that the GPS class preserves a different file for each day. Can you confirm whether the data loss is only about the current logfile or whether it also affects files from previous days?

Only the current file being edited is empty. The rest of the files are okay.

@thgreasi

As a first step, as you mentioned you could try adding an f.flushI() or even an os.fsync() call as the python docs suggest.
On the other hand since your GPS.dump() just opens the file using a with statement, which closes the file right after the single operation finishes, I would expect that all the buffers would have already been flushed.

I’ve tried adding f.flush() and it didn’t help. I’ll try os.fsync() today and let you know.

@thgreasi

Do you have any metrics about how much time might GPS.dump() need to complete on a production device?

No, not really, but I’ve tried to write every 5 seconds instead of every 1 second and the problem remains. I guess if it were a timing problem, 5 seconds to write and flush the contents should be more than enough.

@thgreasi

As far as I can tell, you loop rewrites the file from scratch, is that correct?
That would not only might reduce the lifespan of your storage media, but also increases the chance that the file gets corrupted if there is a sudden power loss.
Ideally you should only append the new records to the file instead of rewriting its whole content.

I’m aware of this, I just wanted to run a quick demo. I’ve also tried to open the file for appending data but the behavior is similar. In this case, the file is not completely empty but it’s still missing some lines.

@thgreasi

An easier approach that would also confirm our suspicions could be to always have a secondary file to write the new data into and then swap it with the primary one. For example, you could always dump your data on a file named gps_log_%Y%m%d_next.json , and then once that’s flushed, rename the files to make the _next be the proper gps_log_%Y%m%d.json .

Indeed. I am just trying to understand first if I’m missing something.

Thanks again!