Question about etcher-sdk & reading a file from SD card in cross-platform way

balenaEtcher
1

Hello,

I am trying to develop an application that uses etcher-sdk to write SD cards for Raspberry Pi and similar SBCs. I wanted to embed this capability in a larger application rather than just telling my users to use balena etcher.

Another requirement I had: I wanted to write custom configuration files to the SD card on top of the disk image. So far I was able to achieve this by using the balena-image-fs module to modify the disk image file itself before I write it to the SD card. This is great because in theory it doesn’t depend on the linux OS to mount the disk image and write the files to the ext-formatted partition. It should work on Windows too.

So that’s great, I’m really happy with it so far.

The next difficult step for me: I want to also read files off of the SD card after it has been flashed. Why? In short, because that custom configuration which was written to the disk image may not actually work when the machine boots. For whatever reason: maybe the user typed the wifi credential incorrectly. Or maybe my code has a bug. Or maybe a goblin bit the wifi chip off of the PCB. Who knows. Point is, it’s not guaranteed to be able to work or connect to the internet. So in case that happens, it is imperative that the user can easily figure out what happened. My plan was, I would write a script to /etc/rc.local, then that script would run on boot & append lines to /var/log/mylog.txt on the SD card as it proceeds through its process. Then when the process fails, the user can simply turn off the SBC, remove the SD card, plug it back into their computer, and then my software will recognize it , read the /var/log/mylog.txt file and display it.

Of course, I want this to work no matter whether my application is running under linux, windows, or Mac OS. I have been hacking away at this problem for a few days exploring lots of different modules and I haven’t found anything that looks like I can make it work very easily.

Just today, I had a new idea, what if I could use the etcher SDK in reverse, to create a new disk image from the SD card. It’s very silly, but it would potentially work as I can definitely read files from the disk image using balena-image-fs .

Anyway, does anyone have any advice or thoughts ?? Please let me know!

4

I have another idea, maybe I can make the /etc/rc.local script create a new FAT partition on the SD card dedicated to logs? Then perhaps mac/windows/linux can all mount and read it natively? I just worry about what happens if that process of creating the FAT partition fails for whatever reason. But maybe that wont happen. IDK

6

As you said, read the file using balena-image-fs, fat and ext partitions are supported on all platforms.
You can open the device file directly with sufficient privileges.

import { interact } from 'balena-image-fs';
import { promisify } from 'util';

async function main(device: string, partition: number | undefined) {
    await interact(device, partition, async (fs) => {
        const readdirAsync = promisify(fs.readdir);
        const files = await readdirAsync('/');
        console.log(files);
    });
}

main('/dev/sda', 1);
8

whoah, I had no idea one could do that !! I thought because it was balena image fs it was limited to images. Also because the underlying primitive class it works on is called FileDisk. Maybe I will raise a PR to update the readme to demonstrate that this is possible since it seems to me, unless you already knew, you would just have to guess that it works and test it.

Sounds like that will work on Windows and Mac OS as well? really cool if true. I will give it a try.

9

Good that this works for you. Don’t hesitate to contact us if you run into any other problems…

11

First of all I have to say thank you for the work you do on this code, in the open and licensed so we can use it. And thank you for taking the time to reply to questions from curious developers like myself.

I was about to reply and say that it doesn’t work on my system, but in fact it does – it’s just extremely finicky. Lots of things that work 100% of the time with the normal fs module don’t seem to work as easily / reliably with the balena-image-fs.

I was going to write a big long post with lots of examples of bugs I was running into in my app. However, most of those problems I was able to solve when I ripped out the offending code and ran it in a simpler, smaller environment. So it was probably bugs in my code not bugs in balena-image-fs.

I think poor error messaging is a contributor to this “perception of bugs” even when the bugs aren’t really there. For example:

with the fs module when you try to create a folder that already exists with mkdir() it throws Error: EEXIST: file already exists, mkdir './testdir'.

But the balena-image-fs throws Error: , Unknown error -2133571407

Here are all the issues I was able to reproduce reliably on linux with an ext4 formatted SD card partition:

Writing a file which is larger than 64kb. Every time I try to write a larger file like a png image or executable binary it gets truncated to 65.5kb. No error is thrown but the file gets truncated. In the following code, desk.png is a 1MB png file.

const balenaImageFs = require('balena-image-fs');
const fs = require('fs');

async function main(device, partition) {
    await balenaImageFs.interact(device, partition, async (imageFs) => {

      const writeStream = imageFs.createWriteStream("/opt/desk.png");
  
      const readStream = fs.createReadStream("desk.png");
      readStream.pipe(writeStream);
  
      await new Promise((resolve, reject) => {
        writeStream.on('error', reject);
        writeStream.on('close', resolve);
      });


    }).catch(err => {console.log(err.stack)});
    
}

main('/dev/sdc', 1);

symlinks are unimplemented, this is fine but it made me sad because I wanted to create a pre-enabled systemd service and apparently systemd uses symlinks to signify that a service is enabled.

This isn’t a blocking bug but during all of my tests today I was getting stderr output in my terminal like /opt/test-file.txt: Unknown code ext2 76 when writing files was actually working fine.

12

Thanks for your feedback. Would you mind creating issues for the above observations on https://github.com/balena-io-modules/balena-image-fs