In my use case I need to flash a board that has 2 Hard Drives, one SSD and one HDD. We need to have the Balena OS installed in the SSD Drive and use the other drive for storage only (so in case that needs to be replaced it doesn’t break BalenaOS).
The issue comes when we use a flash drive to install BalenaOS. The SSD is in sdb and the HDD is in sda, and seems that flash process goes and install first in sda. Also there is a file here that seems to confirm it:
We need to modify the order to be first sdb and then sda. Is there a way to do this?
Hey, I’ve pinged the device team to find out more but I know historically at least the sda/sdb assignments could be affected based on the motherboard ports you plug your devices into, so you could try swapping their ports to see if that reverses the order
Thanks for the response! The issue is that one of the ports is for an SSD drive directly into the motherboard using and M.2 factor connector while our HDD uses a SATA connector… So we are unable to change them as far as I know…
Thanks @_Page, I honestly that being able to modify that would allow a great level of freedom to install BalenaOS not only in simple boards, but across a lot of devices and hardware setups…
The short term fix would be to modify the file itself on the usb drive you use to flash the image.
i.e. When you have flashed the flasher image on a usb flash drive using etcher.
You can then manually mount the USB stick and modify the flasher (will probably need a linux computer), and modify the script file and change the order to suit your needs.
I think it is the first line on this file.
flash-rootA/etc/resin-init-flasher.conf
Perhaps that file should be in the boot partition so people from windows can tweak it easily. hmmm.
Keep in mind, this is a short-term dodgy workaround and might not scale well.
Being able to change the flash order priority is something on our radar but there is a whole lot of architectural restructuring needed to make this configurable in a saleable way on a per application.
Some customers have lots of applications and various production lines. This manual step is a potential point of failure as it isn’t simply a case of download a new OS from the dashboard. Every time a new OS is downloaded from the dashboard, this workaround needs to be manually done.
Thanks @zubairlk! We thought that it could be a potential failure in our devices if we make the change. Thanks for letting us know, we will discuss internally if the option is worth the risk… And as you said, it would be nice to have flash-rootA/etc/resin-init-flasher.conf in the boot partition.
Can you share the output of the following command.
root@83c1fb9:~# ls /dev/disk/by-label/ -al
total 0
drwxr-xr-x 2 root root 200 Feb 12 11:29 .
drwxr-xr-x 7 root root 140 Feb 12 11:29 ..
lrwxrwxrwx 1 root root 10 Feb 12 11:29 flash-boot -> ../../sdb1
lrwxrwxrwx 1 root root 10 Feb 12 11:29 flash-rootA -> ../../sdb2
lrwxrwxrwx 1 root root 10 Feb 12 11:29 flash-rootB -> ../../sdb3
lrwxrwxrwx 1 root root 10 Feb 12 11:29 resin-boot -> ../../sda1
lrwxrwxrwx 1 root root 10 Feb 12 11:29 resin-data -> ../../sda6
lrwxrwxrwx 1 root root 10 Feb 12 11:29 resin-rootA -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 12 11:29 resin-rootB -> ../../sda3
lrwxrwxrwx 1 root root 10 Feb 12 11:29 resin-state -> ../../sda5
root@83c1fb9:~#
I’m also wondering something else. The requirement is to swap sda and sdb. But I’m wondering if the kernel might decide one thing is sda and one thing is sdb. And in another boot, the kernel might change its mind.
The culprit here could be udev. You’ll need a udev rule that will identify which device is which and should be assigned to sda. Then you shouldn’t need to modify the order. Just place a udev rule.
Yes, I have 3 now, the USB with the BalenaOS image, and SSD and an HDD. We need to flash our device in a way that the OS goes to sdb (SSD drive) and sda is left for usage of video recordings…
Ok I think the problem here is also that sda/sdb/sdc can be in any order…
If you reboot a few times, you should see them change.
The solution is probably a udev rule.
This goes further into hand-tuning the rule for that specific nuc/hardware combination…
I’m wondering if there is a manual workaround I can suggest.
hmm.
Flash a nuc flasher image on usb
Edit the file /etc/resin-init-flasher.conf in flash-root
remove sda sdb both. This is to make the service fail to prevent flashing. So that we can select the right device to flash!
Once device boots, manually check lsblk for which device is the ‘right’ one.
Remount fs read-write, mount -o remount,rw /
Modify /etc/resin-init-flasher.conf and put in the right sdX
Start flasher service systemctl start resin-init-flasher
The right solution would be to craft a udev rule.
And a more architectural solution for us would be to have some way to add more information via the dashboard/boot partition instead of the default priority order
Thanks for your answer, quick question, I’m not deeply familiar with udev rules, but will I have to make a different rule every time that I flash a device? Or with one rule added info flash-rootA will work every time that I use that flash drive to install BalenaOS? If yes, where do you recommend to save it?
Quick question, so why it always works for me when I leave the flash order un-touched? I mean, sda in front of sdb?
Hi,
regarding your question I am not quite sure myself, but the sequence of detection and naming disk is a timing issue and could produce all kinds of weird behavior.
Additional confusion can be caused by the already installed balenaOS on your harddisk. Balena will be looking for labels to mount its drives an after a successful installation might find the labels on both drives which would cause substantial unwanted confusion. So we should repartition the HDD and remove the resin labels before each attempt.
As to creating udev rules - I think they need to be placed in your config.json as explained here https://www.balena.io/docs/reference/OS/configuration/#udevrules
The challenge is to create the right rules and place them in the right position (determined by the number in front of each rule).
A starting point might be to take a look at your drives using udevadm info <device path> , eg udevadm info /dev/sda should give you a list of udev properties of that device. Now you have to work out udev filters that will distinguish your devices using the displayed properties and rename the SSD device to a name (eg. /dev/ssd1) that you can then prioritize in the list as done before.
Ways to distinguish:
You could go by something simple like serial but that would bind the rule to the serial and would have to be modified on every install to match the serial number.
You could go by model if you are always expecting the same drive model.
Best would be to create a rule that matches the controller it is attached to.
If you post the output of udevadm I might be able to help you create an appropriate rule, but to be honest with you I have very limited experience with creating and applying udev rules myself, so I might have to find and reach out to an expert.
Regards Thomas
Thanks for answer and your efforts! I’ll start testing UDEV rules and definitely update my rules here.
I’m researching and I think that devices (as well network interfaces) can’t be renamed, but a symlink could be created to the device using UDEV rules… Here is a mention to what I’m trying to say:
I need to test anyways, but if the symlink approach is the proper one, would this work when modifying the order in /etc/resin-init-flasher.conf? I’m also assuming that UDEV rules will be applied before the resin-init-flasher service starts, maybe is a dumb question, but is this correct?
You are correct. Devices cannot be renamed, creating a symlink is the way and the post gives you a good template for the required udev rule.
Regarding the timing the udev processing should be well finished by the time the flasherservice is started and I think putting your symlink name in the beginning of that list should do the job. That said - I have not tried this before - so there might still be surprises waiting for us…
Yeah, I’ll be working on that and checking if there is any other method that makes sense for us… Like flashing a device and then add the extra storage, there is a project that may work for this case:
