RPI3 migration failure from raspbian-9 to balenaos

Actually, I have followed the migration instructions in the link below and I have succeeded to migrate amd64/ubuntu16 cards without any troubles. But the problem occurs with rapsbian-9, the process (execution logs) seems to be fine, I have received “migration successful, please reboot system”. After the reboot, nothing changes, it booted to raspbian-9 and not from the modified initramfs.

Migration instructions: https://github.com/balena-io-playground/balena-migrate.

These are my steps run inside the RPI3 that i want to migrate :
1- git clone https://github.com/balena-io-playground/balena-migrate && cd balena-migrate/migratecfg
2- wget https://files.resin.io/resinos/raspberrypi3/2.43.0%2Brev1.dev/image/balena.img.zip # In order to download the balenaos rpi3 image
3- cp balena-migrate-sample.conf balena-migrate.conf # then updating the IMAGE_FILE environment variable and setting REQUIRE_NMGR_FILE=FALSE
4- ./balena-migrate
5- reboot

I appreciate any hint or help.

Hi there and thanks for reaching out!

As you may have already noticed, that repository is slated for deprecation. We should be able to help you migrate the device directly, I have reached out to some folks internally who may be able to help.

Stay tuned, we will reach out again with more information soon!

Hey @omarsastec, as my colleague mentioned, that repository is deprecated, and there’s a more recent tool in the works, just not quite there yet.

Still to unblock you, we’ve digged into the issue, and could replicate your problem. With a couple of small changes, we could made it work, see the quickfix branch on the tool you used: https://github.com/balena-io-playground/balena-migrate/tree/quickfix

Then you could basically modify your flow, currently checking out the quickfix branch, and use that. Here’s my expanded flow if that helps, hope I haven’t missed anything.

  1. git clone https://github.com/balena-io-playground/balena-migrate && cd balena-migrate/migratecfg && git checkout quickfix This is while we are checking the changes in the quickfix branch, once that is merged, this last part (the git checkout .... will not be needed)
  2. wget https://files.resin.io/resinos/raspberrypi3/2.43.0%2Brev1.dev/image/balena.img.zip In order to download the balenaos rpi3 image. This is unconfigured image, so will need a config.json
  3. unzip balena.img.zip && gzip balena.img
  4. cp balena-migrate-sample.conf balena-migrate.conf
    a. then updating the IMAGE_FILE=balena.img.gz environment variable
    b. setting BALENA_CONFIG=config.json and create that config.json file there from one config download from the dashboard (navigate to the target application, “Add Device” > “Advanced” > “Download configuration file only”)
    d. set NO_FLASH=FALSE and NO_SETUP=FALSE (can also add DEBUG and LOG_... settings if want to debug further, but not required)
  5. run the migrator sudo ./balena-migrate -c balena-migrate.conf

This succeeded for me on a Raspbian 9 Stretch & Raspberry Pi 3.

If it works for you, we’ll merge the changes, and no extra checkout will be needed, so would be eager to hear your feedback! And the new tool would be generally ore robust in the future.

A PR has been opened that incorporates these changes here https://github.com/balena-io-playground/balena-migrate/pull/2 - please try it out and let us know how it goes.

Hi there,

Thanks for the replies. I’ll be testing the fixes ASAP and return a feedback according to the result.



I confirm that the fixes solved the problem. Feel free to merge the PR.

I’ve got two questions btw:

  • What’s the purpose of adding “basename” to INITRAM_REQUIRED_PGMS ?
  • The sync was added to flush the buffer during the cp ? Means the persisted image was corrupted before the fix ?

Thank you.

Hey @omarsastec, glad that it works for you!

The PR is fixing things up:

  • basename needs to be added, as the stage2 updater uses it, but if not copied in, that step will be broken
  • sync is also needed to be added, as after file operations it’s good practice to call sync, so that all the changes are written to disk. If not called, the code can operate on half-written data, or move on while changes are not persisted (just kept in memory/buffers). It doesn’t mean data was corrupted, it was just incomplete (moving on too fast).

Hope this helps!

Hello and thanks for you support,

Actually, during balena-migrate execution, the gzipping process is working fine and a valid backup.tgz was created. The problem occurs after the reboot, the backup.tgz in /mnt/data/ is corrupted for compressed files with a large size.

tar -ztvf backup.tgz 
tar: unexpected end of file
tar: short 

I have tried that twice with different content to backup in each time. I noticed that the restored tgz file is truncated in both tries, and with a size of 127M.

I understand that the truncating is related to memory limit . Is there a chance to backup compressed content with size ~ 20 GB ? Any suggestion ? Disk repartition before stage 2 ?

I appreciate your help.

hey @omarsastec, are you trying to transfer ~20GB of data, hence that number you mention?

The amount of data that can be transferred depends on the available device memory, as the hard drive is modified completely in the migration process, thus everything that needs to be transferred have to fit into a limited amount of space.

What sort of thing are you trying to migrate over, if we may ask? Just to understand your use case better. Are these database files, videos, some other large files? Could any of those actually be part of your application data on the balena side? We appreciate any context that you could share.

Hey @imrehg,

The data type is mainly videos, so we can say that gzipping is useless here, and I am restricted from transferring it to an external storage.

Yeah, I would agree that external storage or some other way to transfer the videos are much better in this case. The backup is mostly conceived to provide a way of getting in settings files, or other smaller stuff that are unique to a device.

The tar/gzip’ing is used, as the device is restoring from a single file, that can hold all folder structure and other file metadata, the compression is sort of secondary aspect, anyways.

Like i have already mentioned, transferring data to an external storage isn’t yet an option.
Isn’t it possible to ensure partial disk repartition ? Means before proceeding to stage 2, we create a partition that holds backup data, then we proceed by installing balena in the left space.

Hi again,

Unfortunately, as mentioned, this is a deprecated project and we don’t support partial repartitioning. As Greg says, balenaOS expects to be installed on an entire media device, as it also dynamically reconfigures the remaining partition space at first boot. Because of this, any changes to the same media node being used to install balenaOS on will be destroyed.

Currently I believe your only option is to store this backup data on an external media node.

Best regards,