takeover script "Failed to remove takeover directory:"

giuseppe · February 15, 2024, 11:19am

Hi,

we are planning on using the takeover script to migrate some devices to balenaOS mostly RPis 3s running devuan GNU. But i seem to be running into some problem around the end of stage 1.
the command i am using looks like this

sudo ./takeover --config config.json  --no-os-check --version 5.1.20+rev1 -i raspberrypi3-5.1.20+rev1-v15.3.0.img.gz --no-nwmgr-check --no-wifis

Everything goes pretty smoothly until

2024-02-15 08:49:53 INFO  Bind-mounted new init as '/balena-takeover/bin/takeover'
2024-02-15 08:49:53 WARN  [takeover::stage1::migrate_info] Failed to remove takeover directory: '/balena-takeover', error : Os { code: 16, kind: ResourceBusy, message: "Resource busy" }
2024-02-15 08:49:53 ERROR [takeover] Migrate stage 1 returned an error: A spawned process returned an error code, context: Call to telinit failed, stderr: error: Found argument 'u' which wasn't expected, or isn't valid in this context

USAGE:
    telinit [FLAGS] [OPTIONS]

For more information try --help

if run again the error message changes to:

2024-02-15 09:02:53 ERROR [takeover] Migrate stage 1 returned an error: An error occurred in an upstream function, context: Failed to bind-mount '/balena-takeover/bin/takeover' to '/sbin/init'
  caused by: ENOENT: No such file or directory

A reboot of the device usually gets it back to the first error.

Interestingly after the failures the command ‘telinit’ functions equivalent to ‘takeover’, but this is probably intended?

Not sure what could be the problem honestly.

mpous · February 15, 2024, 11:38am

Hello @giuseppe first of all welcome to the balena community!

Let me check internally with the devices team how we can help you more.

kb2ma · February 15, 2024, 1:11pm

Hi, thanks for letting us know of the problem. A few follow-ups:

What version of devuan are you using? 32-bit or 64-bit?
What model of RPi 3, how much RAM?
Please post the full log so we can look for any other issues
After the error occurs, run an ls -R /balena-takeover so we can see the contents of that directory

giuseppe · February 15, 2024, 1:48pm

Hi, thanks for the quick response.
The device is a RPi 3 B with 1gb of ram running a 32-bit OS

2024-02-15 14:25:43 INFO  Detected OS Architecture is ARMHF
2024-02-15 14:25:43 DEBUG [takeover::stage1::device_impl::raspberrypi] raspberrypi::is_rpi: entered with model string: 'Raspberry Pi 3 Model B Rev 1.2'
2024-02-15 14:25:43 DEBUG [takeover::stage1::device_impl::raspberrypi] raspberrypi::is_rpi: selection entered with string: '3'
2024-02-15 14:25:43 INFO  Identified RaspberryPi 3
2024-02-15 14:25:43 DEBUG [takeover::common] parse_file: line: 'PRETTY_NAME="Devuan GNU/Linux ascii"'
2024-02-15 14:25:43 INFO  Detected OS name is Devuan GNU/Linux ascii
2024-02-15 14:25:43 WARN  [takeover::stage1::device_impl] The OS 'Devuan GNU/Linux ascii' has not been tested with takeover for device type IntelNuc, proceeding due to no-os-check option
2024-02-15 14:25:43 INFO  Detected device type: Raspberry Pi 3,
2024-02-15 14:25:43 INFO  Configured for application id: 7
2024-02-15 14:25:43 INFO  connection to api: api.*****:80 is ok
2024-02-15 14:25:43 INFO  connection to vpn: vpn.*****:443 is ok
2024-02-15 14:25:43 INFO  config.json is for device type raspberrypi3
2024-02-15 14:25:43 DEBUG [takeover::stage1::migrate_info] image path: '/takeover/raspberrypi3-5.1.20+rev1-v15.3.0.img.gz'
2024-02-15 14:25:43 WARN  [takeover::stage1::migrate_info] No Network manager files were found, the device might not be able to come online
2024-02-15 14:25:43 INFO  Writing hostname to config.json: '*****'
2024-02-15 14:25:43 DEBUG [takeover::common] parse_file: line: 'PRETTY_NAME="Devuan GNU/Linux ascii"'
takeover will prepare your device for migration. Are you sure you want to migrate this device: [Y/n]
y
2024-02-15 14:25:45 INFO  Preparing for takeover..
2024-02-15 14:25:45 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "", stderr: "" }
2024-02-15 14:25:45 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(256)), stdout: "\tnot a dynamic executable\n", stderr: "" }
2024-02-15 14:25:45 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "\tlinux-vdso.so.1 (0x7ee7a000)\n\tlibc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76de2000)\n\t/lib/ld-linux-armhf.so.3 (0x76eee000)\n", stderr: "" }
2024-02-15 14:25:45 DEBUG [takeover::stage1::exe_copy] get_libs: no path for linux-vdso.so.1
2024-02-15 14:25:45 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "\tstatically linked\n", stderr: "" }
2024-02-15 14:25:45 DEBUG [takeover::stage1::exe_copy] get_libs: lib is statically linked: '/lib/ld-linux-armhf.so.3'
2024-02-15 14:25:45 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "\t/lib/ld-linux-armhf.so.3 (0x76ef5000)\n\tlinux-vdso.so.1 (0x7efe8000)\n", stderr: "" }
2024-02-15 14:25:45 DEBUG [takeover::stage1::exe_copy] get_libs: no path for linux-vdso.so.1
2024-02-15 14:25:45 DEBUG [takeover::stage1] Space required for commands: 8269 KiB
2024-02-15 14:25:45 INFO  Found 935 MiB total, 137 MiB free memory
2024-02-15 14:25:45 DEBUG [takeover::common::system] mkdir: '/balena-takeover'
2024-02-15 14:25:45 INFO  Using '/balena-takeover' as takeover directory
2024-02-15 14:25:45 INFO  Mounted tmpfs file system on '/balena-takeover'
2024-02-15 14:25:45 INFO  Created mtab in  '/balena-takeover/etc/mtab'
2024-02-15 14:25:45 INFO  Mounted proc file system on '/balena-takeover/proc'
2024-02-15 14:25:45 INFO  Mounted tmpfs file system on '/balena-takeover/tmp'
2024-02-15 14:25:45 INFO  Mounted sys file system on '/balena-takeover/sys'
2024-02-15 14:25:45 INFO  Mounted dev file system on '/balena-takeover/dev'
2024-02-15 14:25:45 DEBUG [takeover::common::debug] After dev mount
2024-02-15 14:25:45 DEBUG [takeover::common::debug] check if /dev/loop-control exists in new /dev: true
2024-02-15 14:25:45 DEBUG [takeover::common::debug] open /dev/loop-control succeeded
2024-02-15 14:25:45 INFO  Mounted devpts file system on '/balena-takeover/dev/pts'
2024-02-15 14:25:45 INFO  Created directory '/balena-takeover/mnt/old_root'
2024-02-15 14:25:45 DEBUG [takeover::stage1::exe_copy] copy_file: copying '/lib/ld-linux-armhf.so.3' to '/balena-takeover/lib/ld-linux-armhf.so.3'
2024-02-15 14:25:45 DEBUG [takeover::stage1::exe_copy] copy_file: copying '/lib/arm-linux-gnueabihf/libc.so.6' to '/balena-takeover/lib/arm-linux-gnueabihf/libc.so.6'
2024-02-15 14:25:46 INFO  Copied '/takeover/takeover' to '/balena-takeover/bin/takeover'
2024-02-15 14:25:46 INFO  Copied '/bin/dd' to '/balena-takeover/bin/dd'
2024-02-15 14:25:46 INFO  Copied config.json to '/takeover/config.hEWNBAdKXo.json'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/boot", fs_type: "vfat" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/mnt/opt", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/opt", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/mnt/backup", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/backup", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/mnt/docker", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/docker", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/mnt/log", fs_type: "f2fs" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/var/log", fs_type: "f2fs" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::mount] from_mtab: processing mount Mount { mountpoint: "/var/lib/docker", fs_type: "ext4" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] new: Root device number is: 179:2
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p4: PTTYPE=\"dos\" PARTUUID=\"7c4eb51a-04\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p4 got param name: PTTYPE, value: dos
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'PTTYPE'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p4 got param name: PARTUUID, value: 7c4eb51a-04
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p4 got PartitionInfo { uuid: None, block_size: None, fs_type: None, label: None, part_uuid: Some("7c4eb51a-04") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p4"' in '/sys/block/mmcblk0/mmcblk0p4'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p2: LABEL=\"os\" UUID=\"2ce5fb89-ba32-42dd-b08e-62d98fc94c8f\" TYPE=\"ext4\" PARTUUID=\"7c4eb51a-02\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p2 got param name: LABEL, value: os
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'LABEL'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p2 got param name: UUID, value: 2ce5fb89-ba32-42dd-b08e-62d98fc94c8f
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p2 got param name: TYPE, value: ext4
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p2 got param name: PARTUUID, value: 7c4eb51a-02
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p2 got PartitionInfo { uuid: Some("2ce5fb89-ba32-42dd-b08e-62d98fc94c8f"), block_size: None, fs_type: Some("ext4"), label: None, part_uuid: Some("7c4eb51a-02") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p2"' in '/sys/block/mmcblk0/mmcblk0p2'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p7: LABEL=\"log\" UUID=\"1f10ea08-5c42-4079-80d0-a050feb73b2a\" TYPE=\"f2fs\" PARTUUID=\"7c4eb51a-07\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p7 got param name: LABEL, value: log
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'LABEL'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p7 got param name: UUID, value: 1f10ea08-5c42-4079-80d0-a050feb73b2a
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p7 got param name: TYPE, value: f2fs
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p7 got param name: PARTUUID, value: 7c4eb51a-07
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p7 got PartitionInfo { uuid: Some("1f10ea08-5c42-4079-80d0-a050feb73b2a"), block_size: None, fs_type: Some("f2fs"), label: None, part_uuid: Some("7c4eb51a-07") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p7"' in '/sys/block/mmcblk0/mmcblk0p7'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p5: LABEL=\"opt\" UUID=\"ae113f66-2244-4e50-bb9f-2288f11c9937\" TYPE=\"ext4\" PARTUUID=\"7c4eb51a-05\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p5 got param name: LABEL, value: opt
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'LABEL'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p5 got param name: UUID, value: ae113f66-2244-4e50-bb9f-2288f11c9937
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p5 got param name: TYPE, value: ext4
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p5 got param name: PARTUUID, value: 7c4eb51a-05
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p5 got PartitionInfo { uuid: Some("ae113f66-2244-4e50-bb9f-2288f11c9937"), block_size: None, fs_type: Some("ext4"), label: None, part_uuid: Some("7c4eb51a-05") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p5"' in '/sys/block/mmcblk0/mmcblk0p5'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p3: LABEL=\"docker\" UUID=\"60c4fcb1-4573-4a70-b740-f373a9910575\" TYPE=\"ext4\" PARTUUID=\"7c4eb51a-03\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p3 got param name: LABEL, value: docker
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'LABEL'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p3 got param name: UUID, value: 60c4fcb1-4573-4a70-b740-f373a9910575
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p3 got param name: TYPE, value: ext4
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p3 got param name: PARTUUID, value: 7c4eb51a-03
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p3 got PartitionInfo { uuid: Some("60c4fcb1-4573-4a70-b740-f373a9910575"), block_size: None, fs_type: Some("ext4"), label: None, part_uuid: Some("7c4eb51a-03") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p3"' in '/sys/block/mmcblk0/mmcblk0p3'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p1: SEC_TYPE=\"msdos\" LABEL=\"boot\" UUID=\"6B69-7641\" TYPE=\"vfat\" PARTUUID=\"7c4eb51a-01\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p1 got param name: SEC_TYPE, value: msdos
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'SEC_TYPE'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p1 got param name: LABEL, value: boot
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'LABEL'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p1 got param name: UUID, value: 6B69-7641
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p1 got param name: TYPE, value: vfat
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p1 got param name: PARTUUID, value: 7c4eb51a-01
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p1 got PartitionInfo { uuid: Some("6B69-7641"), block_size: None, fs_type: Some("vfat"), label: None, part_uuid: Some("7c4eb51a-01") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p1"' in '/sys/block/mmcblk0/mmcblk0p1'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p8: UUID=\"d0aeaccb-0778-49f5-88f4-4fc974c45f9d\" TYPE=\"ext4\" PARTUUID=\"7c4eb51a-08\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p8 got param name: UUID, value: d0aeaccb-0778-49f5-88f4-4fc974c45f9d
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p8 got param name: TYPE, value: ext4
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p8 got param name: PARTUUID, value: 7c4eb51a-08
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p8 got PartitionInfo { uuid: Some("d0aeaccb-0778-49f5-88f4-4fc974c45f9d"), block_size: None, fs_type: Some("ext4"), label: None, part_uuid: Some("7c4eb51a-08") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p8"' in '/sys/block/mmcblk0/mmcblk0p8'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(0)), stdout: "/dev/mmcblk0p6: LABEL=\"backup\" UUID=\"b58ea293-2787-496e-8828-c00e1b0e9cac\" TYPE=\"ext4\" PARTUUID=\"7c4eb51a-06\"\n", stderr: "" }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p6 got param name: LABEL, value: backup
2024-02-15 14:25:46 WARN  [takeover::stage1::block_device_info::partition] unexpected parameter name found: 'LABEL'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p6 got param name: UUID, value: b58ea293-2787-496e-8828-c00e1b0e9cac
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p6 got param name: TYPE, value: ext4
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: /dev/mmcblk0p6 got param name: PARTUUID, value: 7c4eb51a-06
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info::partition] PartitionInfo::new: for /dev/mmcblk0p6 got PartitionInfo { uuid: Some("b58ea293-2787-496e-8828-c00e1b0e9cac"), block_size: None, fs_type: Some("ext4"), label: None, part_uuid: Some("7c4eb51a-06") }
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] found  partition '"mmcblk0p6"' in '/sys/block/mmcblk0/mmcblk0p6'
2024-02-15 14:25:46 DEBUG [takeover::stage1::block_device_info] new: got device: BlockDevice { name: "mmcblk0", device_num: DeviceNum { major: 179, minor: 0 }, mounted: None, parent: "None" }
2024-02-15 14:25:46 DEBUG [takeover::stage1] Stage 2 config: 
---
log_dev: ~
log_level: DEBUG
flash_dev: /dev/mmcblk0
pretend: true
umount_parts:
  - dev_name: /dev/mmcblk0p3
    mountpoint: /var/lib/docker
    fs_type: ext4
  - dev_name: /dev/mmcblk0p1
    mountpoint: /boot
    fs_type: vfat
  - dev_name: /dev/mmcblk0p7
    mountpoint: /var/log
    fs_type: f2fs
  - dev_name: /dev/mmcblk0p5
    mountpoint: /opt
    fs_type: ext4
  - dev_name: /dev/mmcblk0p6
    mountpoint: /backup
    fs_type: ext4
  - dev_name: /dev/mmcblk0p2
    mountpoint: /
    fs_type: ext4
work_dir: /takeover
image_path: /takeover/raspberrypi3-5.1.20+rev1-v15.3.0.img.gz
config_path: /takeover/config.hEWNBAdKXo.json
backup_path: ~
tty: /dev/pts/0

2024-02-15 14:25:46 INFO  Wrote stage2 config to '/balena-takeover/stage2-config.yml'
2024-02-15 14:25:46 INFO  Bind-mounted new init as '/balena-takeover/bin/takeover'
2024-02-15 14:25:46 DEBUG [takeover::stage1] calling 'telinit u'
2024-02-15 14:25:46 DEBUG [takeover::common] call: output: Output { status: ExitStatus(unix_wait_status(256)), stdout: "", stderr: "error: Found argument 'u' which wasn't expected, or isn't valid in this context\n\nUSAGE:\n    telinit [FLAGS] [OPTIONS]\n\nFor more information try --help\n" }
2024-02-15 14:25:46 WARN  [takeover::stage1::migrate_info] Failed to remove takeover directory: '/balena-takeover', error : Os { code: 16, kind: ResourceBusy, message: "Resource busy" }
2024-02-15 14:25:46 ERROR [takeover] Migrate stage 1 returned an error: A spawned process returned an error code, context: Call to telinit failed, stderr: error: Found argument 'u' which wasn't expected, or isn't valid in this context

USAGE:
    telinit [FLAGS] [OPTIONS]

For more information try --help

As for /balena-takeover its completely empty, unless ran with --no-cleanup then its completely full.

Again thanks for the support.

kb2ma · February 15, 2024, 4:22pm

That’s helpful. As I read it now, there may be some incompatibility with Devuan’s implementation of telinit. What does telinit --help show?

giuseppe · February 15, 2024, 5:04pm

Well it depends, before running takeover or after rebooting the expected behaviour

root@****:~# telinit --help
telinit: invalid option -- '-'
Usage: telinit {-e VAR[=VAL] | [-t SECONDS] {0|1|2|3|4|5|6|S|s|Q|q|A|a|B|b|C|c|U|u}}

After running takeover :

root@****:/takeover# telinit --help
takeover 0.1.4
Thomas Runte <thomasr@balena.io>


USAGE:
    telinit [FLAGS] [OPTIONS]

FLAGS:
    -d, --download-only     Download image only, do not check device and migrate
    -h, --help              Prints help information
        --no-ack            Scripted mode - no interactive acknowledgement of takeover
        --no-api-check      Do not check if balena API is available
        --no-cleanup        Debug - do not cleanup after stage1 failure
        --no-efi-setup      Do not setup EFI boot
        --no-keep-name      Do not migrate host-name
        --no-nwmgr-check    Do not check network manager files exist
        --no-os-check       Do not check if OS is supported
        --no-vpn-check      Do not check if balena VPN is available
        --no-wifis          Do not create network manager configurations for configured wifis
        --pretend           Pretend mode, do not flash device
        --stage2            Internal - stage2 invocation
        --tar-internal      Use internal tar instead of external command

OPTIONS:
        --backup-cfg <BACKUP-CONFIG>     Backup configuration file
        --check-timeout <TIMEOUT>        API/VPN check timeout in seconds.
    -c, --config <CONFIG_JSON>           Path to balena config.json
    -f, --flash-to <INSTALL_DEVICE>      Use INSTALL_DEVICE to flash balena to
    -i, --image <IMAGE>                  Path to balena-os image
        --log-file <LOG_FILE>            Set stage1 log file name
        --log-level <log-level>          Set log level, one of [error,warn,info,debug,trace] [default: info]
    -l, --log-to <LOG_DEVICE>            Write stage2 log to LOG_DEVICE
        --nwmgr-cfg <NWMGR_FILE>...      Supply a network manager file to inject into balena-os
        --s2-log-level <s2-log-level>    Set stage2 log level, one of [error,warn,info,debug,trace]
    -v, --version <VERSION>              Version of balena-os image to download
        --wifi <SSID>...                 Create a network manager configuration for configured wifi with SSID
    -w, --work-dir <DIRECTORY>           Path to working directory

It is an interesting bug for sure. Again thanks for the help

lmbarros · February 15, 2024, 5:30pm

That’s odd. The takeover logs suggest that Devuan’s version of telinit does not support the u command, but the help message you just shared suggests it does.

But to be honest I am not sure what telinit takeover tried to run, since the error message in the logs say For more information try --help, but from your last message it seems that --help is not accepted by Devuan’s telinit.

I would be interested in trying to reproduce this myself (though I might be able to do that only next week). In any case, I’d love to know what exact version of Devuan you are running on this device. I never used Devuan, but in many distros you can get the version with

cat /etc/issue

or

cat /etc/os-release

Thanks!

giuseppe · February 15, 2024, 6:00pm

ts an a strange thing isnt it? it seems like by the end telinit belives its takeover.

The Devuan version is ASCII (which is the main reason to get all these devices migrated ). The OS might be a bit modified (not sure before my time), I will be running some test on vanilla Devuan ASCII tomorrow. Will post if anything comes of it.

Thanks again

lmbarros · February 15, 2024, 6:10pm

At some point during the process, takeover will take the role of the init process. Details are not clear, but I’d guess at this point it is already running as init (PID 1), and this will probably help to explain how we got at this state.

Thanks! I’ve got a couple things on my plate right right now, but I’ll try to reproduce this as soon as I can!

Out of curiosity, would you mind to share a little more about project on a higher level? And how takeover can help you with the problem you have at hand?

We have been discussing the role of takeover and similar tools, so knowing a bit more about your use case would be super helpful!

giuseppe · February 16, 2024, 9:32am

Alright ran some tests here. Yes this seems to be a devuan ASCII problem, as vanilla version of it also suffer from this bug.

Out of curiosity, would you mind to share a little more about project on a higher level? And how takeover can help you with the problem you have at hand?

We got a couple thousand RPis out there running this 32-bit devuan with a in-house solution for fleet management, that up until now was good enough, but bare bones.
We are currently looking if the balena ecosystem would be a good replacement. And takeover would be perfect to facilitated that as it would give us a convenient way to migrate all these devices and not only that but being able set up those data backups on top of that.

lmbarros · February 19, 2024, 7:55pm

Hey, thanks for sharing this background info!

FYI, I was able to reproduce your scenario here. Planning to dig into this over the next couple of days! We’ll let you know of our findings!

lmbarros · February 20, 2024, 11:10pm

Some good progress here!

The root cause of the issue we are seeing with Devuan is the fact that in this distro, telinit is a symlink to init. As I mentioned before, at some point in the process, takeover takes the place of init (by bind-mounting on top of it). Then, the next step is to call telinit u to actually “become” the init process. But, because of the bind-mount, now the tellinit effectively points to takeover, so the telinit u call fails. As you said:

strange thing isnt it? it seems like by the end telinit belives its takeover.

Well, that’s because in a way it is

Anyway, I implemented a quick hack here to sanity check this find: basically I make a copy of init/telinit before bind-mounting, and then call this copy instead. And with this hack in place, I was able to run a simple Devuan-balenaOS migration on a device!

We’ll need a few more days to get this properly addressed (I want to make this change of top some other changes we happen to be doing at the moment), but it seems we can make this work! We’ll keep you up-to-date!

Oh, and by the way, if want to try this yourself (maybe to experiment with the process after this point), I believe you can skip this error on the OS level, by replacing the symlink telinit with a renamed copy of init before running takeover.

Also, I guess you understand this part, but please just be careful with all your takeover tests. Do don’t try on a production device until we get the whole process figured out and run successfully with some test devices first!

giuseppe · February 21, 2024, 8:40am

Anyway, I implemented a quick hack here to sanity check this find: basically I make a copy of init/telinit before bind-mounting, and then call this copy instead. And with this hack in place, I was able to run a simple Devuan-balenaOS migration on a device!

Damn, you guys work fast. I guess i will keep an eye on that branch for a now.

Oh, and by the way, if want to try this yourself (maybe to experiment with the process after this point), I believe you can skip this error on the OS level, by replacing the symlink telinit with a renamed copy of init before running takeover.

Just tried this and i can report it works. The device migrated with no problem.

Again, thanks for all the help. Amazing work!

lmbarros · March 1, 2024, 11:54am

Hi @giuseppe ! We have merged the PR that handles this issue of telinit being a symlink to init. So, using the latest release of takeover shall now work with Devuan. Please, if you try it, let us know about your experience.

giuseppe · March 4, 2024, 7:42am

Yup, just tried it. Seems like everything is working now.

Thanks again