Partition not found: 12. error at configuration of custom Balena OS image for the jetson nano

Langhalsdino · March 9, 2021, 11:18am

Hi balena community,

it’s been a while since i build my last balena OS image (approx 1 year), therefore i might skipped through some major changes in the balena-jetson repo, that caused this issue.

What is the current behavior?

I checked out the v2.69.1+rev4 release and a new device (.svg, .coffee and .json).

These files are an exact copy of the jetson-nano files, only the name was replaced by custom-nano and the following flags have been added:

	"state": "RELEASED",
	"community": "true",
	"private": "true",

Furthermore the meta-balena-jetson layer has been appended with the conf/machine accordingly:

#@TYPE: Machine
##@NAME: custom-nano
##@DESCRIPTION: Machine configuration for custom Jetson Nano

MACHINEOVERRIDES = "jetson-nano:${MACHINE}"
require conf/machine/jetson-nano.conf

After succesfully building the image on a powerfull server, i copied the resulting resin-image-custom-nano.resinos-img to my laptop. When configuring the image a couple of strange things are happening before the configuration fails

balena os configure ~/Downloads/resin-image-custom-nano.resinos-img--application testApp --debug

Could not find OS version:

Could not read OS version from the image. Please specify the balenaOS
version manually with the --version command-line option.

ExpectedError: Could not read OS version from the image. Please specify the balenaOS
version manually with the --version command-line option.
    at getOsVersionFromImage (/usr/local/lib/balena-cli/build/commands/os/configure.js:224:15)

After adding the current version 2.69.1 to the image the 12th partition can not be found.
balena os configure ~/Downloads/resin-image-custom-nano.resinos-img --application testApp --version 2.69.1 --debug

Partition not found: 12.

PartitionNotFound: Partition not found: 12.
    at /usr/local/lib/balena-cli/node_modules/resin-image-fs/node_modules/partitioninfo/build/partitioninfo.js:144:23
    at Generator.next (<anonymous>)
    at fulfilled (/usr/local/lib/balena-cli/node_modules/tslib/tslib.js:111:62)
From previous event:
    at Object.exports.interact (/usr/local/lib/balena-cli/node_modules/resin-image-fs/build/driver.js:125:24)
    at write (/usr/local/lib/balena-cli/node_modules/resin-image-fs/build/imagefs.js:170:31)
    at /usr/local/lib/balena-cli/node_modules/resin-image-fs/build/imagefs.js:206:14
From previous event:
    at Object.exports.write (/usr/local/lib/balena-cli/node_modules/resin-image-fs/build/imagefs.js:203:20)
    at Object.exports.writeConfigJSON (/usr/local/lib/balena-cli/node_modules/balena-device-init/build/utils.js:179:18)
    at /usr/local/lib/balena-cli/node_modules/balena-device-init/build/init.js:87:18
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
From previous event:
    at Object.exports.configure (/usr/local/lib/balena-cli/node_modules/balena-device-init/build/init.js:82:6)
    at OsConfigureCmd.run (/usr/local/lib/balena-cli/build/commands/os/configure.js:75:55)
    at process._tickCallback (internal/process/next_tick.js:68:7)

For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting

While inspecting the partitions and their content and comparing it to the stock image i could not see any differences. In this case i compared them not entirely only by visual inspection of some random files.

Other information on the environment

I am currently using balena cli version 12.40.2 and working on ubuntu (server and laptop).
Eventhough other modifications are planed (device tree, kernal, …) this happens with no configurations to the the layers.

I successfully build working images for the jetson nano one year ago and just wanted to upgrade to the new kernel, docker environment, … (i know i should have done this more often).

Can you help me debugging this problem?

klutchell · March 10, 2021, 6:49pm

Hey there, thanks for reaching out! Have you tried running that image in unmanaged mode to see if it’s just an issue with the os configure step?

If you just take the output from the build and flash it straight to an SD card, we can then see if the device boots and is discoverable via balena scan. It may help to have a monitor and ethernet connected at this point. Then we can balena join to add it to your cloud app, etc.

dtischler · March 10, 2021, 6:50pm

(Also, great to hear from you Frederic!)

Langhalsdino · March 11, 2021, 8:14am

I wanted to try this, but some how I did not follow through. I saw the boot screen, but did not check with balena scan or for serial connection in the dev image, even though it is obvious. I will get back to you after i tried it.

Due to some covid restrictions, I do not have easy access to the jetson, since i missed takinging one home. Therefore it might take a day or two.

Langhalsdino · March 15, 2021, 2:22pm

Thank you for your advise. I build development images, such that debugging is easier using the barys -d flag.

Regarding my debugging process, i tested for the following capabilities for each image i build:

Does the device post (serial or display logs)? If only one of these is present your kernel might be misconfigured.
Is the device discoverable in the local network via balena scan?
Does balena os configure succeed?
Can I deploy a balena app?

What is currently working and what does not?
I can build the jetson-nano image correctly and all capabilities (1-4) are present.
When i build the custom-nano image it posts, but its not discoverable, balena-os does not succeed and i can not deploy apps to it. Therefore i conclude, that i did something wrong while creating a new device.

While reading the devices post logs, i noticed that some of the partitions are incompatible. (More in the picture)

EXT4-fs (nmcblk0p13): could'n mount as ext3 duet to feature incompatibilities
EXT4-fs (nmcblk0p13): could'n mount as ext2 duet to feature incompatibilities
EXT4-fs (nmcblk0p13): mounted filesystem with ordered data mode. Opts: (null)
EXT4-fs (nmcblk0p15): mounted filesystem with ordered data mode. Opts: (null)
EXT4-fs (nmcblk0p13): re-mounted. Opts: (null)

Logs till balenaOS bootet

Next steps
I will go trough Building Your Own Image | balenaOS and create a new device custom-nano-the-second in order to check that i did not miss a setting.

@klutchell do you have any suggestions where if might look in order to find a smoking gun?

A few learnings, that are not directly related:

Always build with the -r flag if you are stuck!

./balena-yocto-scripts/build/barys -m custom-nano -r

Don’t flash the new JP release 32.5.x to your jetson nano, until the balenaOS image supports the relocated boot firmware. (Disclaimer: This broke one of my jetsons, but i have not done any in depth analysis. Therefore this might be due to a different reasen, its currently just a gut feeling).

Boot firmware for all Jetson Nano developer kits updated to relocate boot firmware to integrated QSPI-NOR. Nvidia release notes

Langhalsdino · March 15, 2021, 6:00pm

I have a gut feeling, that the following file might be the reason for the issue. Is the machine configuration custom-nano.conf correct?

#@TYPE: Machine
##@NAME: custom-nano
##@DESCRIPTION: Machine configuration for the custom Jetson Nano

MACHINEOVERRIDES = "jetson-nano:${MACHINE}"
require conf/machine/jetson-nano.conf

If i understand it correctly this should just build the jetson nano image?

Langhalsdino · March 28, 2021, 6:45pm

So after some debugging and try and error, I got the image working. I can ssh on the device and pull containers, but it does not show up in the balena cloud .

I find the following error in the logs:

-- Logs begin at Thu 2021-03-25 22:03:55 UTC, end at Thu 2021-03-25 22:05:07 UTC. --
Mar 25 22:04:07 948af4b resin-supervisor[4434]: resin_supervisor
Mar 25 22:04:07 948af4b resin-supervisor[4485]: active
Mar 25 22:04:11 948af4b resin-supervisor[4489]: Container config has not changed
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Supervisor v12.4.6 starting up...
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Setting host to discoverable
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [warn]    Invalid firewall mode: . Reverting to state: off
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    🔥 Applying firewall mode: off
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Starting logging infrastructure
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Performing database cleanup for container log timestamps
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Starting firewall
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Handling of local mode switch is completed
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    API Binder bound to: https://api.balena-cloud.com/v6/
Mar 25 22:04:14 948af4b resin-supervisor[4682]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or B>
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [success] 🔥 Firewall mode applied
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Starting api binder
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [event]   Event: Supervisor start {}
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Connectivity check enabled: true
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Starting periodic check for IP addresses
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Reporting initial state, supervisor version and API info
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Attempting to load any preloaded applications
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    VPN connection is not active.
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Waiting for connectivity...
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Starting API server
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Supervisor API successfully started on port 48484
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    Applying target state
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Ensuring device is provisioned
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [event]   Event: Device bootstrap {}
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [info]    New device detected. Provisioning...
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [debug]   Finished applying target state
Mar 25 22:04:14 948af4b resin-supervisor[4682]: [success] Device state apply success
Mar 25 22:04:15 948af4b resin-supervisor[4682]: [error]   LogBackend: server responded with status code: 401
Mar 25 22:04:15 948af4b resin-supervisor[4682]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Mar 25 22:04:45 948af4b resin-supervisor[4682]: [event]   Event: Device bootstrap {}
Mar 25 22:04:45 948af4b resin-supervisor[4682]: [info]    New device detected. Provisioning...
Mar 25 22:04:46 948af4b resin-supervisor[4682]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}

Since I move from one issue to another, I decided to just modify the existing jetson-nano machine and added the patches for the adjusted jetson-nano camera driver to it. This works fine with no errors, even though it is not as elegant as having a custom machine config.

Just in case someone is interested, i added the imx driver patches to the kernel build step by appending SRC_URI_append_jetson-nano.

dtischler · March 29, 2021, 2:06am

Interesting…let me ping some folks on that provisioning error @Langhalsdino

klutchell · March 29, 2021, 4:08pm

Hey @langhalsdino, great work on your custom image thus far!

I expect the reason your device does not show in the Dashboard is your custom device type is not supported by the API. This is a manual process for now when we officially add support for devices.

In order to spoof the API and appear as a normal Jetson Nano, you’ll need to make /mnt/boot/device-type.json look like that of a Jetson Nano, either by editing it manually or borrowing that file from an official Jetson Nano image. You’ll also need to edit the deviceType in /mnt/boot/config.json. You can get both of these files from an official Jetson Nano image and replace them on your SD card before provisioning the device.

We are looking at ways to improve the process above and remove the redundancy, but in the meantime those two files will change how your device is recognized by the API.

Langhalsdino · March 29, 2021, 6:45pm

@klutchell , thank you. I suspected something like this, but did not know where to look.

Since the current modified jetson-nano is working for me, i will not proceed on this, unless someone else from the community is interested and needs a guide.