BalenaOS Jetson Xavier NX EMMC 16GB - No space left on device error

Hi Guys,

We are using BalenaOS Jetson v4.0.9+rev1 with Jetson Xavier NX EMMC 16GB. Our image is a custom one that was built with balena-jetson. On two of our other Jetson’s we’ve successfully flashed the image and it worked. We recently flashed the image onto a similar model jetson and we haven’t been able to get our images deployed. There’s something weird also about how it gets formatted it’s not the same as the other Jetsons:

Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        7.4G     0  7.4G   0% /dev
tmpfs                           7.8G  9.1M  7.8G   1% /run
/dev/disk/by-state/resin-rootA  677M  407M  220M  65% /mnt/sysroot/active
/dev/disk/by-state/resin-state   18M  407K   16M   3% /mnt/state
overlay                         677M  407M  220M  65% /
/dev/mmcblk0p13                  13G  5.4G  6.4G  46% /mnt/data
tmpfs                           7.8G     0  7.8G   0% /dev/shm
tmpfs                           4.0M     0  4.0M   0% /sys/fs/cgroup
tmpfs                           7.8G  4.0K  7.8G   1% /tmp
/dev/mmcblk0p9                  120M   76K  120M   1% /mnt/boot
tmpfs                           7.8G   76K  7.8G   1% /var/volatile
/dev/mmcblk0p11                 677M   24K  627M   1% /mnt/sysroot/inactive

The other Jetsons that work have this configuration:

Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        3.5G     0  3.5G   0% /dev
tmpfs                           3.8G  8.6M  3.8G   1% /run
/dev/disk/by-state/resin-rootA  677M  408M  219M  66% /mnt/sysroot/active
/dev/disk/by-state/resin-state   18M  409K   16M   3% /mnt/state
overlay                         677M  408M  219M  66% /
/dev/mmcblk0p13                  13G  6.6G  5.3G  56% /mnt/data
tmpfs                           3.8G     0  3.8G   0% /dev/shm
tmpfs                           4.0M     0  4.0M   0% /sys/fs/cgroup
tmpfs                           3.8G     0  3.8G   0% /tmp
/dev/mmcblk0p9                  120M   72K  120M   1% /mnt/boot
tmpfs                           3.8G   68K  3.8G   1% /var/volatile

This is the error I’m getting in the supervisor logs:

 Scheduling another update attempt in 128000ms due to failure:  Error: Failed to apply state transition steps. (HTTP code 500) server error - failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 8043: write /sys/fs/cgroup/cpuset/system.slice/docker-81747ff07d5b4ebe70821d05e5962ae676fe9401386cb229d243b149f11658b7.scope/cgroup.procs: no space left on device: unknown  Steps:["start","start","start"]
[error]         at fn (/usr/src/app/dist/app.js:10:9736)
[error]       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[error]   Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 8043: write /sys/fs/cgroup/cpuset/system.slice/docker-81747ff07d5b4ebe70821d05e5962ae676fe9401386cb229d243b149f11658b7.scope/cgroup.procs: no space left on device: unknown  Steps:["start","start","start"]

We’ve used the latest version of jetson-flash to flash and it didn’t help. We suspect it’s definitely something to do with partitions. Any help with this will be much appreciated.