Building BalenaOS for Rockchip 3328 / NanoPi Neo3

Hi Michael, I suspect that it’s the boot partition that is too small now. The “Image” the error refers to is probably the kernel image, which by the name seems not to be compressed.
As a workaround try increasing the RESIN_BOOT_SIZE in your local.conf. It’s set in meta-balena/image_types_resin.bbclass at bfbc7cbb6a455a27a833d99bcb243c0a6e663c11 · balena-os/meta-balena · GitHub to 40M, let’s try to double it see what happens.

RESIN_BOOT_SIZE = "81920"

As before, if this works we will have to go back to it and fix it properly by analyzing the size of the files that are going into the boot partition.

Hi Alex,

As regards to a larger RESIN_BOOT_SIZE, no dice, still the same error, with and without setting the other Size and Maxsize parameters.

Copying files into the device: __populate_fs: Could not allocate block in ext2 filesystem while writing file "Image"
mkfs.ext4: Could not allocate block in ext2 filesystem while populating file system
Deleted: sha256:e25e84397823c824d1823092d9c5b1b32326572d299c32dd4ce7bb2bd130d6c1
Deleted: sha256:b39ac24be79db4c4188f6498a176d79d9e8d37f6c82a0d03e81b801265aa87ae
Deleted: sha256:3c1c6f550ee188271fd448b2ad4830902d60994b86ae25727ebe1db432560670
WARNING: exit code 1 from a shell command.

Further, I’ve broken it in a novel way. I spent some time reviewing the recent livestream that you guys recommended on building BalenaOS, and the most notable piece was that my recipes-core/images/resin-image.bbappend had previously contained almost nothing based on what I could get out of the docs, just:

IMAGE_FSTYPES_nanopi-neo3 = "resinos-img"

RESIN_IMAGE_BOOTLOADER = "u-boot"

To this I’ve added, based on what I could garner from other repositories and an attempt at removing these size and allocation errors:

IMAGE_ROOTFS_MAXSIZE = "50000000"

RESIN_BOOT_PARTITION_FILES_nanopi-neo3 = " \
    ${KERNEL_IMAGETYPE}${KERNEL_INITRAMFS}-${MACHINE}.bin:/${KERNEL_IMAGETYPE} \
    rk3328-nanopi-r2-rev00.dtb:/dtb/rk3328-nanopi-r2-rev00.dtb \
    "

(I recognize that this needs references to at least u-boot in here as well, but some of my previous u-boot issues resulted in this dtb not being built - I’m going to dive back into this and see if I can figure out why in the meantime).

At any rate, the good news is this was perhaps the source of the warning about resin-image-initramfs. I am still failing though, in do_rootfs:

Install resin-boot in the rootfs...
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/rootfs/r>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/Image' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/ro>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/dtb' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/root>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/dtb/rk3328-nanopi-r2-rev00.dtb' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-li>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/splash' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/r>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/splash/balena-logo.png' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resi>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/os-release' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0->
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/device-type.json' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-imag>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/system-connections' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-im>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/system-connections/resin-sample.ignore' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/system-connections/README.ignore' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky->
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/resin-image' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/image-version-info' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-im>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/config.json' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0>
'/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-image/1.0-r0/resin-boot/resinos.fingerprint' -> '/home/ubuntu/development/yocto/balena-nanopineo3/build/tmp/work/nanopi_neo3-poky-linux/resin-i>
NOTE: resin-boot: FAT overhead 1034 sectors, data 66002 sectors, largest file 65814 sectors, boot size 81920 sectors.
ERROR: resin-boot: Not enough space for atomic copy operations.

Any suggestions? Still seems as if this is size related somehow, but hoping that this reference to atomic copy operations is informative in hunting down where. Thanks in advance

Hi Michael,

It looks like you’re getting closer. The message about not having space for atomic copy operations is in regards to the method we use to ensure that files are updated atomically on the FAT boot partition. Because FAT doesn’t support journaling or atomic writes, we don’t write directly to existing files, but instead write to a temporary file, and rename it after the write is complete. This requires at least as much free space as the largest file in the partition, which is what this step is checking for.

It seems your boot partition is now large enough to store your boot files, but not large enough for atomic updates. That’s not unusual, as uncompressed kernels can be quite large. You might want to bump up RESIN_BOOT_SIZE to something like 131072 (128 MB) and try again.

You could also strip out unused kernel features to reduce the image size, though that’s much more complicated.

Hi @jakogut ,

Thanks for the response (and giving Alex a break from dealing with my issues, he deserves it).

Good to have some detail about what’s going awry here. I’d love to dig into the unused kernel features at some point, but unfortunately I get this same error with a higher RESIN_BOOT_SIZE as recommended, or if I increase it by 2 orders of magnitude (I’m assuming that while it’s nice to keep to proper powers of 2 in bytes, that 13107200 is just an overly high ceiling for testing purposes).

Is there a list of RESIN size parameters that I can experiment with on my end without bothering you guys each time? If I can figure out which of these is causing the error I’ll report back but I hate to have to post for each error, and I haven’t found anywhere that lists some of these resin-specific parameters in the docs.

Thanks as always,

~Michael

I’m not sure there’s a list anywhere, my process for figuring these things out is usually grepping (I actually use ack) for relevant strings to find the right files to read. For instance, in this case, I would grep for “Not enough space for atomic copy operations”, which would lead me to layers/meta-balena/meta-balena-common/classes/image-resin.bbclass. Reading that Yocto class should give you a better understanding of what’s going on, along with relevant variables.

Thanks Joseph - that’s a good tip, I’ll see what I can find out that way. A bunch of these methods of investigation (including the ones Alex suggested for interrogating bitbake variables and rebuilding the specific recipes) are growing rapidly more useful as I become accustomed to them. Thanks for explaining the how of it!

~Michael

Hi all,

I have (mostly) good news. I’ve finally been able to build the entire image start to finish without errors. That’s not without its caveats or hacks, which I know will need to be reviewed. Similarly, I’m not getting it to boot into a recognizable attachment to a Balena application yet, but I realize that could be any number of things. I’m going to need a debug serial cable and should have one by tomorrow. Until then though, a couple notes and questions:

  • I had a heck of a time getting some of the above variables to work. This may be due to a lack of my understanding how variables propagate in Yocto, but I want to detail them here so that there’s a chance of them being useful:
  1. RESIN_BOOT_SIZE = "131072" - I looked at this a bunch of ways, and ultimately started adding echo’s to try and figure out if it was being respected. Those echos started coming back empty when I set this value in my local.conf.sample, and ultimately I got it to work by setting RESIN_BOOT_SIZE = "131072" in meta-balena/meta-balena-common/classes/image-resin.bbclass. Am I setting this in the wrong place?

  2. IMAGE_ROOTFS_SIZE also gave me trouble. Setting it in local.conf never did the trick, until eventually I hunted it down to meta-balena-common/recipes-core/images/balena-image.bb and set it to IMAGE_ROOTFS_SIZE = "500000"- at which point the build succeeded.

In other news, I had to cheat substantially to get u-boot in place the way it ought to be. Given the advice related to the itb file above, I ended up commenting out substantial portions of my do_deploy in my bsp recipe. I knew it might cause me trouble down the line, but I figured I’d know more by then. The good news is I do know more, the bad news is it’s not quite enough. I had commented out deploy pieces that were copying dependencies that seemed destined to get reference in recipes-core/image, specifically:

if [ -n "${SPL_BINARY}" ]; then
    install -D -m 644 ${B}/${config}/${SPL_BINARY} ${DEPLOYDIR}/${SPL_IMAGE}-${type}-${PV}-${PR}
    rm -f ${DEPLOYDIR}/${SPL_BINARYNAME} ${DEPLOYDIR}/${SPL_SYMLINK}-${type}
    ln -sf ${SPL_IMAGE}-${type}-${PV}-${PR} ${DEPLOYDIR}/${SPL_BINARYNAME}-${type}
    ln -sf ${SPL_IMAGE}-${type}-${PV}-${PR} ${DEPLOYDIR}/${SPL_BINARYNAME}
    ln -sf ${SPL_IMAGE}-${type}-${PV}-${PR} ${DEPLOYDIR}/${SPL_SYMLINK}-${type}
    ln -sf ${SPL_IMAGE}-${type}-${PV}-${PR} ${DEPLOYDIR}/${SPL_SYMLINK}
fi
install -D -m 644 ${WORKDIR}/rkEnv.txt ${DEPLOYDIR}/
install -D -m 644 ${WORKDIR}/boot.scr ${DEPLOYDIR}/

For obvious reasons, I realized this wasn’t going to work. the SPL_BINARY and a few others seem to be common features in other resin-image.bbappend's I’ve looked at. Despite a lot of experimenting and staring, I still haven’t been able to get these files to appear properly while building meta-balena-nanopi-rockchip64. As a sanity check, I went back and built all of the original BSP - meta-nanopi-rockchip64, and it built start to finish without trouble, and of course, produced all of these files. I’m guilty of just copying them in for now, though I’d love to understand what’s going awry. In particular, all I can think of is that inherit resin-u-boot is making some different here, though I don’t see any obvious reason why that should be so. My next step would be to try building the directory structures and everything from scratch now that I know more about the different components I’m looking at, but if anyone has an idea of where to look as to why these files don’t get installed properly inside my build, but do in the BSP layer from which I got these recipes, I’d be really curious.

So hopefully more good news to follow. Until I can do some serial debugging on the image I produced this time I’m not sure exactly where this is going wrong, but I hope to be working on that tomorrow.

Hi Michael,

How are you coming along on this?

I had a heck of a time getting some of the above variables to work. This may be due to a lack of my understanding how variables propagate in Yocto, but I want to detail them here so that there’s a chance of them being useful:

The way Yocto scopes variables is by no means intuitive, and I’d encourage you to check out the documentation, which you can find here.

Also note that the documentation is utterly massive in scope, so I’ll leave you with this summary from the mailing list that’s a bit more approachable:

The first thing to realise is that there is variable scoping in BitBake - you
can set a variable in a recipe or bbappend, and it will only affect that
recipe. You can set a variable in a .bbclass (possibly one that you create)
and it will affect recipes that inherit that class. You can set a variable at
the configuration level and it will affect all recipes. (When I say “affect” I
mean the value you set will potentially be available in the scope specified).

Hi Joseph,

Thanks for the resources. I’ve done my best to dive and occasionally drown in the mega manual, but it only makes it apparent at what stage of this level of knowledge I’m at.

The good news is I made a substantial edit to the layer I’d built - there were a number of things wrong with it that needed to fixed given my evolving understanding of Yocto and BalenaOS. The bad news is that while I’d hoped to see some basic u-boot or BalenaOS errors over serial, what I in fact see is absolutely nothing. That is - I can monitor a working Yocto build just fine over serial (the BSP I’m building off of), but the build I’m producing is completely dead on arrival. If I had to guess, I imagine this may have to do with the resin-core/image section of the build that seems to be fairly bespoke per supported Balena device, but without errors it’s tricky to tell. I’ve done some comparison of the various supported SOC’s on this topic and it remains fairly opaque without being sure how they were produced.

What I’m wondering is kind of what you’d suggest as a reasonable next step. You’ve both been incredibly helpful , and I hope there’s some utility in having some of the troubleshooting process documented publicly for someone working on this in the future. What’s hard for me to tell right now is whether I’ve actually made it over the bigger hurdle in getting to this point, or if in fact what I’ve done so far is really the easy part, and the better part of the difficulty lies in the bespoke pieces of the image required to get it to boot smoothly into BalenaOS

Hi Michael,

…but the build I’m producing is completely dead on arrival. If I had to guess, I imagine this may have to do with the resin-core/image section of the build that seems to be fairly bespoke per supported Balena device

Yes, ARM single-board computers tend to each implement their own boot process, and the method for configuring, building, and deploying the bootloader is quite unique and non-standardized. On the bright side, u-boot does a pretty good job of documenting the process for supported boards, which should include yours.

I would recommend building and flashing u-boot to an SD card first, ignoring balenaOS until you get a working bootloader. For instructions on how to do this, first clone u-boot from the upstream git repo at: U-Boot / U-Boot · GitLab

After cloning the source, you should be able to view the documentation at doc/README.rockchip. Follow that documentation to build and flash your bootloader, and see if you can get serial output from your board. After that, you should have a better idea of how to proceed with creating a balenaOS image.

Thanks Joseph, it’s nice to have a specific place to dig and work. Assuming I can get a working bootloader (as the BSP I built clearly had that with u-boot), is there any detail on BaleanOS’s unique requirements for integrating this into the BalenaOS build, or is the expectation that this will be apparent once I understand the SOC-specific u-boot process?

is there any detail on BaleanOS’s unique requirements for integrating this into the BalenaOS build, or is the expectation that this will be apparent once I understand the SOC-specific u-boot process?

Generally, BSP/board specific customizations live in the layers/meta-balena-{platform} directory, and you shouldn’t need anything beyond a standard u-boot build. If it fails to boot, for example because of a missing filesystem driver, you should get an appropriate error message indicating such.

BalenaOS specific u-boot customizations, for example A/B fallback support, are contained in layers/meta-balena, and are shared between device types. You shouldn’t need to make any modifications to u-boot specifically to support balenaOS, those should be part of meta-balena already. Getting u-boot running on your board should be a good first step.

Hey.

Have you had any success getting u-boot running on your board, as suggested by the colleague?

Cheers.

Hi,

Thanks for the checking in - I’ve actually been hunting down a new dependency in dnsmasq but am about to return to this, should have an update soon.

~Michael

Hi Michael, I just wanted to see if there was an update?

Hi there,

Your timing is impeccable! I’ve been working on this more closely in the last couple days and this point haven’t come as far as I’d hoped. I’m shifting my focus back to trying to understand the working and flashable BSP that’s clearly building u-boot correctly to see if I can work backwards from there. Here’s what I’ve done and found so far:

Per the suggestion above, I went directly to u-boot to see what I could find. In the documenation for the rockchips (found here ) there’s actually only a short bit of instruction regarding this SoC:

Booting from an SD card on Pine64 Rock64 (RK3328)

For Rock64 rk3328 board the following three parts are required:
TPL, SPL, and the u-boot image tree blob.

  • Write TPL/SPL image at 64 sector

    => sudo dd if=idbloader.img of=/dev/mmcblk0 seek=64

  • Write u-boot image tree blob at 16384 sector

    => sudo dd if=u-boot.itb of=/dev/mmcblk0 seek=16384

So this may be of some use to me as I seek to figure out the proper sectors things to, though I have the working BSP to compare that against too. There’s greater detail it turns out in how to build this elsewhere in the u-boot repository ( here ). Unfortunately, I couldn’t replicate building these either on a completely fresh docker image or on the Ubuntu machine I’ve been using to build Yocto. It’s unclear to me how recently these have been updated, since certain directory instructions are incorrect (one example, RK3328TRUST.ini is not found in rkbin/tools, it’s found at rkbin/RKTRUST).

At any rate, I’m unable to replicate the steps laid out here. I can, for example, apparently compile the ATF properly, but it never produces a bl31.bin - though it finishes other bl31 files like bl31.elf. I can also compile u-boot successfully, with some added instructions from elsewhere like setting the path for bl31.elf, but I haven’t gotten as far as actually being to flash the resulting image. Hard for me to tell if this is wrong on my end or theirs, but if anyone’s feeling really bold I’d be curious to hear if these instructions work out of the box for others.

I’ll update here if I can work out the differences ultimately in my non-functional BalenaOS u-boot vs. the functional one in the BSP.

~Michael

Thanks for the update, Michael. Working with u-boot can be challenging, especially before you get any working output from your device. You’ve narrowed down the problem, however, and once you get the bootloader building and booting, you’re not far off from a working image. Unfortunately, the bootloader is one of the few pieces in adding support for a new device type that requires a lot of legwork.

Keep in mind that, as far as docs go, you generally only want to rely on the documentation from your u-boot source tree, as it’s subject to change, and typically updated to match the code. In other words, make sure you’re not using a vendor fork of u-boot with upstream docs, or using docs from the most recent revision with an older tag of u-boot.

I look forward to hearing how you make out on this.

Hi Joseph,

Thanks for both the encouragement and the caution regarding which u-boot fork. It’s good to hear that this may be the part that requires some of the most careful work. Hopefully more progress and an update by the end of the week with any luck,

~Michael

Hi Michael, you might (key word is might! ) find some snippets and tidbits in a bit of work I did to support a similar but not exact SoC, the Rockchip PX30, located here: lec-px3: Port changes by acostach · Pull Request #24 · balena-os/balena-nanopc-t4 · GitHub

I have never used or built for a 3328, so I can’t say with any certainty if any of my work there is helpful, but figured I’d pass it along. :slight_smile:

Hi David,

I appreciate it! I’ll take a look through - every little bit of explicable work someone’s done brings me a little closer to the whole picture I think, so at the least it can’t hurt.

Happy to report on a Friday that I was able to successfully build a version of u-boot that I can see running over serial. The tips above about branches and the like led me to at least do an exhaustive search for anything rk3328 related, which surfaced a 3rd set of instructions here: doc/board/rockchip/rockchip.rst · v2020.07 · U-Boot / U-Boot · GitLab

lo and behold some of the issues I had, like not building bl31.bin weren’t actually required despite what the other instructions suggested, as well as the bit about the miniloader. I’ll be revisiting this as part of my review of the BSP which does include instructions for the miniloader, as well as the use of the u-boot.itb that @alexgg noted to me earlier on is not used by BalenaOS. I hope to harmonize some of these ideas next week.

Thanks again,

~Michael

1 Like