Just want to share a finding with the community to hopefully save others’ headaches when standing up a new openbalena instance. I couldn’t figure out why a fresh install of openbalena wasn’t allowing me to push new images to the registry, and I noticed that the minio instance in the open-balena-s3 container was kicking up some low level file system errors along the lines of Resource requested is unreadable, please reduce your request rate during each push.
After a few weeks of debugging, and many calls with Microsoft Azure support (as the first assumption was that this was some kind of storage resource error), it turns out there is an issue with the way minio provisions new storage in the open-balena-s3 instance. Starting with RELEASE.2022-06-02T02-11-04Z, minio implements a zero-parity erasure coded backend for single-node single-drive deployments (see this link for details). Unfortunately, while I’m sure this is an improvement to minio, this change breaks a lot of other stuff, and prevents open-balena-registry from pushing new images.
It appears the balena team is tuned into this issue, as the minio version in open-balena-s3 is pegged to RELEASE.2022-10-24T18-35-07Z which is the last version of minio to support the legacy fs mode and necessary for open-balena-registry to function properly. But while the last version of minio to support legacy fs mode works fine for pre-existing s3 instances, which were provisioned back when legacy fs mode was standard, it provisions new instances using the newer, breaking, erasure-coded structure.
To fix this, you either need to pre-provision your storage volume to use legacy fs mode, by creating a file at .minio.sys/format.json in the volume that you are mounting to the /export folder in the open-balena-s3 container with the following contents:
{"version":"1","format":"fs","id":"put-some-uuid-looking-string-here","fs":{"version":"2"}}
or alternatively run the code shown in this issue which I’ve opened in the open-balena-s3 repo, which will update your format.json file and restart the minio server.