A fresh downloaded image of Resin OS 2.14.3+rev1 on the QEMU (64bit) architecture is failing to download our container. An older OS (such as v2.13.6) continues to work fine. It infinitely loops with the following error:
21.08.18 17:25:46 (+1000) Downloading image 'registry2.resin.io/v2/589b1a2881c03d18feb4d466dba14bc8@sha256:a0676f63ca0a7224befd87d5cb6643b517b546705f3f05e66151e48dd0d0d70e'
21.08.18 17:31:10 (+1000) Failed to download image 'registry2.resin.io/v2/882eaff7631f943eb125e80d91abea1d@sha256:b225dd37708baf14700773ee599e260784836e739d52d45af105df3e7c4f567e' due to 'failed to register layer: Error processing tar file(exit status 1): write /usr/lib/openssh/ssh-keysign: cannot allocate memory'
21.08.18 17:35:09 (+1000) Failed to download image 'registry2.resin.io/v2/589b1a2881c03d18feb4d466dba14bc8@sha256:a0676f63ca0a7224befd87d5cb6643b517b546705f3f05e66151e48dd0d0d70e' due to 'failed to register layer: Error processing tar file(exit status 1): write /usr/lib/postgresql/9.4/bin/postgres: cannot allocatememory
An example device is available 7f0fee0c43e2e7660976435880e2eea5 with support access granted. This device has been provisioned with 700mb memory.
We also tested at 2GB which failed with a similar message of:
Failed to download image 'registry2.resin.io/v2/3a10b846c241512b28fc2d08ecbc1e4d@sha256:c8396cd326f154987f6eb3494b641f6756a164b6815ba6c4eae94e5bdbf1cc92' due to 'failed to register layer: Error processing tar file(exit status 1):'
Hey @SplitIce I tried to replicate this, our QEMU device downloaded the application right away.
Do you have a device that is currently experiencing the issue? The device that you mentioned doesn’t seem to exist anymore.
Also, for the 2GB layer one: how big is your image? The QEMU image by default has ~1GB space for the docker storage. The message seems as if it was running out of space. If it’s running out of space, you can take QEMU image that you use (when it’s not running) and add more space, e.g. adding ~5GB storage:
where you replace the resin-appname-qemux86-64-2.14.3+rev1-v7.16.6.img with your image’s name. But this is just the hunch, and only relevant to the second part of the issue you mentioned.
The container is around 800mb and it’s complaining about being out of memory not ram not disk (not that it actually seems to be using all that much ram when checked after the errors). The disk on one test device was 32GB.
@brdrcol (colleague) has also tested some VM’s with stupidly enlarged specifications with no luck.
The device (7f0fee0c43e2e7660976435880e2eea5) also definitely still exists, I just confirmed as much in the dashboard and is running.
Previously I converted the QEMU disk to VirtualBox vdi with qemu-img, increased the disk size to 8GB and memory to 6000MB and let it download/fail for 7 hours. I can’t run that VM at the moment.
I have started a new device (a64a7fb0f4d50f882bd2d44d808a94f6) with plain QEMU on Linux and granted support access. Commands were:
`24.08.18 00:19:08 (+1000) Failed to download image [...] due to 'failed to register layer: Error processing tar file(exit status 1): '`
`24.08.18 00:24:52 (+1000) Failed to download image [...] due to 'failed to register layer: Error processing tar file(signal: killed): '`
@imrehg no we havent. The process however is the same as converting a QEMU image. We also use KVM (QEMU compatible cpu) fyi so the QEMU distribution is a natural fit.
Hey @SplitIce, I think a new resinOS version is coming out soon with the fix included for this issue, fixed in balena over the weekend. Will keep you posted!
Are you still seeing this issue on a recent release of balenaOS? This issue should have been fixed months ago, so if you are still seeing it, please let us know!