OpenBalena: Preload fails

OpenBalena version: v3.2(tried with 3.3 as well)

root@yokoso-tv:~/open-balena# git branch
  master
* v3.2.0-branch
  v3.3.0-branch
root@yokoso-tv:~/open-balena# docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                                             NAMES
292b0a2c8c7a        openbalena_haproxy                    "/docker-entrypoint.…"   43 minutes ago      Up 43 minutes       0.0.0.0:80->80/tcp, 222/tcp, 3128/tcp, 5432/tcp, 0.0.0.0:443->443/tcp, 6379/tcp   openbalena_haproxy_1
afc0e268126b        balena/open-balena-vpn:v9.17.4        "/usr/bin/entry.sh"      43 minutes ago      Up 43 minutes       80/tcp, 443/tcp, 3128/tcp                                                         openbalena_vpn_1
1ea57c3fda30        balena/open-balena-api:v0.119.5       "/usr/bin/entry.sh"      43 minutes ago      Up 43 minutes       80/tcp                                                                            openbalena_api_1
a410eac2316e        balena/open-balena-registry:v2.14.4   "/usr/bin/entry.sh"      43 minutes ago      Up 43 minutes       80/tcp                                                                            openbalena_registry_1
833a68a57fa6        redis:alpine                          "docker-entrypoint.s…"   43 minutes ago      Up 43 minutes       6379/tcp                                                                          openbalena_redis_1
d582b3ed373a        openbalena_cert-provider              "/entry.sh /usr/src/…"   43 minutes ago      Up 43 minutes       80/tcp                                                                            openbalena_cert-provider_1
ec9407fa5e75        balena/open-balena-db:v4.1.0          "/balena-entrypoint.…"   43 minutes ago      Up 43 minutes       5432/tcp                                                                          openbalena_db_1
0fc966e74b6b        balena/open-balena-s3:v2.9.9          "/usr/bin/entry.sh"      43 minutes ago      Up 43 minutes       80/tcp                                                                            openbalena_s3_1

Tried to preload an application to the RPi 0 image(It worked well in last month, but suddenly stopped now)

ubuntu@ubuntu-desktop:/$ balena --version
12.44.14
ubuntu@ubuntu-desktop:/$ balena preload raspberrypi_2.67.3_rev1_dev.img --add-certificate ca.crt --app yokosoApp --commit current --splash-image yokosoTVBalena/yokoso/img/splash.jpg --debug
[debug] new argv=[/home/ubuntu/balena-cli/balena,/snapshot/versioned-source/bin/balena,preload,raspberrypi_2.67.3_rev1_dev.img,--add-certificate,ca.crt,--app,yokosoApp,--commit,current,--splash-image,yokosoTVBalena/yokoso/img/splash.jpg] length=12
Building Docker preloader image. [===                     ] 12%
Step 1/7 : FROM docker:20.10.6-dind
Building Docker preloader image. [======                  ] 25%
Step 2/7 : RUN apk update && apk add --no-cache py3-pip parted btrfs-progs util-linux sfdisk file coreutils sgdisk
 ---> Using cache
Building Docker preloader image. [=========               ] 37%
Step 3/7 : COPY ./requirements.txt /tmp/
 ---> Using cache
Building Docker preloader image. [============            ] 50%
Step 4/7 : RUN pip3 install -r /tmp/requirements.txt
 ---> Using cache
Building Docker preloader image. [===============         ] 62%
Step 5/7 : COPY ./src /usr/src/app
 ---> Using cache
Building Docker preloader image. [==================      ] 75%
Step 6/7 : WORKDIR /usr/src/app
 ---> Using cache
Building Docker preloader image. [=====================   ] 87%
Step 7/7 : CMD ["python3", "/usr/src/app/preload.py"]
 ---> Using cache
 ---> 0328b01cf811
Successfully built 0328b01cf811
Building Docker preloader image. [========================] 100%
| Checking that the image is a writable file
| Finding a free tcp port and getting balena settings
| Checking if the image is an edison zip archive
/ Creating preloader container
- Starting preloader container
\ Fetching application yokosoapp
/ Reading image informationWaiting for Docker to start...
| Reading image informationDocker started
| Reading image information
\ Fetching application 14
| Estimating required additional space
/ Cleaning up temporary files
manifest.manifest.layers is not iterable

TypeError: manifest.manifest.layers is not iterable
    at Preloader._getLayersSizes (/snapshot/versioned-source/node_modules/balena-preload/build/preload.js:489:51)
    at Preloader._getApplicationSize (/snapshot/versioned-source/node_modules/balena-preload/build/preload.js:507:40)
    at runNextTicks (internal/process/task_queues.js:62:5)
    at processImmediate (internal/timers.js:429:9)
    at process.topLevelDomainCallback (domain.js:137:15)
    at async Preloader._getSize (/snapshot/versioned-source/node_modules/balena-preload/build/preload.js:525:16)
    at async Preloader._getRequiredAdditionalSpace (/snapshot/versioned-source/node_modules/balena-preload/build/preload.js:531:34)
From previous event:
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
    at Preloader.preload (/snapshot/versioned-source/node_modules/balena-preload/build/preload.js:780:14)
    at PreloadCmd.prepareAndPreload (/snapshot/versioned-source/build/commands/preload.js:305:25)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async PreloadCmd.run (/snapshot/versioned-source/build/commands/preload.js:118:13)
    at async PreloadCmd._run (/snapshot/versioned-source/node_modules/@oclif/command/lib/command.js:43:20)
    at async Config.runCommand (/snapshot/versioned-source/node_modules/@oclif/config/lib/config.js:175:24)
    at async CustomMain.run (/snapshot/versioned-source/node_modules/@oclif/command/lib/main.js:27:9)
    at async CustomMain._run (/snapshot/versioned-source/node_modules/@oclif/command/lib/command.js:43:20)
    at async Promise.all (index 1)
    at async oclifRun (/snapshot/versioned-source/build/app.js:75:5)
    at async Object.run (/snapshot/versioned-source/build/app.js:88:9)

For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting

Tried with balena cli v12.38.5 and 12.44.0, but all failed.

Seems a bug on the openBalena server?

I had to add a new CNAME record(registry2.yokosotv.com) because it was saying that registry2.yokosotv.com was not found…
Not sure why registry2 is needed?

Any update here?

Deleted the local application images and pulled from the openBalena registry during the preload process. And it works! But not sure why! :slight_smile:

Good to hear that it works. Thank for updating the issue with a solution.

Well, it is not working again…
Deleted ALL local containers and tried this morning, but failed…

A bit confusing… :confused:

Could you please share more logs?

Same with the original message above.
Already submitted an issue here - manifest.manifest.layers is not iterable · Issue #243 · balena-io-modules/balena-preload · GitHub

Cheers,
Shane.

Thank you Shane for submitting the issue. Let us know if you discover anything new. Let’s stay connected

Could check out: Balena Preload points to wrong registry? - #35 by dfunckt

Thanks for your comment. I have already added that CNAME record to my server so this doesn’t occur for me. :slight_smile:

Any update here? @mpous

Hello,

This issue seems to have been fixed and updating to balenCLI v 12.44.16 or later has been reported to fix the issue.

Seems you are the author of the issue and the last comment :slight_smile: Can you confirm that this works for you please?

Yeah, it works with the latest CLI version as mentioned in the gitlab issue.

Thanks for confirming :slight_smile: