hi,
I am currently attempting to push a first basic version of my application from the balena cloud to my jetson nano-sd. when reaching about 75% it seems to time out and restart. is it possible to increase the timeout?
i tried adding RESIN_SUPERVISOR_HANDOVER_TIMEOUT as device configuration, but no result
09.02.21 17:13:45 (+0100) Failed to download image 'registry2.balena-cloud.com/‘stuff’ due to ‘connect ECONNREFUSED /var/run/balena-engine.sock’
That config var only has any effect with the “handover” update strategy, which is not the default. It looks like balena-engine crashes on the device – what device is it? What OS version? Does it eventually succeed? Does rebooting help?
I am using a jetson nano with sd card. i tried rebooting and purging data, but now it remains somewhere between 0-45% with the same message.
the os is jetson-nano-2.67.3+rev2-dev-v12.3.0
depoying locally also works fine
Hi, could you paste the output from running journalctl --no-pager -u balenaon the hostOS?
root@bdded57:~# journalctl --no-pager -u balena
-- Logs begin at Wed 2021-02-10 15:00:18 UTC, end at Thu 2021-02-11 07:48:55 UTC. --
Feb 10 15:00:25 bdded57 balenad[4044]: time="2021-02-10T15:00:25.885575915Z" level=info msg="Starting up"
Feb 10 15:00:25 bdded57 balenad[4044]: time="2021-02-10T15:00:25.885685084Z" level=warning msg="Running experimental build"
Feb 10 15:00:25 bdded57 balenad[4044]: time="2021-02-10T15:00:25.887586011Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
Feb 10 15:00:25 bdded57 balenad[4044]: time="2021-02-10T15:00:25.904177309Z" level=info msg="libcontainerd: started new balena-engine-containerd process" pid=4086
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.094154982Z" level=info msg="starting containerd" revision= version=1.2.0+unknown
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.095049794Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.095704131Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.107762987Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="modprobe aufs failed: \"modprobe: FATAL: Module aufs not found in directory /lib/modules/4.9.140-l4t-r32.4\\n\": exit status 1: skip plugin" type=io.containerd.snapshotter.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.107865228Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.108701913Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.109459221Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.109549276Z" level=info msg="metadata content store policy set" policy=shared
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113042476Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113128051Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113232585Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113286336Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113335295Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113395661Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113443371Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113490195Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113536810Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113583634Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.113862442Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.114086458Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115536489Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115622376Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115741858Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115795557Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115843787Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115888945Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115932279Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.115984988Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.116031760Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.116074209Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.116117283Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.116993135Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system"
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117056158Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117117513Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117167514Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117212671Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117646587Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd-debug.sock
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117812841Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock.ttrpc
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117951489Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock
Feb 10 15:00:26 bdded57 balenad[4086]: time="2021-02-10T15:00:26.117994720Z" level=info msg="containerd successfully booted in 0.027910s"
Feb 10 15:00:28 bdded57 balenad[4044]: time="2021-02-10T15:00:28.952446000Z" level=error msg="refusing to remove empty link for layer 1cd83e2d7e606317194a5b26ae46a0017b8f8e5f7f750303de30f886e55d4081" storage-driver=overlay2
Feb 10 15:00:28 bdded57 balenad[4044]: time="2021-02-10T15:00:28.957437932Z" level=info msg="Pruned 2 unused graph driver layers"
Feb 10 15:00:29 bdded57 balenad[4044]: time="2021-02-10T15:00:29.142989520Z" level=warning msg="Your kernel does not support cgroup rt period"
Feb 10 15:00:29 bdded57 balenad[4044]: time="2021-02-10T15:00:29.143048896Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Feb 10 15:00:29 bdded57 balenad[4044]: time="2021-02-10T15:00:29.143073896Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Feb 10 15:00:29 bdded57 balenad[4044]: time="2021-02-10T15:00:29.143096970Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Feb 10 15:00:29 bdded57 balenad[4044]: time="2021-02-10T15:00:29.143454217Z" level=info msg="Loading containers: start."
Feb 10 15:00:31 bdded57 balenad[4044]: time="2021-02-10T15:00:31.840887779Z" level=info msg="Removing stale sandbox 865d1372e0cd83abbb2619405c3b3a7af8ddbe9d6557b96d7339149e8077371f (d8f5543895ea4ee215c42c9eb887ce9542477714ffcc798979a3189baeff5892)"
Feb 10 15:00:31 bdded57 balenad[4044]: time="2021-02-10T15:00:31.879101839Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dba3bde3f88fbbb38aef46ded9bc1de16579408107f72d1baeb388cc6995b4fb 86afbd5f65f20cd70501e9c2bdefe0c2ff25c2d15ac85bc456e4173db1153128], retrying...."
Feb 10 15:00:34 bdded57 balenad[4086]: time="2021-02-10T15:00:34.465196885Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/ffd911136ef793fe494aa7dc967466596474bd76125719495fb3f0cfd018dee7.sock debug=false pid=4341
Feb 10 15:00:34 bdded57 balenad[4086]: time="2021-02-10T15:00:34.983052731Z" level=info msg="shim reaped" id=d8f5543895ea4ee215c42c9eb887ce9542477714ffcc798979a3189baeff5892
Feb 10 15:00:34 bdded57 balenad[4044]: time="2021-02-10T15:00:34.997218872Z" level=error msg="stream copy error: reading from a closed fifo"
Feb 10 15:00:34 bdded57 balenad[4044]: time="2021-02-10T15:00:34.998007848Z" level=error msg="stream copy error: reading from a closed fifo"
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.408398105Z" level=error msg="d8f5543895ea4ee215c42c9eb887ce9542477714ffcc798979a3189baeff5892 cleanup: failed to delete container from containerd: no such container"
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.408515608Z" level=error msg="Failed to start container d8f5543895ea4ee215c42c9eb887ce9542477714ffcc798979a3189baeff5892: OCI runtime create failed: container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/var/log/supervisor-log\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/515eca391047203e8e6ba2d11c49e89e429c90f218f6ccf8915b21ace4dc55c3/merged\\\\\\\" at \\\\\\\"/var/log\\\\\\\" caused \\\\\\\"stat /var/log/supervisor-log: no such file or directory\\\\\\\"\\\"\": unknown"
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.408669778Z" level=info msg="Loading containers: done."
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.576839477Z" level=info msg="Docker daemon" commit=074a481789174b4b6fd2d706086e8ffceb72e924 graphdriver(s)=overlay2 version=19.03.13-dev
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.576993699Z" level=info msg="Daemon has completed initialization"
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.614871450Z" level=info msg="API listen on [::]:2375"
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.615127966Z" level=info msg="API listen on /var/run/balena.sock"
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.615205416Z" level=info msg="API listen on /run/balena-engine.sock"
Feb 10 15:00:36 bdded57 balenad[4044]: time="2021-02-10T15:00:36.615309324Z" level=info msg="API listen on /var/run/balena-engine.sock"
Feb 11 07:48:27 bdded57 balenad[4086]: time="2021-02-11T07:48:27.558336116Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/ffd911136ef793fe494aa7dc967466596474bd76125719495fb3f0cfd018dee7.sock debug=false pid=4534
Feb 11 07:48:27 bdded57 balenad[4086]: time="2021-02-11T07:48:27.608327151Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/e3ea837029ed6f73286e4375d4eae74179efae11f071c9b19eaa7887fac3f764.sock debug=false pid=4540
Feb 11 07:48:29 bdded57 balenad[4086]: time="2021-02-11T07:48:29.375734548Z" level=info msg="shim reaped" id=6845e344449966efe7c13c689f02499b1fe5033cc2eff261913798204b5766b8
Feb 11 07:48:29 bdded57 balenad[4044]: time="2021-02-11T07:48:29.384372548Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 11 07:48:34 bdded57 d8f5543895ea[4044]: [info] Supervisor v12.3.0 starting up...
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Setting host to discoverable
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [warn] Invalid firewall mode: . Reverting to state: off
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] 🔥 Applying firewall mode: off
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Starting logging infrastructure
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Performing database cleanup for container log timestamps
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Starting firewall
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Previous engine snapshot was not stored. Skipping cleanup.
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Handling of local mode switch is completed
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] API Binder bound to: https://api.balena-cloud.com/v6/
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [success] 🔥 Firewall mode applied
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Starting api binder
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [event] Event: Supervisor start {}
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Connectivity check enabled: true
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Starting periodic check for IP addresses
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Reporting initial state, supervisor version and API info
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Skipping preloading
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] VPN connection is not active.
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Waiting for connectivity...
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Starting API server
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Supervisor API successfully started on port 48484
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [info] Applying target state
Feb 11 07:48:36 bdded57 d8f5543895ea[4044]: [debug] Ensuring device is provisioned
Feb 11 07:48:37 bdded57 d8f5543895ea[4044]: [debug] Starting current state report
Feb 11 07:48:37 bdded57 d8f5543895ea[4044]: [debug] Starting target state poll
Feb 11 07:48:37 bdded57 d8f5543895ea[4044]: [debug] Spawning journald with: chroot /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
Feb 11 07:48:37 bdded57 d8f5543895ea[4044]: [error] LogBackend: server responded with status code: 404
Feb 11 07:48:37 bdded57 d8f5543895ea[4044]: [event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437@sha256:5093b20662286f4cfd125a3cd3690d8b8fbc4995f9587d1d7c6e4623cb872d94","appId":1774169,"serviceId":835637,"serviceName":"main","imageId":3264933,"releaseId":1691920,"dependent":0,"dockerImageId":null}}
Feb 11 07:48:39 bdded57 balenad[4044]: time="2021-02-11T07:48:39.261881537Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Feb 11 07:48:39 bdded57 balenad[4044]: time="2021-02-11T07:48:39.262333579Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Feb 11 07:48:39 bdded57 d8f5543895ea[4044]: [event] Event: Image download error {"error":{"message":"(HTTP code 404) unexpected - pull access denied for registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ","stack":"Error: (HTTP code 404) unexpected - pull access denied for registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437, repository does not exist or may require 'docker login': denied: requested access to the resource is denied \n at /usr/src/app/dist/app.js:10:2302614\n at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:10:2302501)\n at IncomingMessage.emit (events.js:322:22)\n at endReadableNT (_stream_readable.js:1187:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"image":{"name":"registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437@sha256:5093b20662286f4cfd125a3cd3690d8b8fbc4995f9587d1d7c6e4623cb872d94","appId":1774169,"serviceId":835637,"serviceName":"main","imageId":3264933,"releaseId":1691920,"dependent":0,"dockerImageId":null}}
Feb 11 07:48:39 bdded57 d8f5543895ea[4044]: [error] Scheduling another update attempt in 1000ms due to failure: Error: Failed to apply state transition steps. Steps:["fetch"]
Feb 11 07:48:39 bdded57 d8f5543895ea[4044]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Feb 11 07:48:39 bdded57 d8f5543895ea[4044]: [error] Device state apply error Error: Failed to apply state transition steps. Steps:["fetch"]
Feb 11 07:48:39 bdded57 d8f5543895ea[4044]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Feb 11 07:48:41 bdded57 d8f5543895ea[4044]: [info] Applying target state
Feb 11 07:48:41 bdded57 d8f5543895ea[4044]: [event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437@sha256:5093b20662286f4cfd125a3cd3690d8b8fbc4995f9587d1d7c6e4623cb872d94","appId":1774169,"serviceId":835637,"serviceName":"main","imageId":3264933,"releaseId":1691920,"dependent":0,"dockerImageId":null}}
Feb 11 07:48:46 bdded57 balenad[4044]: time="2021-02-11T07:48:46.274236755Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Feb 11 07:48:46 bdded57 balenad[4044]: time="2021-02-11T07:48:46.274574627Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Feb 11 07:48:46 bdded57 d8f5543895ea[4044]: [event] Event: Image download error {"error":{"message":"(HTTP code 404) unexpected - pull access denied for registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ","stack":"Error: (HTTP code 404) unexpected - pull access denied for registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437, repository does not exist or may require 'docker login': denied: requested access to the resource is denied \n at /usr/src/app/dist/app.js:10:2302614\n at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:10:2302501)\n at IncomingMessage.emit (events.js:322:22)\n at endReadableNT (_stream_readable.js:1187:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"image":{"name":"registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437@sha256:5093b20662286f4cfd125a3cd3690d8b8fbc4995f9587d1d7c6e4623cb872d94","appId":1774169,"serviceId":835637,"serviceName":"main","imageId":3264933,"releaseId":1691920,"dependent":0,"dockerImageId":null}}
Feb 11 07:48:46 bdded57 d8f5543895ea[4044]: [error] LogBackend: server responded with status code: 404
Feb 11 07:48:46 bdded57 d8f5543895ea[4044]: [error] Scheduling another update attempt in 2000ms due to failure: Error: Failed to apply state transition steps. Steps:["fetch"]
Feb 11 07:48:46 bdded57 d8f5543895ea[4044]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Feb 11 07:48:46 bdded57 d8f5543895ea[4044]: [error] Device state apply error Error: Failed to apply state transition steps. Steps:["fetch"]
Feb 11 07:48:46 bdded57 d8f5543895ea[4044]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Feb 11 07:48:48 bdded57 d8f5543895ea[4044]: [info] Applying target state
Feb 11 07:48:49 bdded57 d8f5543895ea[4044]: [event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437@sha256:5093b20662286f4cfd125a3cd3690d8b8fbc4995f9587d1d7c6e4623cb872d94","appId":1774169,"serviceId":835637,"serviceName":"main","imageId":3264933,"releaseId":1691920,"dependent":0,"dockerImageId":null}}
Feb 11 07:48:49 bdded57 d8f5543895ea[4044]: [error] LogBackend: server responded with status code: 404
Feb 11 07:48:50 bdded57 balenad[4044]: time="2021-02-11T07:48:50.500466638Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Feb 11 07:48:50 bdded57 balenad[4044]: time="2021-02-11T07:48:50.500896596Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Feb 11 07:48:50 bdded57 d8f5543895ea[4044]: [event] Event: Image download error {"error":{"message":"(HTTP code 404) unexpected - pull access denied for registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ","stack":"Error: (HTTP code 404) unexpected - pull access denied for registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437, repository does not exist or may require 'docker login': denied: requested access to the resource is denied \n at /usr/src/app/dist/app.js:10:2302614\n at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:10:2302501)\n at IncomingMessage.emit (events.js:322:22)\n at endReadableNT (_stream_readable.js:1187:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"image":{"name":"registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437@sha256:5093b20662286f4cfd125a3cd3690d8b8fbc4995f9587d1d7c6e4623cb872d94","appId":1774169,"serviceId":835637,"serviceName":"main","imageId":3264933,"releaseId":1691920,"dependent":0,"dockerImageId":null}}
Feb 11 07:48:51 bdded57 d8f5543895ea[4044]: [error] Scheduling another update attempt in 4000ms due to failure: Error: Failed to apply state transition steps. Steps:["fetch"]
Feb 11 07:48:51 bdded57 d8f5543895ea[4044]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Feb 11 07:48:51 bdded57 d8f5543895ea[4044]: [error] Device state apply error Error: Failed to apply state transition steps. Steps:["fetch"]
Feb 11 07:48:51 bdded57 d8f5543895ea[4044]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Feb 11 07:48:54 bdded57 d8f5543895ea[4044]: [error] LogBackend: server responded with status code: 404
Feb 11 07:48:55 bdded57 d8f5543895ea[4044]: [info] Applying target state
Feb 11 07:48:55 bdded57 d8f5543895ea[4044]: [event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/03e5f274d5d6e177f40554788db09437@sha256:5093b20662286f4cfd125a3cd3690d8b8fbc4995f9587d1d7c6e4623cb872d94","appId":1774169,"serviceId":835637,"serviceName":"main","imageId":3264933,"releaseId":1691920,"dependent":0,"dockerImageId":null}}I just flashed a new os on a different sd-card with a preloaded image, and it seems to work now.
Hi Jasper, thanks for those logs. I will pass them over to the Engine maintainer to have a look at. In the meantime however, can you tell me how large this container that you are building is? I am curious to know it’s built size.
Additionally, how is this device connected? Are you using WiFi or Ethernet? Thanks!
the image size is around 1.8 Gb and the device was connected via ethernet
Hello can you enable support access for this device and share with me in a PM the device UUID so I can take a quick look. By the logs it seems that the device can’t authenticate with the registry.
Ping us when you do.
as i mentioned, i re flashed the module on a different sd, and it works fine now. the jetson devkits seem to be acting up sometimes in different ways, but switching to production modules have shown no issues so far.
Good to hear that it works fine now. It’s hard to tell but judging from the logs, a connection hiccup might have caused this. Let us know if you experience this issue again and we could help out debugging it.
Cheers…