Hello Balenafam,
I can see on the right there are various similar titled posts, yet I thought this was different, or focused enough to warrant its own thread.
I have 3 raspberry pi devices, all in one fleet.
1 of which is stuck in deployment loop of some sort,
13.10.22 10:24:03 (+0100) Supervisor starting
13.10.22 10:24:05 (+0100) Downloading image 'registry2.balena-cloud.com/v2/f9d910efb66f680563a501ebebd51f41@sha256:250fb003d1e35bef1cd2cb798f061686817cdd3d020b6442ef799abeea756ceb'
13.10.22 10:28:38 (+0100) Supervisor starting
13.10.22 10:28:41 (+0100) Downloading image 'registry2.balena-cloud.com/v2/f9d910efb66f680563a501ebebd51f41@sha256:250fb003d1e35bef1cd2cb798f061686817cdd3d020b6442ef799abeea756ceb'
So i tried to pull manually, then I got
root@camf4a9490:~# balena pull registry2.balena-cloud.com/v2/f9d910efb66f680563a501ebebd51f41@sha256:250fb003d1e35bef1cd2cb798f061686817cdd3d020b6442ef799abeea756ceb
registry2.balena-cloud.com/v2/f9d910efb66f680563a501ebebd51f41@sha256:250fb003d1e35bef1cd2cb798f061686817cdd3d020b6442ef799abeea756ceb: Pulling from v2/f9d910efb66f680563a501ebebd51f41
error pulling image configuration: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
root@camf4a9490:~#
I tried a curl to see whats happening,
root@camf4a9490:~# curl -v https://registry2.balena-cloud.com
* STATE: INIT => CONNECT handle 0x5598028320; line 1780 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => RESOLVING handle 0x5598028320; line 1826 (connection #0)
* family0 == v6, family1 == v4
* Trying 2600:1f18:6600:7f00:d5c:76dc:d421:5665:443...
* STATE: RESOLVING => CONNECTING handle 0x5598028320; line 1908 (connection #0)
* Connected to registry2.balena-cloud.com (2600:1f18:6600:7f00:d5c:76dc:d421:5665) port 443 (#0)
* STATE: CONNECTING => PROTOCONNECT handle 0x5598028320; line 1971 (connection #0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* Didn't find Session ID in cache for host HTTPS://registry2.balena-cloud.com:443
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* STATE: PROTOCONNECT => PROTOCONNECTING handle 0x5598028320; line 1991 (connection #0)
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* Didn't find Session ID in cache for host HTTPS://registry2.balena-cloud.com:443
* Added Session ID to cache for HTTPS://registry2.balena-cloud.com:443 [server]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=balena.io
* start date: Jul 7 00:00:00 2022 GMT
* expire date: Aug 5 23:59:59 2023 GMT
* subjectAltName: host "registry2.balena-cloud.com" matched cert's "*.balena-cloud.com"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* STATE: PROTOCONNECTING => DO handle 0x5598028320; line 2010 (connection #0)
> GET / HTTP/1.1
> Host: registry2.balena-cloud.com
> User-Agent: curl/7.78.0
> Accept: */*
>
* STATE: DO => DID handle 0x5598028320; line 2068 (connection #0)
* STATE: DID => PERFORMING handle 0x5598028320; line 2187 (connection #0)
* Mark bundle as not supporting multiuse
* HTTP 1.1 or later with persistent connection
< HTTP/1.1 200 OK
< Date: Thu, 13 Oct 2022 09:34:25 GMT
< Content-Length: 0
< Connection: keep-alive
< Cache-Control: no-cache
<
* STATE: PERFORMING => DONE handle 0x5598028320; line 2386 (connection #0)
* multi_done
* Connection #0 to host registry2.balena-cloud.com left intact
* Expire cleared (transfer 0x5598028320)
root@camf4a9490:~#
crazy thing is, on this device, i have a usb ethernet adapter, so to use the faster ethernet, i did a nmcli radio wifi off to use the usb ethernet only.
that seemed to strangely, slow things down, so i did a nmcli radio wifi on to turn wifi back on again, and removed the usb ethernet to be able to troubleshoot.
that is the only viable difference between this one device and the other 2 in the fleet.
could that have changed something somewhere?
the other 2 devices are working as expected.
any help would be great.
kindly
puc.
updated:
adding stuff that might help
root@camf4a9490:~# systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.
root@camf4a9490:~#
but balena-supervisor seems to be looping too,
find: /mnt/root/tmp/balena-supervisor/services: No such file or directory
[info] Supervisor v14.2.7 starting up...
[info] Setting host to discoverable
[debug] Starting systemd unit: avahi-daemon.service
[debug] Starting systemd unit: avahi-daemon.socket
[debug] Starting logging infrastructure
[info] Starting firewall
[warn] Invalid firewall mode: . Reverting to state: off
[info] Applying firewall mode: off
find: /mnt/root/tmp/balena-supervisor/services: No such file or directory
[info] Supervisor v14.2.7 starting up...
[info] Setting host to discoverable
[debug] Starting systemd unit: avahi-daemon.service
[debug] Starting systemd unit: avahi-daemon.socket
[debug] Starting logging infrastructure
[info] Starting firewall
[warn] Invalid firewall mode: . Reverting to state: off
[info] Applying firewall mode: off
[success] Firewall mode applied
[debug] Starting api binder
[debug] Performing database cleanup for container log timestamps
[info] Previous engine snapshot was not stored. Skipping cleanup.
[debug] Handling of local mode switch is completed
(node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[info] API Binder bound to: https://api.balena-cloud.com/v6/
[event] Event: Supervisor start {}
[debug] Connectivity check enabled: true
[debug] Starting periodic check for IP addresses
[info] Reporting initial state, supervisor version and API info
[debug] Skipping preloading
[info] Starting API server
[info] Supervisor API successfully started on port 48484
[debug] VPN status path exists.
[info] Applying target state
[debug] Ensuring device is provisioned
[info] VPN connection is active.
[info] Waiting for connectivity...
[debug] Starting current state report
[debug] Starting target state poll
[debug] Spawning journald with: chroot /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[info] Reported current state to the cloud
[event] Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/f9d910efb66f680563a501ebebd51f41@sha256:250fb003d1e35bef1cd2cb798f061686817cdd3d020b6442ef799abeea756ceb","appId":1976283,"appUuid":"6ef336f722e84810a4f8c3efe935fa17","serviceId":1794327,"serviceName":"main","imageId":5561335,"releaseId":2330581,"commit":"e7ddb2b6102b8f90ec29c875a854b53c","dependent":0}}
[info] Internet Connectivity: OK
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
