Slow updates and install on fresh BalenaOS devices.

Hello everyone :slight_smile:,

I have strange behaviors trying to update or install my Fleet on my Raspberry PI 4 rev1.5 using balenaCloud.

For example i have:

  • One device where BalenaOS 2.95.8 was installed before the last 2 weeks.
  • One device with BalenaOS 2.95.8 installed today
  • One device with BalenaOS 2.98.33 installed today

The first devices installed my newest fleet version without any problem, but the last 2 are 3 hours in the progress of downloading docker images and, sometimes fails with strange errors like

Failed to download image 'registry2.balena-cloud.com/v2/4386bd3d2c58b4b54155b5f67e9b61ec@sha256:91c88e3f5551f57675e750e9eb49f36911a6011eeebd2e23bb65629529367ff9'
due to 'connect ECONNREFUSED /var/run/balena-engine.sock

Or

Failed to download image 'registry2.balena-cloud.com/v2/7055a43d63d6a8018d640ee4ea51cd37@sha256:4e2ed557ebed9d7d4fb0999efa108002c1766130d452423ea227cc21d1ee4750'
due to '(HTTP code 500) server error - Get "https://registry2.balena-cloud.com/v2/": net/http: TLS handshake timeout 

I’ve also tested with Development & PRoduction editions of BalenaOS it still act the same evn on latest BalenaOS version (2.107.1+rev1 at the time of writing).

To complete test i’ve tried another internet access without any success.

I’d be happy to give you more informations about this issue.

Thanks in advance !
Have a nice day :wink:

Hi there, thanks for reaching us. Could you share the engine logs from before you get the connect ECONNREFUSED /var/run/balena-engine.sock?

You can use the time when the error happens in the logs to isolate the results, for instance

journalctl -a --until 'YYYY-MM-DD hh:mm:ss' --no-pager

Could you also share the results of what happens when downloading one of our public images manually? Do you still get TLS handshake timeouts?

# Pull a supervisor image
balena pull registry2.balena-cloud.com/v2/0ff5f211eb0787cde6700b220dbabe88

Could you also share some more details about your connection setup? TLS handshake timeouts usually happen due to connectivity issues. Have you noticed other issues with connectivity?

Thank you

Hey,

After running the journalctl command i’ve came up with theese logs (i could not post then in the body of the message, there was too much caracters).

balena.log (124.0 KB)

I’ve tried downloading the image but it seems that the Engine is not running.

Using default tag: latest
root@c2f9720:~# balena pull registry2.balena-cloud.com/v2/0ff5f211eb0787cde6700b220dbabe88
Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?

To clarify things, not only the image downloading is really slow, but accessing the devices in balenaCloud dashboard is an hassle too. It take maybe 10 to 15 minutes for new devices to show on my dashboard.

For my connection setup i use a Usb to Ethernet adapter as my connection to internet, the internal Ethernet port is used for local networking communications with other devices.
I do have system-connections files in the /boot directory of the SD Card which are the following:

balena-enp1s0u1 for the USB to Ethernet adapter

[connection]
id=ext-net
type=ethernet
interface-name=enp1s0u1
permissions=
secondaries=

[ethernet]
mac-address-blacklist=

[ipv4]
dns-search=
method=auto
route-metric=100

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

balena-eth0 for the internal ethernet port

[connection]
id=int-net
type=ethernet
interface-name=eth0

[ipv4]
address1=192.168.1.2/24
method=auto
never-default=1
ingore-auto-routes=1
route-metric=0
route1=192.168.1.3/32
route2=192.168.1.10/32

[ipv6]
addr-gen-mode=stable-privacy
method=auto

We have a fiber access at the office that works fine, no packet loss and no speed issues.

How can i provide you more details ?

Thanks for the fast reply!

After some investigation I just found out that Balena Services are not in good shapes with devices where i have trouble with.

Here is the systemctl result on a faulty device.

root@c2f9720:~# systemctl | grep balena
  sys-devices-virtual-net-balena0.device                                                                                 loaded active     plugged         /sys/devices/virtual/net/balena0
  sys-subsystem-net-devices-balena0.device                                                                               loaded active     plugged         /sys/subsystem/net/devices/balena0
  etc-balena\x2dsupervisor.mount                                                                                         loaded active     mounted         /etc/balena-supervisor
  usr-share-ca\x2dcertificates-balena.mount                                                                              loaded active     mounted         /usr/share/ca-certificates/balena
  balena-hostname-conf.path                                                                                              loaded active     waiting         balena-hostname path watch
  balena-net-config-conf.path                                                                                            loaded active     waiting         balena-net-config path watch
  balena-ntp-config-conf.path                                                                                            loaded active     waiting         balena-ntp-config path watch
  balena-supervisor-conf.path                                                                                            loaded active     waiting         balena-supervisor path watch
  extract-balena-ca-conf.path                                                                                            loaded active     waiting         extract-balena-ca path watch
  balena-device-uuid.service                                                                                             loaded active     exited          Balena device UUID
  balena-hostname.service                                                                                                loaded active     exited          Balena Hostname Configuration
  balena-info@tty1.service                                                                                               loaded active     exited          Balena info on tty1
  balena-net-config.service                                                                                              loaded active     exited          Resin network configure service
  balena-ntp-config.service                                                                                              loaded active     exited          Resin NTP server configure service
  balena-persistent-logs.service                                                                                         loaded active     exited          Balena persistent logs
  balena-proxy-config.service                                                                                            loaded active     exited          Resin proxy configuration service
● balena-supervisor-conf.service                                                                                         loaded failed     failed          balena-supervisor.json watcher service
  balena-supervisor.service                                                                                              loaded inactive   dead      start Balena supervisor
  balena.service                                                                                                         loaded activating start     start Balena Application Container Engine
  bind-etc-balena-supervisor.service                                                                                     loaded active     exited          Bind mount for /etc/balena-supervisor
  bind-usr-share-ca-certificates-balena.service                                                                          loaded active     exited          Bind mount for /usr/share/ca-certificates/balena
  system-balena\x2dinfo.slice                                                                                            loaded active     active          Slice /system/balena-info
  balena-engine.socket                                                                                                   loaded active     listening       Docker Socket for the API
  balena-host.socket                                                                                                     loaded active     listening       Docker Socket for the API
  update-balena-supervisor.timer                                                                                         loaded active     waiting         Balena supervisor updater timer

And Here is on a good working device

root@c74f18f:~# systemctl | grep balena
  sys-devices-virtual-net-balena0.device                                                                                 loaded active plugged   /sys/devices/virtual/net/balena0
  sys-subsystem-net-devices-balena0.device                                                                               loaded active plugged   /sys/subsystem/net/devices/balena0
  etc-balena\x2dsupervisor.mount                                                                                         loaded active mounted   /etc/balena-supervisor
  usr-share-ca\x2dcertificates-balena.mount                                                                              loaded active mounted   /usr/share/ca-certificates/balena
  balena-hostname-conf.path                                                                                              loaded active waiting   balena-hostname path watch
  balena-net-config-conf.path                                                                                            loaded active waiting   balena-net-config path watch
  balena-ntp-config-conf.path                                                                                            loaded active waiting   balena-ntp-config path watch
  balena-supervisor-conf.path                                                                                            loaded active waiting   balena-supervisor path watch
  extract-balena-ca-conf.path                                                                                            loaded active waiting   extract-balena-ca path watch
  balena-device-uuid.service                                                                                             loaded active exited    Balena device UUID
  balena-hostname.service                                                                                                loaded active exited    Balena Hostname Configuration
  balena-info@tty1.service                                                                                               loaded active exited    Balena info on tty1
  balena-net-config.service                                                                                              loaded active exited    Resin network configure service
  balena-ntp-config.service                                                                                              loaded active exited    Resin NTP server configure service
  balena-persistent-logs.service                                                                                         loaded active exited    Balena persistent logs
  balena-proxy-config.service                                                                                            loaded active exited    Resin proxy configuration service
  balena-supervisor.service                                                                                              loaded active running   Balena supervisor
  balena.service                                                                                                         loaded active running   Balena Application Container Engine
  bind-etc-balena-supervisor.service                                                                                     loaded active exited    Bind mount for /etc/balena-supervisor
  bind-usr-share-ca-certificates-balena.service                                                                          loaded active exited    Bind mount for /usr/share/ca-certificates/balena
  system-balena\x2dinfo.slice                                                                                            loaded active active    Slice /system/balena-info
  balena-engine.socket                                                                                                   loaded active running   Docker Socket for the API
  balena-host.socket                                                                                                     loaded active listening Docker Socket for the API
  update-balena-supervisor.timer                                                                                         loaded active waiting   Balena supervisor updater timer

On the faulty one the Balena.service, which i assume is the balenaEngine, seems to be dead.

Maybe it gives you some hints about the situation. :slight_smile:

Update on this subject,

It appears that the SD Card (PNY) i was using on both Raspberry pi was not adapted, i tried with a Sandisk one and it went back to normal.

I can’t explain this because read write speeds were nearly the same on both PNY and Sandisk SD Cards.

I can’t explain either why BalenaEngine was in a faulty state on the PNY SD cards.

Can close this.

Hello, are you still experiencing this issue? If so, can you please run balena ps in a shell on the device? If balena engine is not running then it should fail. You can also check the status by running systemctl status balena. Next question is which supervisor version is your device running? If it is running a supervisor version < 14.0.0, then the following may be a solution for the trouble with the engine:

  • run journalctl -xe to see if the socket is already in use. If it is, it will look something like this:
root@40b5a3e:~# journalctl -xe
Feb 07 16:52:08 40b5a3e systemd[1]: balena-supervisor.service: Failed with result 'exit-code'.
Feb 07 16:52:08 40b5a3e systemd[1]: Failed to start Balena supervisor.
Feb 07 16:52:18 40b5a3e systemd[1]: balena-engine.socket: Failed to create listening socket (/run/balena-engine.sock): Address already in use
Feb 07 16:52:18 40b5a3e systemd[1]: balena-engine.socket: Failed to listen on sockets: Address already in use
Feb 07 16:52:18 40b5a3e systemd[1]: balena-engine.socket: Failed with result 'resources'.
Feb 07 16:52:18 40b5a3e systemd[1]: Failed to listen on Docker Socket for the API.
Feb 07 16:52:18 40b5a3e systemd[1]: Dependency failed for Balena Application Container Engine.
Feb 07 16:52:18 40b5a3e systemd[1]: balena.service: Job balena.service/start failed with result 'dependency'.
  • If the socket is in use, check if it is a valid socket and not a directory via ls -la /run | grep 'balena-engine'. The output may look as follows:
root@40b5a3e:~# ls -la /run | grep 'balena-engine'
drwx------  7 root   root           160 Feb  7 18:52 balena-engine
-rw-r--r--  1 root   root             7 Feb  7 18:52 balena-engine.pid
drwxr-xr-x  2 root   root            40 Jan 28 20:21 balena-engine.sock
srw-rw----  1 root   balena-engine    0 Jan 13 18:07 balena-host.sock
lrwxrwxrwx  1 root   root            22 Jan 13 18:07 balena.pid -> /run/balena-engine.pid
srw-rw----  1 root   balena-engine    0 Feb  7 18:52 balena.sock
lrwxrwxrwx  1 root   root            22 Jan 13 18:07 docker.pid -> /run/balena-engine.pid
  • Judging by the above output, you could see that balena-engine.sock is a directory. This is the source of the bug since the OS cannot recover from this. To fix it so the socket can be recreated correctly, you might have to restart the engine with systemctl restart balena.

Hope that helps. To avoid future instances of this issue, we advise that you upgrade to Supervisor v14.0.0+. Please let us know the results of any of the above points ^ (assuming the info matches your case)