Balena Container Loses Connection to Network

Product support balenaOS
,
1

We have an install of approximately 50 units running Pi Compute hardware and the Balena Fin base OS image, running off 2 routers. Sometimes a few devices will drop off Balena, showing disconnected or heartbeat-only in the dashboard, BUT the router shows they are still connected. From the router console, refreshing the connection (I suppose disconnecting and letting DHCP run again, or otherwise refreshing the link) causes the device to come back online with Balena. While the device is not fully online with Balena, applications in the container cannot communicate with the network. This means the devices are non functional and it is becoming a problem. Is it possible that the container gets in a state where it no longer has network access ?

Another issue is frequently dropped ssh (dashboard) connections. We captured the last 500 lines of journalctl in one case. After ssh is up, openvpn resets, then crashes, restarts but the ssh session drops due to timeout (perhaps vpn went away from under it).

3

Is there any way to attach files to this ?

Here is the tail end of journalctl output which contains the openvpn crash and ssh timeout described above.

Nov 24 16:07:37 57448a8 sshd[16113]: error: Unable to load host key: /etc/ssh/hostkeys/ssh_host_dsa_key
Nov 24 16:07:37 57448a8 sshd[16113]: Accepted none for root from 52.4.252.97 port 35032 ssh2
Nov 24 16:08:31 57448a8 sshd[16278]: error: Unable to load host key: /etc/ssh/hostkeys/ssh_host_dsa_key
Nov 24 16:08:32 57448a8 sshd[16278]: Accepted none for root from 52.4.252.97 port 42028 ssh2
Nov 24 16:09:04 57448a8 fde85245a63d[1812]: [api]     GET /v1/healthy 200 - 14.039 ms
Nov 24 16:09:04 57448a8 resin-supervisor[2451]: [api]     GET /v1/healthy 200 - 14.039 ms
Nov 24 16:10:18 57448a8 balenad[1853]: time="2021-11-24T16:10:18.250070596Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/5f81481f573a1e0cf226ad2e2e1dc27718fe90e1ab63aea7d66e947c76abc8e7.sock debug=false pid=16795
Nov 24 16:10:23 57448a8 balenad[1853]: time="2021-11-24T16:10:23.172657671Z" level=info msg="shim reaped" id=0cc0475f95f50554caca98e8feff8fc68e70fb517f664c559803bd7bf4d13637
Nov 24 16:10:23 57448a8 balenad[1812]: time="2021-11-24T16:10:23.176501917Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 24 16:10:55 57448a8 openvpn[3834]: Wed Nov 24 16:10:55 2021 Connection reset, restarting [-1]
Nov 24 16:10:55 57448a8 openvpn[3834]: Wed Nov 24 16:10:55 2021 /etc/openvpn-misc/downscript.sh resin-vpn 1500 1555 10.240.39.2 52.4.252.97 restart
Nov 24 16:10:55 57448a8 fde85245a63d[1812]: [info]    VPN connection is not active.
Nov 24 16:10:55 57448a8 resin-supervisor[2451]: [info]    VPN connection is not active.
Nov 24 16:10:55 57448a8 openvpn[3834]: Wed Nov 24 16:10:55 2021 SIGUSR1[soft,connection-reset] received, process restarting
Nov 24 16:10:55 57448a8 openvpn[3834]: Wed Nov 24 16:10:55 2021 Restart pause, 5 second(s)
Nov 24 16:11:00 57448a8 openvpn[3834]: Wed Nov 24 16:11:00 2021 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 24 16:11:00 57448a8 openvpn[3834]: Wed Nov 24 16:11:00 2021 TCP/UDP: Preserving recently used remote address: [AF_INET]3.227.28.93:443
Nov 24 16:11:00 57448a8 openvpn[3834]: Wed Nov 24 16:11:00 2021 Socket Buffers: R=[131072->131072] S=[16384->16384]
Nov 24 16:11:00 57448a8 openvpn[3834]: Wed Nov 24 16:11:00 2021 Attempting to establish TCP connection with [AF_INET]3.227.28.93:443 [nonblock]
Nov 24 16:11:01 57448a8 openvpn[3834]: Wed Nov 24 16:11:01 2021 TCP connection established with [AF_INET]3.227.28.93:443
Nov 24 16:11:01 57448a8 openvpn[3834]: Wed Nov 24 16:11:01 2021 Socket flags: TCP_NODELAY=1 succeeded
Nov 24 16:11:01 57448a8 openvpn[3834]: Wed Nov 24 16:11:01 2021 TCP_CLIENT link local: (not bound)
Nov 24 16:11:01 57448a8 openvpn[3834]: Wed Nov 24 16:11:01 2021 TCP_CLIENT link remote: [AF_INET]3.227.28.93:443
Nov 24 16:11:01 57448a8 openvpn[3834]: Wed Nov 24 16:11:01 2021 TLS: Initial packet from [AF_INET]3.227.28.93:443, sid=217412fc 712f1510
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 VERIFY OK: depth=1, C=AU, ST=Some-State, O=Internet Widgits Pty Ltd
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 VERIFY KU OK
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 Validating certificate extended key usage
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 VERIFY EKU OK
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 VERIFY OK: depth=0, C=US, ST=WA, O=balena.io, OU=balenaCloud, CN=vpn.balena-cloud.com
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
Nov 24 16:11:02 57448a8 openvpn[3834]: Wed Nov 24 16:11:02 2021 [vpn.balena-cloud.com] Peer Connection Initiated with [AF_INET]3.227.28.93:443
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 SENT CONTROL [vpn.balena-cloud.com]: 'PUSH_REQUEST' (status=1)
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 PUSH: Received control message: 'PUSH_REPLY,sndbuf 0,rcvbuf 0,route 52.4.252.97,ping 10,ping-restart 60,socket-flags TCP_NODELAY,ifconfig 10.240.105.65 52.4.252.97,peer-id 0,cipher AES-128-GCM'
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: timers and/or timeouts modified
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: --sndbuf/--rcvbuf options modified
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Socket Buffers: R=[131072->131072] S=[44800->44800]
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: --socket-flags option modified
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Socket flags: TCP_NODELAY=1 succeeded
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: --ifconfig/up options modified
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: route options modified
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: peer-id set
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: adjusting link_mtu to 1627
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 OPTIONS IMPORT: data channel crypto options modified
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Data Channel: using negotiated cipher 'AES-128-GCM'
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Outgoing Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Incoming Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Preserving previous TUN/TAP instance: resin-vpn
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 /etc/openvpn-misc/upscript.sh resin-vpn 1500 1555 10.240.39.2 52.4.252.97 restart
Nov 24 16:11:03 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 resin-supervisor[2451]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 resin-supervisor[2451]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 resin-supervisor[2451]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 resin-supervisor[2451]: [info]    VPN connection is active.
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Closing TUN/TAP interface
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 /sbin/ip addr del dev resin-vpn local 10.240.39.2 peer 52.4.252.97
Nov 24 16:11:03 57448a8 openvpn[16998]: ip: RTNETLINK answers: Operation not permitted
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 Linux ip addr del failed: external program exited with error status: 2
Nov 24 16:11:03 57448a8 openvpn[3834]: Wed Nov 24 16:11:03 2021 /etc/openvpn-misc/downscript.sh resin-vpn 1500 1555 10.240.39.2 52.4.252.97 init
Nov 24 16:11:03 57448a8 fde85245a63d[1812]: [info]    VPN connection is not active.
Nov 24 16:11:03 57448a8 resin-supervisor[2451]: [info]    VPN connection is not active.
Nov 24 16:11:04 57448a8 fde85245a63d[1812]: [info]    Waiting for connectivity...
Nov 24 16:11:04 57448a8 resin-supervisor[2451]: [info]    Waiting for connectivity...
Nov 24 16:11:04 57448a8 openvpn[3834]: Wed Nov 24 16:11:04 2021 ROUTE_GATEWAY 10.0.0.1/255.255.255.0 IFACE=wlan0 HWADDR=48:a4:93:49:19:a2
Nov 24 16:11:04 57448a8 openvpn[3834]: Wed Nov 24 16:11:04 2021 ERROR: Cannot ioctl TUNSETIFF resin-vpn: Operation not permitted (errno=1)
Nov 24 16:11:04 57448a8 openvpn[3834]: Wed Nov 24 16:11:04 2021 Exiting due to fatal error
Nov 24 16:11:04 57448a8 systemd[1]: openvpn.service: Main process exited, code=exited, status=1/FAILURE
Nov 24 16:11:04 57448a8 systemd[1]: openvpn.service: Failed with result 'exit-code'.
Nov 24 16:11:14 57448a8 prepare-openvpn[17020]: prepare-openvpn: [INFO] Balena.io VPN authentication.
Nov 24 16:11:15 57448a8 openvpn[17023]: Wed Nov 24 16:11:15 2021 WARNING: file '/var/volatile/vpn-auth' is group or others accessible
Nov 24 16:11:15 57448a8 openvpn[17023]: Wed Nov 24 16:11:15 2021 OpenVPN 2.4.7 arm-poky-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Feb 20 2019
Nov 24 16:11:15 57448a8 openvpn[17023]: Wed Nov 24 16:11:15 2021 library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
Nov 24 16:11:15 57448a8 openvpn[17023]: Wed Nov 24 16:11:15 2021 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 24 16:11:15 57448a8 openvpn[17023]: Wed Nov 24 16:11:15 2021 TCP/UDP: Preserving recently used remote address: [AF_INET]35.169.76.143:443
Nov 24 16:11:15 57448a8 openvpn[17023]: Wed Nov 24 16:11:15 2021 Socket Buffers: R=[131072->131072] S=[16384->16384]
Nov 24 16:11:15 57448a8 openvpn[17023]: Wed Nov 24 16:11:15 2021 Attempting to establish TCP connection with [AF_INET]35.169.76.143:443 [nonblock]
Nov 24 16:11:16 57448a8 openvpn[17023]: Wed Nov 24 16:11:16 2021 TCP connection established with [AF_INET]35.169.76.143:443
Nov 24 16:11:16 57448a8 openvpn[17023]: Wed Nov 24 16:11:16 2021 TCP_CLIENT link local: (not bound)
Nov 24 16:11:16 57448a8 openvpn[17023]: Wed Nov 24 16:11:16 2021 TCP_CLIENT link remote: [AF_INET]35.169.76.143:443
Nov 24 16:11:16 57448a8 openvpn[17023]: Wed Nov 24 16:11:16 2021 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Nov 24 16:11:16 57448a8 openvpn[17023]: Wed Nov 24 16:11:16 2021 TLS: Initial packet from [AF_INET]35.169.76.143:443, sid=80e91818 8a7e8388
Nov 24 16:11:16 57448a8 openvpn[17023]: Wed Nov 24 16:11:16 2021 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 VERIFY OK: depth=1, C=AU, ST=Some-State, O=Internet Widgits Pty Ltd
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 VERIFY KU OK
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 Validating certificate extended key usage
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 VERIFY EKU OK
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 VERIFY OK: depth=0, C=US, ST=WA, O=balena.io, OU=balenaCloud, CN=vpn.balena-cloud.com
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
Nov 24 16:11:17 57448a8 openvpn[17023]: Wed Nov 24 16:11:17 2021 [vpn.balena-cloud.com] Peer Connection Initiated with [AF_INET]35.169.76.143:443
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 SENT CONTROL [vpn.balena-cloud.com]: 'PUSH_REQUEST' (status=1)
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 PUSH: Received control message: 'PUSH_REPLY,sndbuf 0,rcvbuf 0,route 52.4.252.97,ping 10,ping-restart 60,socket-flags TCP_NODELAY,ifconfig 10.240.17.100 52.4.252.97,peer-id 0,cipher AES-128-GCM'
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: timers and/or timeouts modified
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: --sndbuf/--rcvbuf options modified
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 Socket Buffers: R=[131072->131072] S=[44800->44800]
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: --socket-flags option modified
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 Socket flags: TCP_NODELAY=1 succeeded
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: --ifconfig/up options modified
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: route options modified
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: peer-id set
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: adjusting link_mtu to 1627
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 OPTIONS IMPORT: data channel crypto options modified
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 Data Channel: using negotiated cipher 'AES-128-GCM'
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 Outgoing Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 Incoming Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 ROUTE_GATEWAY 10.0.0.1/255.255.255.0 IFACE=wlan0 HWADDR=48:a4:93:49:19:a2
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 TUN/TAP device resin-vpn opened
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 TUN/TAP TX queue length set to 100
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 /sbin/ip link set dev resin-vpn up mtu 1500
Nov 24 16:11:18 57448a8 NetworkManager[1802]: <info>  [1637770278.8598] manager: (resin-vpn): new Tun device (/org/freedesktop/NetworkManager/Devices/11)
Nov 24 16:11:18 57448a8 systemd-udevd[17052]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 /sbin/ip addr add dev resin-vpn local 10.240.17.100 peer 52.4.252.97
Nov 24 16:11:18 57448a8 openvpn[17023]: Wed Nov 24 16:11:18 2021 /etc/openvpn-misc/upscript.sh resin-vpn 1500 1555 10.240.17.100 52.4.252.97 init
Nov 24 16:11:19 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:19 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:19 57448a8 resin-supervisor[2451]: [info]    VPN connection is active.
Nov 24 16:11:19 57448a8 openvpn[17023]: Wed Nov 24 16:11:19 2021 /sbin/ip route add 52.4.252.97/32 via 52.4.252.97
Nov 24 16:11:19 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:19 57448a8 resin-supervisor[2451]: [info]    VPN connection is active.
Nov 24 16:11:19 57448a8 openvpn[17063]: ip: RTNETLINK answers: File exists
Nov 24 16:11:19 57448a8 fde85245a63d[1812]: [info]    VPN connection is active.
Nov 24 16:11:19 57448a8 resin-supervisor[2451]: [info]    VPN connection is active.
Nov 24 16:11:19 57448a8 openvpn[17023]: Wed Nov 24 16:11:19 2021 ERROR: Linux route add command failed: external program exited with error status: 2
Nov 24 16:11:19 57448a8 openvpn[17023]: Wed Nov 24 16:11:19 2021 GID set to openvpn
Nov 24 16:11:19 57448a8 openvpn[17023]: Wed Nov 24 16:11:19 2021 UID set to openvpn
Nov 24 16:11:19 57448a8 openvpn[17023]: Wed Nov 24 16:11:19 2021 Initialization Sequence Completed
Nov 24 16:11:24 57448a8 fde85245a63d[1812]: [info]    Internet Connectivity: OK
Nov 24 16:11:24 57448a8 resin-supervisor[2451]: [info]    Internet Connectivity: OK
Nov 24 16:12:06 57448a8 sshd[16113]: Timeout, client not responding from user root 52.4.252.97 port 35032
Nov 24 16:12:09 57448a8 sshd[3916]: Timeout, client not responding from user root 52.4.252.97 port 44194
Nov 24 16:12:10 57448a8 sshd[16278]: Timeout, client not responding from user root 52.4.252.97 port 42028
Nov 24 16:12:51 57448a8 sshd[17616]: error: Unable to load host key: /etc/ssh/hostkeys/ssh_host_dsa_key
Nov 24 16:12:51 57448a8 sshd[17616]: Accepted none for root from 52.4.252.97 port 37636 ssh2
Nov 24 16:13:25 57448a8 balenad[1853]: time="2021-11-24T16:13:25.160186306Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/573e4cd3b8f0b338ca4d525324564316491e6aceff54ce997a33118249e46883.sock debug=false pid=17756
Nov 24 16:13:30 57448a8 balenad[1853]: time="2021-11-24T16:13:30.885024355Z" level=info msg="shim reaped" id=9a881b2fee9e32875d485bd362ad1da4359ae18f022aca55d1108ac4159f33bc
Nov 24 16:13:30 57448a8 balenad[1812]: time="2021-11-24T16:13:30.888809695Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 24 16:13:31 57448a8 fde85245a63d[1812]: [debug]   Attempting container log timestamp flush...
Nov 24 16:13:31 57448a8 fde85245a63d[1812]: [debug]   Container log timestamp flush complete
Nov 24 16:13:31 57448a8 resin-supervisor[2451]: [debug]   Attempting container log timestamp flush...
Nov 24 16:13:31 57448a8 resin-supervisor[2451]: [debug]   Container log timestamp flush complete
Nov 24 16:14:07 57448a8 fde85245a63d[1812]: [api]     GET /v1/healthy 200 - 13.683 ms
Nov 24 16:14:07 57448a8 resin-supervisor[2451]: [api]     GET /v1/healthy 200 - 13.683 ms
Nov 24 16:16:32 57448a8 balenad[1853]: time="2021-11-24T16:16:32.545000102Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/8706116e0483e6bfb1804bad58059bacfe2b9c859fc62c6b0fd0bfb3084f0b38.sock debug=false pid=18298
Nov 24 16:16:37 57448a8 balenad[1853]: time="2021-11-24T16:16:37.147581374Z" level=info msg="shim reaped" id=0374d95e6646cffe7f2ff7b1280b3134b3f796aad0acbaa55a75b0e37d550665
Nov 24 16:16:37 57448a8 balenad[1812]: time="2021-11-24T16:16:37.159065988Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 24 16:19:10 57448a8 fde85245a63d[1812]: [api]     GET /v1/healthy 200 - 15.608 ms
Nov 24 16:19:10 57448a8 resin-supervisor[2451]: [api]     GET /v1/healthy 200 - 15.608 ms
1 Like
5

Hey,
Yes, you can attach (upload) files. When creating a new post, there is a button on the toolbar(roughly in the middle) with a picture on it of an arrow pointing upwards. Would be good to have journalctl logs from the device(s).
I see that the openvpn process is running into an error on the TUNSETIFF ioctl but can’t guess exactly why that is occurring.
How long have you been running into this issue and how many devices exhibit this behaviour? Would it be possible for you to test it a different router? We have never seen this issue earlier, hence I think it might be something to do with your network.

Thanks and regards,
Pranav

8

I’ve also been seeing this occasionally, and it just happened this morning. Nobody was SSHed into the device. No clues immediately before “Connection reset, restarting [-1]”. https://status.balena.io/ reports no issues with VPN (or anything else) today.

Jan 29 09:57:40 ceb0b73 kernel: w1_master_driver w1_bus_master1: Attaching one wire slave 00.fd2000
000000 crc 48
Jan 29 09:57:40 ceb0b73 kernel: w1_master_driver w1_bus_master1: Family 0 for 00.fd2000000000.48 is
 not registered.
Jan 29 09:58:45 ceb0b73 openvpn[16181]: Sat Jan 29 09:58:45 2022 Connection reset, restarting [-1]
Jan 29 09:58:45 ceb0b73 openvpn[16181]: Sat Jan 29 09:58:45 2022 /etc/openvpn-misc/downscript.sh re
sin-vpn 1500 1555 10.240.6.169 52.4.252.97 restart
Jan 29 09:58:45 ceb0b73 294aff3e2a9d[6876]: [info]    VPN connection is not active.
Jan 29 09:58:45 ceb0b73 balena-supervisor[18665]: [info]    VPN connection is not active.
Jan 29 09:58:45 ceb0b73 openvpn[16181]: Sat Jan 29 09:58:45 2022 SIGUSR1[soft,connection-reset] rec
eived, process restarting
Jan 29 09:58:45 ceb0b73 openvpn[16181]: Sat Jan 29 09:58:45 2022 Restart pause, 5 second(s)
Jan 29 09:58:50 ceb0b73 openvpn[16181]: Sat Jan 29 09:58:50 2022 NOTE: the current --script-securit
y setting may allow this configuration to call user-defined scripts

The Cannot ioctl TUNSETIFF resin-vpn: Operation not permitted that happens later is openvpn deescalates privileges which causes a hard failure on reconnect to different endpoint · Issue #1779 · balena-os/meta-balena (github.com)