# SSH has stopped working to all devices **URL:** \ **Category:** openBalena\ **Tags:** ssh\ **Created:** [May 1, 2021, 8:51am UTC](https://forums.balena.io/t/ssh-has-stopped-working-to-all-devices/306831 "2021-05-01T08:51:35Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![dash](https://avatars.discourse-cdn.com/v4/letter/d/7c8e57/32.png) [@dash](https://forums.balena.io/u/dash)\ **Post date:** [May 1, 2021, 8:51am UTC](https://forums.balena.io/t/ssh-has-stopped-working-to-all-devices/306831/1 "2021-05-01T08:51:35Z")
Hi All, I have a 20x devices running for around a year or so without any issues. I am running openBalena 2.0.1 and the devices are running BalenaOS versions between 2.44 and 2.58. I have previously been able to ssh them using proxy tunnel as described in ’ [HowTo: SSH into host device](https://forums.balena.io/t/howto-ssh-into-host-device/5312) @richbayliss. Just in the last couple of days though I have tried to ssh any of the devices I am running and receiving the following error: $ ssh root@d6a20843efed99bbe56f813ac4b797e2.balena Via xxx.xxx.xxx.xxx:3128 → d6a20843efed99bbe56f813ac4b797e2.balena:22222 analyze\_HTTP: readline failed: Connection closed by remote host ssh\_exchange\_identification: Connection closed by remote host All of the devices show online status and in the logs I can see them making request to the API regularly. What is interesting is that when looking at the certificate presented by the API end it was automatically renewed earlier this week which makes suspect there is some sort of certificate error occurring with the VPN. This is also interesting as Letsencrypt recently added a new ROOT CA - I’m wondering if perhaps these devices do not have this Root CA installed in their base OS? In short has anyone else experienced this or have any advice on how I might be able to debug VPN/SSH to identify the root cause. Cheers, Chris ---
**Author:** ![dash](https://avatars.discourse-cdn.com/v4/letter/d/7c8e57/32.png) [@dash](https://forums.balena.io/u/dash)\ **Post date:** [May 2, 2021, 3:42am UTC](https://forums.balena.io/t/ssh-has-stopped-working-to-all-devices/306831/3 "2021-05-02T03:42:03Z")
Just updating with debugging done to date: - Inspected logs of all openBalena server containers. Can see all of the devices regularly checking in with API using /device/v2/(uuid)/state endpoint. - Updated all OS packages on openBalena server and rebooted. - Tested certificates on endpoints with openssl results as follows: - [api.xxxx.com:443](http://api.xxxx.com:443) - Ok - [vpn.xxxx.com:443](http://vpn.xxxx.com:443) - Ok but certificate is that same as that used for [api.xxxx.com](http://api.xxxx.com)? - [vpn.xxxx.com:3128](http://vpn.xxxx.com:3128) - This is the VPN port I believe? Should this have SSL? - result is: > ## CONNECTED(00000005) 140263854088640:error:1408F10B:SSL routines:ssl3\_get\_record:wrong version number:…/ssl/record/ssl3\_record.c:332: > > ## no peer certificate available > > ## No client certificate CA names sent > > ## SSL handshake has read 5 bytes and written 327 bytes Verification: OK > > New, (NONE), Cipher is (NONE) > Secure Renegotiation IS NOT supported > Compression: NONE > Expansion: NONE > No ALPN negotiated > Early data was not sent > Verify return code: 0 (ok) - Provisioned new device from the same image as was used for most recently provisioned devices (intel-nuc 2.50.1). This new device is not connecting to either API or VPN nothing specific to it coming through in logs. My intention now is to try provision a new device with 2.58.6 x86 Generic image and connect it to this openBalena server and see if it will connect. Any guidance on how to debug this would be much appreciated. ---
**Author:** ![dash](https://avatars.discourse-cdn.com/v4/letter/d/7c8e57/32.png) [@dash](https://forums.balena.io/u/dash)\ **Post date:** [May 2, 2021, 4:45am UTC](https://forums.balena.io/t/ssh-has-stopped-working-to-all-devices/306831/4 "2021-05-02T04:45:04Z")
Okay further investigation. As my suspicion is SSL based I started looking through the certificates in the open-balena/config/certs directory. Looking at /open-balena/config/certs/root/issues/\*.mydomain.com.crt I noticed the following: > ``` > Certificate: > Data: > Version: 3 (0x2) > Serial Number: > db:a1:be:74:fa:31:f1:08:74:8c:2e:7c:1b:5e:1a:50 > Signature Algorithm: sha256WithRSAEncryption > Issuer: CN=ca.mydomain.com > Validity > Not Before: May 1 03:33:13 2019 GMT > Not After : Apr 30 03:33:13 2021 GMT > > ``` Also the VPN-CA: > Certificate: > Data: > Version: 3 (0x2) > Serial Number: > 88:5e:45:21:43:c0:32:18:dc:2d:ce:4e:f4:60:2b:da > Signature Algorithm: sha256WithRSAEncryption > Issuer: [CN=vpn-ca.mydomain.com](http://CN=vpn-ca.mydomain.com) > Validity > Not Before: May 1 03:33:15 2019 GMT > Not After : Apr 30 03:33:15 2021 GMT > Subject: [CN=vpn.mydomain.com](http://CN=vpn.mydomain.com) This would line up with when ssh stopped working (nothing has been changed about this server or openbalena config in months. Would the the root CA expiring stop VPN connects working properly/particularly tunnels via the VPN? Also what is the procedure for renewing the CA and safely having existing devices continue to work after renewal? ---
**Author:** ![dash](https://avatars.discourse-cdn.com/v4/letter/d/7c8e57/32.png) [@dash](https://forums.balena.io/u/dash)\ **Post date:** [May 2, 2021, 8:49am UTC](https://forums.balena.io/t/ssh-has-stopped-working-to-all-devices/306831/5 "2021-05-02T08:49:53Z")
Actually looking at logs for VPN there is alot of this: > May 02 08:47:51 4adc57eb6870 haproxy[231]: Routing 223.186.35.235:1373@tcp-443 to vpn-cluster/vpn1:59996 [C:2/2 Q:0/0 T:1/0] > May 02 08:47:51 4adc57eb6870 balena-vpn-api[1489]: TCP connection established with [AF\_INET]127.0.0.1:59996 > May 02 08:47:52 4adc57eb6870 balena-vpn-api[1489]: 127.0.0.1:59992 Connection reset, restarting [0] > May 02 08:47:52 4adc57eb6870 balena-vpn-api[1489]: 127.0.0.1:59996 Connection reset, restarting [0] VPN connections appear to be being immediately reset. Will see if I can find some more verbosity on that\> ---
**Author:** ![dash](https://avatars.discourse-cdn.com/v4/letter/d/7c8e57/32.png) [@dash](https://forums.balena.io/u/dash)\ **Post date:** [May 2, 2021, 10:42am UTC](https://forums.balena.io/t/ssh-has-stopped-working-to-all-devices/306831/6 "2021-05-02T10:42:20Z")
Sorry for the spam. Just trying to share all the information I have and steps I have tried. So I have setup a new device with dev BalenaOS version. It isn’t connected to the openBalena server (its status is offline). When I ssh into it locally and look at the logs for supervisor I see this: > root@00cc9cf:~# balena logs resin\_supervisor --tail 10000 -f > [api] GET /v1/healthy 200 - 12.964 ms > [api] GET /v1/healthy 200 - 1.319 ms > [debug] Attempting container log timestamp flush… > [debug] Container log timestamp flush complete > WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping > Warning: Ignoring extra certs from `/etc/ssl/certs/balenaRootCA.pem`, load failed: error:02001002:system library:fopen:No such file or directory > [success] Device state apply success This would suggest it is connecting to API and updating state correctly however its state remains offline when query device with balena cli. Looking in config.json on the image that was used to provision this device ‘balenaRootCA’ is clearly set. Also checking /mnt/boot/config.json balenaRootCA is clearly set on this device as well. I can also tail the logs for openvpn on this device and can see this: > root@00cc9cf:/resin-boot# systemctl status openvpn.service > openvpn.service - OpenVPN > Loaded: loaded (/lib/systemd/system/openvpn.service; enabled; vendor preset: enabled) > Active: active (running) since Sun 2021-05-02 10:16:48 UTC; 36min ago > Main PID: 1150 (openvpn) > Tasks: 1 (limit: 2358) > Memory: 2.1M > CGroup: /system.slice/openvpn.service > └─1150 /usr/sbin/openvpn --writepid /run/openvpn/openvpn.pid --cd /etc/openvpn/ --config /etc/openvpn/openvpn.conf --connect-retry 5 120 > > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 TLS: Initial packet from [AF\_INET]3.104.60.33:443, sid=9aa02bbf c23f58b3 > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 VERIFY OK: depth=1, [CN=vpn-ca.mydomain.com](http://CN=vpn-ca.mydomain.com) > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 VERIFY ERROR: depth=0, error=certificate has expired: [CN=vpn.mydomain.com](http://CN=vpn.mydomain.com) > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 OpenSSL: error:1416F086:SSL routines:tls\_process\_server\_certificate:certificate verify failed > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 TLS\_ERROR: BIO read tls\_read\_plaintext error > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 TLS Error: TLS object → incoming plaintext read error > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 TLS Error: TLS handshake failed > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 Fatal TLS error (check\_tls\_errors\_co), restarting > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 SIGUSR1[soft,tls-error] received, process restarting > May 02 10:52:05 00cc9cf openvpn[1150]: Sun May 2 10:52:05 2021 Restart pause, 120 second(s) > root@00cc9cf:/resin-boot# Seems to be the smoking gun re: vpn certificate expiry being the issue, will see if I can work out how to renew safely at server end. ---
**Author:** ![dash](https://avatars.discourse-cdn.com/v4/letter/d/7c8e57/32.png) [@dash](https://forums.balena.io/u/dash)\ **Post date:** [May 2, 2021, 11:39am UTC](https://forums.balena.io/t/ssh-has-stopped-working-to-all-devices/306831/7 "2021-05-02T11:39:14Z")
Okay all solved. Ma man @wolf_karl has all the answers you seek over here if you need to renew vpn certificate: [VPN Certs seems to be expired - #10 by wolf\_karl](https://forums.balena.io/t/vpn-certs-seems-to-be-expired/262191/10)