Abnormal number of sshd processes and connectivity problems

Product support
1

Device type: Raspberry Pi (v1 / Zero / Zero W)
OS version: balenaOS 2.46.1+rev1
Supervisor version: 10.6.27

I was having a lot of difficulty establishing any communication with a device, even though it was marked as having VPN and could get a heartbeat. Reboot requests would time out. At one point I was finally able to connect and saw

bash-5.0# uptime
 07:59:00 up  5:19,  load average: 40.72, 39.90, 35.04

where normally it’s about 1.5.

I then saw that the number of processes was double what it normally is and the result of ps was:

bash-5.0# ps
PID   USER     TIME  COMMAND
    1 root     25:16 {systemd} /sbin/init
    2 root      0:00 [kthreadd]
    6 root      0:00 [mm_percpu_wq]
    7 root      1:46 [ksoftirqd/0]
    8 root      0:00 [kdevtmpfs]
    9 root      0:00 [netns]
   11 root      0:00 [khungtaskd]
   12 root      0:00 [oom_reaper]
   13 root      0:00 [writeback]
   14 root      0:00 [kcompactd0]
   15 root      0:00 [crypto]
   16 root      0:00 [kblockd]
   17 root      0:00 [watchdogd]
   18 root      0:00 [rpciod]
   19 root      0:00 [kworker/u3:0-hc]
   20 root      0:00 [xprtiod]
   21 root      0:01 [kswapd0]
   22 root      0:00 [nfsiod]
   33 root      0:00 [kthrotld]
   34 root      0:00 [iscsi_eh]
   35 root      0:00 [dwc_otg]
   36 root      0:00 [DWC Notificatio]
   37 root      0:00 [vchiq-slot/0]
   38 root      0:00 [vchiq-recy/0]
   39 root      0:00 [vchiq-sync/0]
   40 root      0:00 [vchiq-keep/0]
   41 root      0:00 [ipv6_addrconf]
   42 root      0:02 [irq/86-mmc1]
   49 root      0:00 [mmc_complete]
  325 root      0:00 sshd: root@pts/38
  336 root      0:00 /bin/bash -l
  404 root      0:00 [jbd2/mmcblk0p2-]
  405 root      0:00 [ext4-rsv-conver]
  421 root      0:00 [jbd2/mmcblk0p5-]
  426 root      0:00 [ext4-rsv-conver]
  500 root      1:55 /lib/systemd/systemd-journald
  569 root      0:02 [jbd2/mmcblk0p6-]
  570 root      0:00 [ext4-rsv-conver]
  609 root      0:01 /lib/systemd/systemd-udevd
  638 root      0:00 [spi0]
  673 root      0:00 [cfg80211]
  683 root      0:00 [brcmf_wq/mmc1:0]
  684 root      0:00 [brcmf_wdog/mmc1]
  716 root      0:07 /usr/sbin/rngd -f -r /dev/hwrng
  734 root      0:10 /sbin/syslogd -n
  735 root      3:22 /lib/systemd/systemd-logind
  752 997      14:29 /usr/bin/dbus-daemon --system --address=systemd: --nofork 
  758 root      0:00 /bin/bash -l
  765 root      0:00 /sbin/klogd -n
  778 root      0:00 /usr/sbin/chronyd -s -d
  795 root      0:00 /usr/bin/hciattach /dev/serial1 bcm43xx 3000000 flow - b8:
  796 root      0:00 [kworker/u3:2-hc]
  844 994       0:11 avahi-daemon: running [1e46499.local]
  845 994       0:00 avahi-daemon: chroot helper
  853 root      6:14 /usr/sbin/NetworkManager --no-daemon
  858 996       2:02 /usr/sbin/openvpn --writepid /run/openvpn/openvpn.pid --cd
  861 nobody    0:02 /usr/bin/dnsmasq -x /run/dnsmasq.pid -a 127.0.0.2,10.114.1
  866 root      5:58 /usr/bin/balenad --experimental --log-driver=journald -s a
  870 root      0:00 /proc/self/exe --healthcheck /usr/lib/balena/balena-health
  889 root      0:33 /usr/sbin/wpa_supplicant -u
  918 root      3:59 balena-engine-containerd --config /var/run/balena-engine/c
 1078 root      0:00 sshd: root@pts/17
 1108 root      0:00 /bin/bash -l
 1185 root      1:55 balena-engine-containerd-shim -namespace moby -workdir /va
 1232 root      0:01 /bin/bash -c . update.sh && . cmd.sh
 1334 root      0:00 udevd --daemon
 1443 root      0:21 python3 LED.py
 1444 root      0:31 /bin/bash ./connectCheck.sh
 1446 root      0:27 node main.js
 1535 root      0:21 python3 LED.py
 1595 root      0:00 {start-resin-sup} /bin/sh /usr/bin/start-resin-supervisor
 1598 root      0:00 /proc/self/exe --healthcheck /usr/lib/resin-supervisor/res
 1783 root      0:25 balena start --attach resin_supervisor
 1798 root      2:20 balena-engine-containerd-shim -namespace moby -workdir /va
 1814 root      1:31 node /usr/src/app/dist/app.js
 1853 999       0:00 /usr/bin/dbus-daemon --system
 1860 998       0:04 avahi-daemon: running [1e46499.local]
 1861 998       0:00 avahi-daemon: chroot helper
 1917 root      0:01 sshd: root@pts/12
 2022 root      0:00 /bin/bash -l
 2267 nobody    0:00 /usr/bin/dnsmasq --conf-file=/dev/null --no-hosts --keep-i
 3068 root      0:02 [kworker/0:0-eve]
 3996 root      0:00 sshd: root@pts/32
 4041 root      0:00 /bin/bash -l
 4548 root      0:00 sshd: root@pts/25
 4767 root      0:00 /bin/bash -l
 4945 root      0:00 sshd: root@pts/18
 5127 root      0:00 /bin/bash -l
 5197 root      0:00 [kworker/0:2H]
 5260 root      0:00 sshd: root@pts/36
 5369 root      0:00 /bin/bash -l
 6403 root      0:00 sshd: root@pts/33
 6545 root      0:00 /bin/bash -l
 6588 root      0:00 sshd: root@pts/26
 6715 root      0:00 [kworker/u2:3-ev]
 6857 root      0:00 /bin/bash -l
 6990 root      0:01 sshd: root@pts/13
 7114 root      0:00 /bin/bash -l
 7149 root      0:01 sshd: root@pts/1
 7241 root      0:00 /bin/bash -l
 7598 root      0:00 sshd: root@pts/39
 7727 root      0:00 sh -c X=/usr/bin/;ENG=rce;[ -x $X$ENG ]||ENG=docker;[ -x $
 8096 root      0:01 [kworker/0:1-eve]
 8107 root      0:00 balena exec -it 2a8eaac6d85947661ff82d74fe64c65eacac524f0a
 8155 root      0:00 /bin/bash
 8711 root      0:01 sshd: root@pts/14
 8789 root      0:00 sshd: root@pts/27
 8802 root      0:00 /bin/bash -l
 8826 root      0:01 sshd: root@pts/0
 8913 root      0:00 /bin/bash -l
 8993 root      0:00 /bin/bash -l
10423 root      0:01 sshd: root@pts/15
10464 root      0:00 /bin/bash -l
10844 root      0:01 sshd: root@pts/2
10922 root      0:00 /bin/bash -l
11251 root      0:00 sshd: root@pts/28
11328 root      0:00 sshd: root@pts/19
11419 root      0:00 /bin/bash -l
11467 root      0:00 /bin/bash -l
11554 root      0:00 [kworker/0:2-eve]
11644 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11645 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11646 root      0:00 /usr/sbin/sshd -i -f /etc/ssh/sshd_config_readonly
11647 root      0:00 /usr/sbin/sshd -i -f /etc/ssh/sshd_config_readonly
11648 root      0:00 /usr/sbin/sshd -i -f /etc/ssh/sshd_config_readonly
11649 root      0:00 /usr/sbin/sshd -i -f /etc/ssh/sshd_config_readonly
11650 root      0:00 /usr/sbin/sshd -i -f /etc/ssh/sshd_config_readonly
11651 root      0:00 /usr/sbin/sshd -i -f /etc/ssh/sshd_config_readonly
11652 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11653 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11654 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11655 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11656 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11657 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11658 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11659 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11660 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11661 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11662 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11663 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11664 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11665 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11666 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11667 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11668 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11669 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11670 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11671 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11672 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11673 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11674 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11675 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11676 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11677 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11678 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11680 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11681 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11682 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11683 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11684 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11685 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11686 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11687 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11689 root      0:00 ps
11690 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11691 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11692 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11693 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11695 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11696 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11697 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11699 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11700 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11702 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11703 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11704 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11705 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11707 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11708 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11709 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11710 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11711 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11714 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11715 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11719 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11720 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11721 root      0:00 cat /home/root/.ssh/authorized_keys
11722 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11723 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11724 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11725 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11726 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11727 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11728 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11729 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11730 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11731 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11732 root      0:00 {ssh_keys_merger} /bin/sh /usr/sbin/ssh_keys_merger
11733 root      0:00 /bin/bash ./connectCheck.sh
11734 root      0:00 nmcli c show comms-wifi
11735 root      0:00 grep 802-11-wireless.ssid
13416 root      0:00 sshd: root@pts/20
13604 root      0:00 /bin/bash -l
14112 root      0:00 sshd: root@pts/34
14249 root      0:01 sshd: root@pts/4
14324 root      0:00 /bin/bash -l
14387 root      0:00 /bin/bash -l
15516 root      0:00 sshd: root@pts/29
15521 root      0:00 sshd: root@pts/21
15693 root      0:00 /bin/bash -l
15739 root      0:00 /bin/bash -l
16223 root      0:00 [kworker/0:0H-kb]
16697 root      0:01 sshd: root@pts/5
16778 root      0:00 /bin/bash -l
17243 root      0:01 sshd: root@pts/16
17493 root      0:00 /bin/bash -l
17981 root      0:01 sshd: root@pts/6
18011 root      0:00 /bin/bash -l
19358 root      0:01 sshd: root@pts/7
19376 root      0:00 /bin/bash -l
19585 root      0:00 sshd: root@pts/22
19776 root      0:00 /bin/bash -l
21058 root      0:00 sshd: root@pts/3
21130 root      0:00 /bin/bash -l
21968 root      0:01 sshd: root@pts/8
22033 root      0:00 /bin/bash -l
23801 root      0:00 sshd: root@pts/23
24160 root      0:00 /bin/bash -l
24314 root      0:00 sshd: root@pts/30
24545 root      0:00 /bin/bash -l
24600 root      0:00 sshd: root@pts/37
24779 root      0:01 sshd: root@pts/9
24781 root      0:00 /bin/bash -l
25016 root      0:00 /bin/bash -l
26285 root      0:01 sshd: root@pts/10
26388 root      0:00 /bin/bash -l
27896 root      0:00 [kworker/u2:1-ev]
30696 root      0:04 [kworker/0:3-cgr]
31355 root      0:00 sshd: root@pts/31
31646 root      0:00 /bin/bash -l
32122 root      0:00 sshd: root@pts/35
32476 root      0:01 sshd: root@pts/11
32530 root      0:00 /bin/bash -l
32580 root      0:00 /bin/bash -l
32678 root      0:00 sshd: root@pts/24

A ridiculous number of bash shells and sshd processes. We had previously seen that on this system, it had trouble connecting to ntp servers. Result of chronyc sources:

root@1e46499:~# chronyc sources
210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^? time.cloudflare.com           0  14     0     -     +0ns[   +0ns] +/-    0ns
^? saturno.unad.edu.co           0  14     0     -     +0ns[   +0ns] +/-    0ns
^? 2803:eb80:8000:e::2           0  14     0     -     +0ns[   +0ns] +/-    0ns
^? host63.181.119.30.baehos>     0  14     0     -     +0ns[   +0ns] +/-    0ns

And

root@1e46499:~# journalctl -f -u chronyd.service

-- Logs begin at Fri 2020-04-10 03:30:48 UTC. --
Apr 10 07:14:11 1e46499 chronyd[778]: 2020-04-10T07:14:11Z Source 200.25.3.17 replaced with 2803:eb80:8000:e::2
-- Reboot --
Apr 10 07:14:14 1e46499 chronyd[768]: 2020-04-10T07:14:14Z Source 186.118.171.88 replaced with 94.16.122.254
Apr 10 16:26:00 1e46499 chronyd[768]: 2020-04-10T16:26:00Z Source 162.159.200.1 replaced with 2803:eb80:8000:e::2

At that point I lost connectivity again. I suspect part of the reason we have so much trouble connecting is difficulty getting correct time. But also the ridiculous number of sshd processes can’t be good.

2

Hello, it looks like you are hitting this issue: https://github.com/balena-os/meta-balena/issues/1837
This will be fixed in the next balena OS release.
In the meantime, the fix is to run systemctl stop sshd@* --state=activating

4

Yup - looks like it. Looks like the fix is to run that command on systems a-prior once we recognize that there are multiple ssd’s. Problem is that when that happens it’s very difficult to connect to said systems. How about the fix indicated marked to do with extreme caution? https://github.com/balena-os/meta-balena/issues/1837#issuecomment-589176326

5

It should work.

6

It says extreme caution because it’s a oneliner remounting the root partition read write and using sed to delete some lines in a text file.

8

Actually may be a different issue:

root@1e46499:~# systemctl list-units --state=activating
UNIT                                                LOAD   ACTIVE     SUB          DESCRIPTION                                      
sshd@42-10.240.4.33:22222-52.4.252.97:37542.service loaded activating auto-restart OpenSSH Per-Connection Daemon (52.4.252.97:37542)
sshd@66-10.240.4.33:22222-52.4.252.97:57362.service loaded activating auto-restart OpenSSH Per-Connection Daemon (52.4.252.97:57362)

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed.
root@1e46499:~# systemctl stop sshd@* --state=activating
root@1e46499:~# systemctl list-units --state=activating
UNIT                                                LOAD   ACTIVE     SUB          DESCRIPTION                                      
sshd@66-10.240.4.33:22222-52.4.252.97:57362.service loaded activating auto-restart OpenSSH Per-Connection Daemon (52.4.252.97:57362)

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed.
root@1e46499:~# 
root@1e46499:~# !ps
ps | grep ssh
  378 root      5436 S    sshd: root@pts/32
  979 root      5436 S    sshd: root@pts/6
 1573 root      5436 S    sshd: root@pts/33
 1613 root      5436 S    sshd: root@pts/7
 2085 sshd      2872 S    /usr/bin/dbus-daemon --system
 2092 sshd-aut  3564 S    avahi-daemon: running [1e46499.local]
 2093 sshd-aut  3432 S    avahi-daemon: chroot helper
 2298 root      5436 S    sshd: root@pts/8
 2809 root      5436 S    sshd: root@pts/1
 2851 root      5436 S    sshd: root@pts/34
 2950 root      5436 S    sshd: root@pts/0
 3550 root      5436 S    sshd: root@pts/2
 3894 root      5436 S    sshd: root@pts/9
 4032 root      5436 S    sshd: root@pts/35
 4730 root      5436 S    sshd: root@pts/10
 5702 root      5436 S    sshd: root@pts/11
 5810 root      5436 S    sshd: root@pts/36
 5974 root      5436 S    sshd: root@pts/37
 6327 root      5436 S    sshd: root@pts/38
 6797 root      5436 S    sshd: root@pts/12
 6828 root      5436 S    sshd: root@pts/14
 7274 root      5436 S    sshd: root@pts/39
 7800 root      5436 S    sshd: root@pts/40
 7957 root      5436 S    sshd: root@pts/13
 8262 root      5436 S    sshd: root@pts/41
 8753 root      5436 S    sshd: root@pts/42
 9166 root      5436 S    sshd: root@pts/43
 9192 root      5436 S    sshd: root@pts/15
 9223 root      5436 S    sshd: root@pts/44
 9414 root      2892 S    grep ssh
10306 root      5436 S    sshd: root@pts/16
11430 root      5436 S    sshd: root@pts/17
12604 root      5436 S    sshd: root@pts/18
13772 root      5436 S    sshd: root@pts/19
15082 root      5436 S    sshd: root@pts/20
16262 root      5436 S    sshd: root@pts/21
17506 root      5436 S    sshd: root@pts/22
18634 root      5436 S    sshd: root@pts/23
19880 root      5436 S    sshd: root@pts/24
21048 root      5436 S    sshd: root@pts/25
22104 root      5436 S    sshd: root@pts/26
22136 root      5436 S    sshd: root@pts/3
23350 root      5436 S    sshd: root@pts/27
24652 root      5436 S    sshd: root@pts/28
25763 root      5436 S    sshd: root@pts/4
29312 root      5436 S    sshd: root@pts/5
29361 root      5436 S    sshd: root@pts/29
30349 root      5436 S    sshd: root@pts/30
31597 root      5436 S    sshd: root@pts/31

Most of those devices are in a running state. Here’s a subset of the results of the command looking at all states:

sshd@54-10.240.4.33:22222-52.4.252.97:46206.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:46206)                                      
sshd@55-10.240.4.33:22222-52.4.252.97:59650.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:59650)                                      
sshd@56-10.240.4.33:22222-52.4.252.97:45518.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:45518)                                      
sshd@57-10.240.4.33:22222-52.4.252.97:41264.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:41264)                                      
sshd@58-10.240.4.33:22222-52.4.252.97:53552.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:53552)                                      
sshd@59-10.240.4.33:22222-52.4.252.97:38088.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:38088)                                      
sshd@60-10.240.4.33:22222-52.4.252.97:51616.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:51616)                                      
sshd@61-10.240.4.33:22222-52.4.252.97:36194.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:36194)                                      
sshd@62-10.240.4.33:22222-52.4.252.97:50124.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:50124)                                      
sshd@63-10.240.4.33:22222-52.4.252.97:36286.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:36286)                                      
sshd@64-10.240.4.33:22222-52.4.252.97:60832.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:60832)                                      
sshd@65-10.240.4.33:22222-52.4.252.97:44612.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:44612)                                      
sshd@66-10.240.4.33:22222-52.4.252.97:57362.service                                          loaded activating auto-restart OpenSSH Per-Connection Daemon (52.4.252.97:57362)                                      
sshd@67-10.240.4.33:22222-52.4.252.97:60648.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:60648)                                      
sshd@68-10.240.4.33:22222-52.4.252.97:45608.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:45608)                                      
sshd@69-10.240.4.33:22222-52.4.252.97:58716.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:58716)                                      
sshd@70-10.240.4.33:22222-52.4.252.97:44120.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:44120)                                      
sshd@71-10.240.4.33:22222-52.4.252.97:57374.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:57374)                                      
sshd@72-10.240.4.33:22222-52.4.252.97:41046.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:41046)                                      
sshd@73-10.240.4.33:22222-52.4.252.97:52734.service                                          loaded activating auto-restart OpenSSH Per-Connection Daemon (52.4.252.97:52734)                                      
sshd@74-10.240.4.33:22222-52.4.252.97:54298.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:54298)                                      
sshd@75-10.240.4.33:22222-52.4.252.97:39206.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:39206)                                      
sshd@76-10.240.4.33:22222-52.4.252.97:53370.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:53370)                                      
sshd@77-10.240.4.33:22222-52.4.252.97:37890.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:37890)                                      
sshd@78-10.240.4.33:22222-52.4.252.97:52116.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:52116)                                      
sshd@79-10.240.4.33:22222-52.4.252.97:37636.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:37636)                                      
sshd@80-10.240.4.33:22222-52.4.252.97:50622.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:50622)                                      
sshd@81-10.240.4.33:22222-52.4.252.97:50830.service                                          loaded activating auto-restart OpenSSH Per-Connection Daemon (52.4.252.97:50830)                                      
sshd@82-10.240.4.33:22222-52.4.252.97:35902.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:35902)                                      
sshd@83-10.240.4.33:22222-52.4.252.97:44896.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:44896)                                      
sshd@84-10.240.4.33:22222-52.4.252.97:48950.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:48950)                                      
sshd@85-10.240.4.33:22222-52.4.252.97:33530.service                                          loaded active     running      OpenSSH Per-Connection Daemon (52.4.252.97:33530)

I’m quite certain we don’t have that many connections open.

10

Hi
Did you try the one-liner fix? Are you getting the above logs after the fix, or is this before that?

12

Hi @0xff just a note on this, the increased number of dangling SSHd process has been fixed in balenaOS 2.47.1 and above, so moving to 2.48 (due to be released this week or so) should sort that problem out completely.

13

Thanks - I noticed that 2.48 was out. We’ll test it out.

14

Let us know how you get on.

James.