balena ssh <uuid> gives Permission denied (publickey). local IP & putty work for host OS

mahesh1 · February 22, 2020, 7:00pm

hi, i’m using Windows 10. i spent a whole lot of time yesterday in trying to get balena ssh <uuid> to work for device . to no avail; i get:

g_[username]@ssh.balena-devices.com: Permission denied (publickey).

balena devices and balena device <short-uuid> work. balena <local_ip> works. i can putty to the public device name <uuiid>.balena-devices.com:22. i can putty to <local_ip>:22222, but all for the host OS only. i’ve generated an ssh key on my laptop and added it to balena’s keys online. i’ve also added it to a file in the host OS (ugh, i can’t remember the name of it now). i’ve set DEBUG = 1 on my laptop:

set DEBUG = 1

C:\Users\mahes>balena devices
[debug] original argv0="C:\Program Files\balena-cli\bin\\..\client\bin\node.exe" argv=[C:\Program Files\balena-cli\client\bin\node.exe,C:\Program Files\balena-cli\client\bin\run,devices] length=3
ID      UUID    DEVICE NAME DEVICE TYPE  APPLICATION NAME STATUS IS ONLINE SUPERVISOR VERSION OS VERSION           DASHBOARD URL
<id> <short-uuid> <some-words>  raspberry-pi <myAppName>            Idle   false     9.14.0             balenaOS 2.32.0+rev1 https://dashboard.balena-cloud.com/devices/<full-uuid>/summary
<id2> <short-uuid> <other-words>  raspberry-pi <myAppName>           Idle   true      9.14.0             balenaOS 2.32.0+rev1 https://dashboard.balena-cloud.com/devices/<full-uuid>/summary

here’s the full output (--verbose) of trying to connect using balena ssh:

C:\Users\[user]>balena ssh [short-uuid] --verbose
[debug] original argv0="C:\Program Files\balena-cli\bin\\..\client\bin\node.exe" argv=[C:\Program Files\balena-cli\client\bin\node.exe,C:\Program Files\balena-cli\client\bin\run,ssh,[short-uuid],--verbose] length=5
[Debug]   Fetching application by name [short-uuid] (string)
[Debug]   Application not found
[Debug]   Fetching device by UUID [short-uuid] (string)
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug3: Failed to open file:C:/Users/[user]/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolving "ssh.balena-devices.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to ssh.balena-devices.com [54.147.226.224] port 22.
debug1: Connection established.
debug3: Failed to open file:C:/Users/[user]/.ssh/id_rsa error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_rsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_rsa type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_rsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_dsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_dsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ecdsa.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ecdsa-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_ecdsa-cert type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ed25519 error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ed25519.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_ed25519 type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_ed25519-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_xmss.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/[user]/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/[user]/.ssh/id_xmss-cert.pub error:2
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\[user]/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to ssh.balena-devices.com:22 as 'g_[user]h'
debug3: hostkeys_foreach: reading file "/dev/null"
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp256,ssh-dss
debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:g6EWPpcXbcBHTxrPdYezd+9YFd//GPJKiNC8f1ryaaI
debug3: hostkeys_foreach: reading file "/dev/null"
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: hostkeys_foreach: reading file "/dev/null"
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
Warning: Permanently added 'ssh.balena-devices.com,54.147.226.224' (ECDSA) to the list of known hosts.
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug3: unable to connect to pipe \\\\.\\pipe\\openssh-ssh-agent, error: 2
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug2: key: C:\\Users\\[user]/.ssh/id_rsa (0000000000000000)
debug2: key: C:\\Users\\[user]/.ssh/id_dsa (0000000000000000)
debug2: key: C:\\Users\\[user]/.ssh/id_ecdsa (0000000000000000)
debug2: key: C:\\Users\\[user]/.ssh/id_ed25519 (0000000000000000)
debug2: key: C:\\Users\\[user]/.ssh/id_xmss (0000000000000000)
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: C:\\Users\\[user]/.ssh/id_rsa
debug3: no such identity: C:\\Users\\[user]/.ssh/id_rsa: No such file or directory
debug1: Trying private key: C:\\Users\\[user]/.ssh/id_dsa
debug3: no such identity: C:\\Users\\[user]/.ssh/id_dsa: No such file or directory
debug1: Trying private key: C:\\Users\\[user]/.ssh/id_ecdsa
debug3: no such identity: C:\\Users\\[user]/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: C:\\Users\\[user]/.ssh/id_ed25519
debug3: no such identity: C:\\Users\\[user]/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: C:\\Users\\[user]/.ssh/id_xmss
debug3: no such identity: C:\\Users\\[user]/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
g_[user]h@ssh.balena-devices.com: Permission denied (publickey).

C:\Users\[user]>

mahesh1 · February 22, 2020, 8:13pm

i think i was able to log into the public url using putty, but i can’t finish logging in anymore using either putty or using Tera Term. see images below.

anujdeshpande · February 24, 2020, 12:29pm

Hi
Can you confirm that the key files are located in the .ssh folder in your user directory?
AFAIK, the files have to be copied from whatever directory you created the keys in, to the .ssh folder. Any chance that you might have missed that?

mahesh1 · February 26, 2020, 6:19pm

hi, here’s my .ssh folder. is there anything i’ve missed doing on the raspi? thanks, mahesh

 Directory of C:\Users\<USER>\.ssh

02/22/2020  08:42 AM    <DIR>          .
02/22/2020  08:42 AM    <DIR>          ..
05/30/2019  04:30 PM             1,679 github_rsa
05/30/2019  04:30 PM               392 github_rsa.pub
02/22/2020  08:42 AM               227 known_hosts
01/05/2020  06:23 PM               228 known_hosts - Copy *i made this copy*
02/22/2020  03:44 AM             1,679 <user>@<domain>
02/22/2020  03:44 AM               405 <user>@<domain>.pub
               6 File(s)          4,610 bytes

tmigone · February 26, 2020, 6:53pm

Hi, can you share the command you used to generate the keys? It looks like you generated the keys on a different directory and not on .ssh.

tmigone · February 26, 2020, 6:54pm

Wait, did you create the keys on Feb 22nd? That <user>@<domain> keys look suspicious…

mahesh1 · February 29, 2020, 2:26am

hi, i don’t remember what i used to generate the keys. possibly windows’ ssh CLI tool? i may have created them in a different folder and moved them – i’m kinda fuzzy about this. and, yes, i did muck around around Feb 22nd. what should i do? while the web console is serviceable, it has limitations such as buggy rendering of the nano editor interface. thanks!

majorz · March 2, 2020, 12:39pm

See how in the logs the balena ssh tool looks up for keys like ../.ssh/id_rsa but it cannot find those.

It also looks for ~/.ssh/config file which is not present as well. It is a good practice to create ~/.ssh/config file. An example balena section would look like so:

Host ssh.balena-devices.com
  User <USER>
  IdentityFile ~/.ssh/<PRIVATE_KEY>

All in all it looks like your ssh keys are not configured properly in the ~/.ssh folder. For more information you may check our SSH keys documentation:

For general ssh configuration you may check: https://www.ssh.com/ssh/config

mahesh1 · March 30, 2020, 1:13pm

thanks, i’ll do that when i get a chance. we’re a bit understaffed at the moment and working on other parts of the system. stay safe. best, mahesh