Run your private The Things Stack network server on a Raspberry Pi with balena

Please use the thread below to discuss the related blog post:

https://balena.io/blog/run-your-private-the-things-stack-network-server-on-a-raspberry-pi-with-balena/

3 Likes

Thank you for your message Andrew! Looking forward to see how people use this :slight_smile:

Feel free to post your messages here.

Hi Folks!

I use “The Things Stack network server on a Raspberry Pi with balena” and so far everything works perfectly!
One question: Is it possible to update The Things Stack? The current version is v3.15.0. Version v3.13.1 is used at balena. That would be important to stay up to date.

Regards
Martin

1 Like

Hey Martin i think it’s possible but i haven’t tested myself.

Try to update this line the-things-stack-balena/Dockerfile.template at 5038bb91cd0d5eaf56cb256211a7911108ee8705 · xoseperez/the-things-stack-balena · GitHub

use

FROM thethingsnetwork/lorawan-stack:3.15.0-armv7

this should work. Could you please test and let us know?

Thanks

Hi Marc.

Thank you for the feedback.
But I am not sure if you understood me correctly. I want to update a running (The Things Stack network server) system. You can’t do that with your description, right?

Regards

@loeffma if you clone the repo on your computer, change that line and deploy using balena CLI balena push to your fleet. Did you try this?

Could you please try and let us know if this works? :slight_smile:

I clone the repo on my computer, change the line and push it to my fleet. This process worked. But after that, the-things-stack doesent´t run any more. See the Screenshot of the Logs…

After that, I push the original repo and than the system ran again without any errors.
Do you have any suggestions for a solution?

I found the solution myself :wink:
From version 3.13.0.x to 3.14.x there was a change in the database.
Therefore you have to migrate the database.
So I uploaded version 3.15.0 again and then executed the command “ttn-lw-stack is-db migrate” in the terminal. And voilà! 3.15.0 is online!

1 Like

Good to hear that @loeffma do you think you can PR the original repo? :slight_smile:

Thanks!

Hi,

Big thanks to this Tutorial!
For my master thesis I started working with this balena stack.

Main goal is to create a own LoRaWAN infrastructure (GW + TTS Server) which should work offline. This means that it will work at places where no internetconnection is.

I added a nodered service to this stack and to configure everything I worked online. All works fine. Communication with some nodes in the field works fine. Going offline without a reset also works fine. Nodes still communicate with the private Net.

But after reseting the GW and TTS Server (RaspberryPi with RAK833 Hat) the system won’t work anymore. Nodered server is still available but the tts server is not able to run and the nodes are also not able to cummunicate with the server. The tts console is also not able to load (Static IP).

Connecting the RaspberryPi with RAK833 again to the internet and restarting the stack service the system works again.

I thing that has to do because the stack is exchaging some key or something like this.

Do you my have an idea how to solve my problem? So that the whole system is able to run offline also after an reset?

Best regards
Corsin Obrist

1 Like

Hello @Balena_Obrist thanks for your message and welcome to the community!

Could you please share more information (logs, etc) when you reset the GW and TTS Server? what did you do? Reset services from balenaCloud? unpower the pi? What do you use?

Hi,

Thanks for this fast reply!

I may have to explain it a little more detail:

I connected the Raspberry with a static IP (192.169.1.111) to my laptop via LAN. The laptop has also a static IP in the same net. No connection to internet via LAN.

Then I connect the Laptop and Raspberry to the same WLAN with internet connection. This I do that I can see the device in the balena dashboard.

So over 192.168.1.111:80 i can access the TTN console. After connection is made the nodes are sending bidirectionally, All is fine. Than I turn off the WLAN.
The TTN console is still running (192.168.1.111) and there is still a bidirectional connection to the nodes over LoRaWAN.

Then I restet the Raspberry via a Power off (plug off powersupply then plug in again). The WLAN is still off so only connection to the Raspberry is over the LAN cable IP:192.168.1.111.

Then entering the 192.168.1.111 in the browser pups on the ttn console for a second an then its offline and the url looks like this: http://oauth/authorize?client_id=console&redirect_uri=%2F%2F%2Fconsole%2Foauth%2Fcallback&response_type=code&state=wkVr8nGkFiOUD5Ez

Over the Balena Dashboard I can not access the Device because of no Internet connection but with balena cli its possible and the nodered server is running on 192.168.1.111:8080 and I can access it over the browser.

How can I access the logs with balena shh? Where do I find the log files of the stack service?

1 Like

@Balena_Obrist did you configured the static ip address on your Pi?

i imagine that you already checked this → Run your private The Things Stack network server on a Raspberry Pi with balena

You can find the documentation here → SSH access - Balena Documentation

Yes i did the whole tutorial and as I said it works fine until you reboot when the device is not connected to the internet.

Here are the enabled services after boot:

mnt-sysroot-inactive.automount                                   enabled
var-lib-docker.mount                                             enabled
config-json.path                                                 enabled
autovt@.service                                                  enabled
avahi-daemon.service                                             enabled
balena-device-uuid.service                                       enabled
balena-engine.service                                            enabled
balena-hostname.service                                          enabled
balena-net-config.service                                        enabled
balena-net-connectivity-wait.service                             enabled
balena-ntp-config.service                                        enabled
balena-proxy-config.service                                      enabled
balena-supervisor.service                                        enabled
balena.service                                                   enabled
bind-etc-balena-supervisor.service                               enabled
bind-etc-docker.service                                          enabled
bind-etc-hostname.service                                        enabled
bind-etc-NetworkManager-conf.d.service                           enabled
bind-etc-NetworkManager-system-connections.service               enabled
bind-etc-openvpn.service                                         enabled
bind-etc-ssh-hostkeys.service                                    enabled
bind-etc-ssl-certs.service                                       enabled
bind-etc-udev-rules.d.service                                    enabled
bind-home-root-.docker.service                                   enabled
bind-home-root-.rnd.service                                      enabled
bind-home-root-.ssh.service                                      enabled
bind-usr-share-ca-certificates-balena.service                    enabled
bind-var-lib-bluetooth.service                                   enabled
bind-var-lib-chrony.service                                      enabled
bind-var-lib-NetworkManager.service                              enabled
bind-var-lib-systemd.service                                     enabled
bluetooth.service                                                enabled
chronyd.service                                                  enabled
dbus-org.bluez.service                                           enabled
dbus-org.freedesktop.Avahi.service                               enabled
dbus-org.freedesktop.ModemManager1.service                       enabled
dbus-org.freedesktop.nm-dispatcher.service                       enabled
dnsmasq.service                                                  enabled
extract-balena-ca.service                                        enabled
fake-hwclock.service                                             enabled
getty@.service                                                   enabled
hciuart.service                                                  enabled
ip6tables.service                                                enabled
iptables.service                                                 enabled
ModemManager.service                                             enabled
NetworkManager-dispatcher.service                                enabled
NetworkManager-wait-online.service                               enabled
NetworkManager.service                                           enabled
openvpn-resin.service                                            enabled
openvpn.service                                                  enabled
os-config-devicekey.service                                      enabled
os-config.service                                                enabled
os-networkmanager.service                                        enabled
os-sshkeys.service                                               enabled
os-udevrules.service                                             enabled
redsocks.service                                                 enabled
resin-boot.service                                               enabled
resin-data.service                                               enabled
resin-filesystem-expand.service                                  enabled
resin-init.service                                               enabled
resin-persistent-logs.service                                    enabled
resin-state-reset.service                                        enabled
resin-state.service                                              enabled
resin-supervisor.service                                         enabled
rngd.service                                                     enabled
rollback-altboot.service                                         enabled
rollback-clear-bootcount.service                                 enabled
rollback-health.service                                          enabled
sshdgenkeys.service                                              enabled
systemd-remount-fs.service                                       enabled-runtime
timeinit-buildtime.service                                       enabled
timeinit-rtc.service                                             enabled
update-balena-supervisor.service                                 enabled
update-resin-supervisor.service                                  enabled
var-volatile-cache.service                                       enabled
var-volatile-lib.service                                         enabled
var-volatile-spool.service                                       enabled
var-volatile-srv.service                                         enabled
avahi-daemon.socket                                              enabled
balena-host.socket                                               enabled
sshd.socket                                                      enabled
dev-zram0.swap                                                   enabled
ctrl-alt-del.target                                              enabled
reboot.target                                                    enabled
remote-fs.target                                                 enabled
runlevel6.target                                                 enabled
fake-hwclock-update.timer                                        enabled
os-config.timer                                                  enabled
periodic-vacuum-logs.timer                                       enabled
update-balena-supervisor.timer                                   enabled
update-resin-supervisor.timer                                    enabled

These services should be running:

These are the logs of the mainOS:

root@c1b375b:~# journalctl --all -n 200 -u balena
-- Logs begin at Fri 2021-11-26 07:00:01 UTC, end at Fri 2021-11-26 07:08:38 UTC. --
Nov 26 07:02:56 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:02:56 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:02:56 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport:>Nov 26 07:02:56 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transpor>Nov 26 07:03:05 c1b375b c747aee5e925[1470]: [event]   Event: Device state report failure {"error":{"message":""}}
Nov 26 07:03:17 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport:>Nov 26 07:03:17 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transpor>Nov 26 07:03:26 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:03:26 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:03:39 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport:>Nov 26 07:03:39 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transpor>Nov 26 07:03:48 c1b375b 553609eb3bc1[1470]:   INFO Request handled                          duration=100µs http.method=GET http.path=/healthz/live http.status=200 namespace=web peer.address=127.0.0.1:56772 request_i>Nov 26 07:03:48 c1b375b bd2739a32467[1470]: Using MODEL: SX1301
Nov 26 07:03:48 c1b375b bd2739a32467[1470]: Server: wss://localhost:8887
Nov 26 07:03:48 c1b375b bd2739a32467[1470]: Resetting gateway concentrator on GPIO 17
Nov 26 07:03:50 c1b375b balenad[1535]: time="2021-11-26T07:03:50.452166995Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/fae98e3206e8af945ed0785ff99e267152628938f0a4aadb95f7c>Nov 26 07:03:51 c1b375b bd2739a32467[1470]: A station with pid=37 is still running (use -f to take over)
Nov 26 07:03:52 c1b375b balenad[1535]: time="2021-11-26T07:03:52.789155796Z" level=info msg="shim reaped" id=bd2739a32467ef680bcfa4fbb4e772895b717a58d3660edd7553ba88d5eeaa48
Nov 26 07:03:52 c1b375b balenad[1470]: time="2021-11-26T07:03:52.801813817Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 26 07:03:52 c1b375b c747aee5e925[1470]: [event]   Event: Service exit {"service":{"appId":1885414,"serviceId":1321902,"serviceName":"basicstation","commit":"4b571cb99698b8a8b9020dc6cb5a1c54","releaseId":1992516}}Nov 26 07:03:52 c1b375b balenad[1535]: time="2021-11-26T07:03:52.944350484Z" level=info msg="shim reaped" id=b8b1ef7ca418d601f131e29341c8e61f76d63061651476aac451e8e3434f3691
Nov 26 07:03:52 c1b375b balenad[1470]: time="2021-11-26T07:03:52.947587880Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 26 07:03:53 c1b375b balenad[1535]: time="2021-11-26T07:03:53.125309234Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/a44ac34ae0f0a878ae72d024df009568f41bf49bd7a439f241758>Nov 26 07:03:54 c1b375b bd2739a32467[1470]: Gateway EUI: b827ebfffe603dc4
Nov 26 07:03:54 c1b375b c747aee5e925[1470]: [event]   Event: Service restart {"service":{"appId":1885414,"serviceId":1321902,"serviceName":"basicstation","commit":"4b571cb99698b8a8b9020dc6cb5a1c54","releaseId":19925>Nov 26 07:03:54 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:03:54 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:04:01 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport:>lines 1-29...skipping...
-- Logs begin at Fri 2021-11-26 07:00:01 UTC, end at Fri 2021-11-26 07:08:38 UTC. --
Nov 26 07:02:56 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:02:56 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:02:56 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:02:56 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:03:05 c1b375b c747aee5e925[1470]: [event]   Event: Device state report failure {"error":{"message":""}}
Nov 26 07:03:17 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:03:17 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:03:26 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:03:26 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:03:39 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:03:39 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:03:48 c1b375b 553609eb3bc1[1470]:   INFO Request handled                          duration=100µs http.method=GET http.path=/healthz/live http.status=200 namespace=web peer.address=127.0.0.1:56772 request_id=01FNDGXB104DEV2ZQNP>
Nov 26 07:03:48 c1b375b bd2739a32467[1470]: Using MODEL: SX1301
Nov 26 07:03:48 c1b375b bd2739a32467[1470]: Server: wss://localhost:8887
Nov 26 07:03:48 c1b375b bd2739a32467[1470]: Resetting gateway concentrator on GPIO 17
Nov 26 07:03:50 c1b375b balenad[1535]: time="2021-11-26T07:03:50.452166995Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/fae98e3206e8af945ed0785ff99e267152628938f0a4aadb95f7c3a340c2d84d.sock debu>
Nov 26 07:03:51 c1b375b bd2739a32467[1470]: A station with pid=37 is still running (use -f to take over)
Nov 26 07:03:52 c1b375b balenad[1535]: time="2021-11-26T07:03:52.789155796Z" level=info msg="shim reaped" id=bd2739a32467ef680bcfa4fbb4e772895b717a58d3660edd7553ba88d5eeaa48
Nov 26 07:03:52 c1b375b balenad[1470]: time="2021-11-26T07:03:52.801813817Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 26 07:03:52 c1b375b c747aee5e925[1470]: [event]   Event: Service exit {"service":{"appId":1885414,"serviceId":1321902,"serviceName":"basicstation","commit":"4b571cb99698b8a8b9020dc6cb5a1c54","releaseId":1992516}}
Nov 26 07:03:52 c1b375b balenad[1535]: time="2021-11-26T07:03:52.944350484Z" level=info msg="shim reaped" id=b8b1ef7ca418d601f131e29341c8e61f76d63061651476aac451e8e3434f3691
Nov 26 07:03:52 c1b375b balenad[1470]: time="2021-11-26T07:03:52.947587880Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 26 07:03:53 c1b375b balenad[1535]: time="2021-11-26T07:03:53.125309234Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/a44ac34ae0f0a878ae72d024df009568f41bf49bd7a439f241758d72adffefb7.sock debu>
Nov 26 07:03:54 c1b375b bd2739a32467[1470]: Gateway EUI: b827ebfffe603dc4
Nov 26 07:03:54 c1b375b c747aee5e925[1470]: [event]   Event: Service restart {"service":{"appId":1885414,"serviceId":1321902,"serviceName":"basicstation","commit":"4b571cb99698b8a8b9020dc6cb5a1c54","releaseId":1992516}}
Nov 26 07:03:54 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:03:54 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:04:01 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:04:02 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:04:24 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:04:24 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:04:25 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:04:26 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:04:35 c1b375b c747aee5e925[1470]: [event]   Event: Device state report failure {"error":{"message":""}}
Nov 26 07:04:49 c1b375b 553609eb3bc1[1470]:   INFO Request handled                          duration=100µs http.method=GET http.path=/healthz/live http.status=200 namespace=web peer.address=127.0.0.1:56774 request_id=01FNDGZ6QW70JT9XJAJ>
Nov 26 07:04:50 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:04:50 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:04:50 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:04:52 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:04:54 c1b375b bd2739a32467[1470]: Using MODEL: SX1301
Nov 26 07:04:54 c1b375b bd2739a32467[1470]: Server: wss://localhost:8887
Nov 26 07:04:54 c1b375b bd2739a32467[1470]: Resetting gateway concentrator on GPIO 17
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.869 [SYS:INFO] Logging     : stderr (maxsize=10000000, rotate=3)
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.869 [SYS:INFO] Station Ver : 2.0.5(rpi/std) 2021-11-25 09:37:25
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.871 [SYS:INFO] Package Ver : 1.0.0
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.871 [SYS:INFO] proto EUI   : 0:b827:eb60:3dc4   (/sys/class/net/eth0/address)
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.871 [SYS:INFO] prefix EUI  : ::1        (builtin)
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.871 [SYS:INFO] Station EUI : b827:ebff:fe60:3dc4
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.871 [SYS:INFO] Station home: ./ (builtin)
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.872 [SYS:INFO] Station temp: /var/tmp/  (builtin)
Nov 26 07:04:57 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:57.872 [SYS:WARN] Station in NO-CUPS mode
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.074 [TCE:INFO] Starting TC engine
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.078 [any:INFO] ./tc.trust:
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: cert. version     : 3
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: serial number     : 02:5E:9E:B3:F3:57:86:0B:A4:24:81:3F:85:83:FA:6A:61:F4:CD:48
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: issuer name       : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: subject name      : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: issued  on        : 2021-11-25 15:10:00
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: expires on        : 2026-11-24 15:10:00
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: signed using      : RSA with SHA-256
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: RSA key size      : 2048 bits
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: basic constraints : CA=true, max_p
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.086 [TCE:INFO] Connecting to INFOS: wss://localhost:8887
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.317 [AIO:INFO] TLS server certificate verification failed: The certificate is not correctly signed by the trusted CA
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.317 [AIO:DEBU] [3] WS connection shutdown...
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.318 [TCE:INFO] INFOS reconnect backoff 0s (retry 0)
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.318 [any:INFO] ./tc.trust:
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: cert. version     : 3
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: serial number     : 02:5E:9E:B3:F3:57:86:0B:A4:24:81:3F:85:83:FA:6A:61:F4:CD:48
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: issuer name       : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: subject name      : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: issued  on        : 2021-11-25 15:10:00
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: expires on        : 2026-11-24 15:10:00
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: signed using      : RSA with SHA-256
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: RSA key size      : 2048 bits
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: basic constraints : CA=true, max_p
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.320 [TCE:INFO] Connecting to INFOS: wss://localhost:8887
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.527 [AIO:INFO] TLS server certificate verification failed: The certificate is not correctly signed by the trusted CA
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.527 [AIO:DEBU] [3] WS connection shutdown...
Nov 26 07:04:58 c1b375b bd2739a32467[1470]: 2021-11-26 07:04:58.527 [TCE:INFO] INFOS reconnect backoff 10s (retry 1)
Nov 26 07:05:05 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:05:05 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:08.529 [any:INFO] ./tc.trust:
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: cert. version     : 3
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: serial number     : 02:5E:9E:B3:F3:57:86:0B:A4:24:81:3F:85:83:FA:6A:61:F4:CD:48
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: issuer name       : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: subject name      : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: issued  on        : 2021-11-25 15:10:00
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: expires on        : 2026-11-24 15:10:00
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: signed using      : RSA with SHA-256
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: RSA key size      : 2048 bits
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: basic constraints : CA=true, max_p
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:08.531 [TCE:INFO] Connecting to INFOS: wss://localhost:8887
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:08.756 [AIO:INFO] TLS server certificate verification failed: The certificate is not correctly signed by the trusted CA
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:08.756 [AIO:DEBU] [3] WS connection shutdown...
Nov 26 07:05:08 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:08.756 [TCE:INFO] INFOS reconnect backoff 20s (retry 2)
Nov 26 07:05:21 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:05:23 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:05:23 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:05:23 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:28.757 [any:INFO] ./tc.trust:
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: cert. version     : 3
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: serial number     : 02:5E:9E:B3:F3:57:86:0B:A4:24:81:3F:85:83:FA:6A:61:F4:CD:48
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: issuer name       : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: subject name      : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: issued  on        : 2021-11-25 15:10:00
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: expires on        : 2026-11-24 15:10:00
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: signed using      : RSA with SHA-256
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: RSA key size      : 2048 bits
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: basic constraints : CA=true, max_p
Nov 26 07:05:28 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:28.759 [TCE:INFO] Connecting to INFOS: wss://localhost:8887
Nov 26 07:05:29 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:28.987 [AIO:INFO] TLS server certificate verification failed: The certificate is not correctly signed by the trusted CA
Nov 26 07:05:29 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:28.987 [AIO:DEBU] [3] WS connection shutdown...
Nov 26 07:05:29 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:28.987 [TCE:INFO] INFOS reconnect backoff 30s (retry 3)
Nov 26 07:05:44 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:05:44 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:05:50 c1b375b 553609eb3bc1[1470]:   INFO Request handled                          duration=200µs http.method=GET http.path=/healthz/live http.status=200 namespace=web peer.address=127.0.0.1:56792 request_id=01FNDH12E95AZB496J3>
Nov 26 07:05:51 c1b375b c747aee5e925[1470]: [api]     GET /v1/healthy 200 - 30.488 ms
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:58.989 [any:INFO] ./tc.trust:
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: cert. version     : 3
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: serial number     : 02:5E:9E:B3:F3:57:86:0B:A4:24:81:3F:85:83:FA:6A:61:F4:CD:48
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: issuer name       : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: subject name      : C=ES, ST=Catalunya, L=Barcelona, O=TTN Catalunya, CN=TTN Catalunya CA
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: issued  on        : 2021-11-25 15:10:00
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: expires on        : 2026-11-24 15:10:00
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: signed using      : RSA with SHA-256
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: RSA key size      : 2048 bits
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: basic constraints : CA=true, max_p
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:58.991 [TCE:INFO] Connecting to INFOS: wss://localhost:8887
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:59.199 [AIO:INFO] TLS server certificate verification failed: The certificate is not correctly signed by the trusted CA
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:59.199 [AIO:DEBU] [3] WS connection shutdown...
Nov 26 07:05:59 c1b375b bd2739a32467[1470]: 2021-11-26 07:05:59.199 [TCE:INFO] INFOS reconnect backoff 40s (retry 4)
Nov 26 07:06:05 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:06:05 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
Nov 26 07:06:06 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {cp.packetbroker.net:443 cp.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing >
Nov 26 07:06:06 c1b375b 553609eb3bc1[1470]:   WARN [core]grpc: addrConn.createTransport failed to connect to {iam.packetbroker.net:443 iam.packetbroker.net:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialin>
Nov 26 07:06:21 c1b375b c747aee5e925[1470]: [error]   LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com
Nov 26 07:06:21 c1b375b c747aee5e925[1470]: [error]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:64:26)
reconnect backoff 60s (retry 7)

Best regards
Corsin Obrist

Dear Marc,

Did my previous reply with some of the output log helped you, to understand what might goes wrong on my system?

Best regards
Corsin

Hello @Balena_Obrist sorry for my late answer!

tbh i’m not sure where the problem could come from. I have been reading a similar issue reported here and it looks like the solution was open a port. However here i’m not sure.

Next week i will try to reproduce on my device.

Hello @mpous

After several tests, I still think the Stack needs some internet connection at the beginning for some key-exchange or something like this,

Did you found some time to try it at your place?

Best regards
Corsin Obrist

In a couple of weeks i will test this myself @Balena_Obrist

let me know if you can discover anything as well :slight_smile:

Hi @mpous,

I successfully created two different balena fleets: one private the things stack server (tts-lns) and one tts gateway (basicstation). they run on two separate raspberry pies within the same local network, because later I would like to add more gateways within the building that all connect to this one private server.

I first testet the basicstation device, which connected fine to the public eu1 server. I then created a new key in the private tts-lns dashboard and copied it to the TC-KEY device variable so that the basicstation can connect directly to the private server (I also changed the TC_URI to the local IP address). Unfortunately, I’m getting certificate errors:

 basicstation  cert. version     : 3
 basicstation  serial number     : 44:AF:B0:80:D6:A3:27:BA:89:30:39:86:2E:F8:40:6B
 basicstation  issuer name       : O=Digital Signature Trust Co., CN=DST Root CA X3
 basicstation  subject name      : O=Digital Signature Trust Co., CN=DST Root CA X3
 basicstation  issued  on        : 2000-09-30 21:12:19
 basicstation  expires on        : 2021-09-30 14:01:15
 basicstation  signed using      : RSA with SHA1
 basicstation  RSA key size      : 2048 bits
 basicstation  basic constraints : CA=true
 basicstation  key usage         : Key Cert Sign, CRL Sign
 basicstation  2022-03-02 10:49:12.131 [AIO:INFO] tc has no cert configured - running server auth and client auth with token
 basicstation  2022-03-02 10:55:47.809 [AIO:ERRO] [3] WS connect failed: NET - The connection to the given server / port failed
 basicstation  2022-03-02 10:55:47.809 [AIO:DEBU] [3] WS connection shutdown...
 basicstation  2022-03-02 10:55:47.809 [TCE:ERRO] TC connect failed - URI: wss://192.168.1.138:8887
 basicstation  2022-03-02 10:55:47.809 [TCE:INFO] INFOS reconnect backoff 60s (retry 143)

It looks like the secure websocket connection can’t be established. I tried to change the hosts file on the tts-lns server, but it looks like I don’t have the authorisation to do so. I also tried to use the public device url instead of the local IP address, but as far as I understand only port 80 is open and the wss would need port 8887, right?

How can I fix this certificate issue? Or are my device variable settings just wrong?

By the way, I also tried to change the TTS_SUBJECT_* variables in order to force the tts-lns server to create new certificates, but without luck.

I would really appreciate a hint. Many thanks,
Lukas

1 Like

Hello @Lukas welcome to the balena Community! Thank you for your feedback!

Could you please confirm that you can ping from the basicstation Pi to the TTS Pi on that IP?

Thanks

Hi @mpous,

Thanks for the warm welcome :slightly_smiling_face: Yes, I can confirm that the TTS Pi is pingable from the basicstation Pi.

Did some more digging and saw, that I didn’t add the TC_KEY on the tts-lns side (though, I always though this key is only needed on the basicstation side). But now I get a different error message:

basicstation  key usage         : Key2022-03-04 22:12:03.233 [AIO:INFO] tc has no cert configured - running server auth and client auth with token
basicstation  2022-03-04 22:12:03.235 [TCE:INFO] Connecting to INFOS: wss://192.168.1.138:8887
basicstation  2022-03-04 22:12:03.411 [AIO:INFO] TLS server certificate verification failed: The certificate Common Name (CN) does not match with the expected CN
basicstation  2022-03-04 22:12:03.411 [AIO:DEBU] [3] WS connection shutdown...

Could it be, that I’m missing the TC_TRUST variable on the basicstation side? If so, what to put there?

1 Like