Device goes offline but is online and pinging

Product support balenaOS
1

We’re working with a Raspberry Pi 4 in development mode with Balena Cloud. We’ve changed it to local-mode and back a few times, using both the CLI and the cloud dashboard. Now the device never shows it’s back online again, so it doesn’t update any more. Commands like deploy and push don’t have any effect, showing error:

Some services failed to build:
	db: (HTTP code 500) server error - Get https://registry-1.docker.io/v2/: dial tcp 1.2.3.4:443: connect: connection refused 
	jellyfin: (HTTP code 500) server error - Get https://registry-1.docker.io/v2/: dial tcp 1.2.3.4:443: connect: connection refused 
	filebrowser: (HTTP code 500) server error - Get https://registry-1.docker.io/v2/: dial tcp 1.2.3.4:443: connect: connection refused 
	code: (HTTP code 500) server error - Get https://ghcr.io/v2/: dial tcp 1.2.3.4:443: connect: connection refused 
	portal: (HTTP code 500) server error - Get https://registry-1.docker.io/v2/: dial tcp 1.2.3.4:443: connect: connection refused 
	metube: (HTTP code 500) server error - Get https://registry-1.docker.io/v2/: dial tcp 1.2.3.4:443: connect: connection refused

But when we ssh into the device it’s online, pinging 8.8.8.8 normally.

2

We found out this was a dns error using nslookup. Everything returned with the 1.2.3.4 address.

We used configizer to set DNSSERVERS="8.8.8.8" on the device’s config.json file.

After a reboot the device was again online and updating.

UPDATE

But we’re still having connection problems that we can’t trace to our Internet connection. Here are some common logs that journalctl spits out::


balenad[1371]: time="2021-10-16T14:59:45.197662037Z" level=warning msg="Error getting v2 registry: Get https://registry2.balena-cloud.com/v2/: dial tcp 54.166.87.228:443: connect: connection refused"

balenad[1371]: time="2021-10-16T14:59:45.197807182Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry2.balena-cloud.com/v2/: dial tcp 54.166.87.228:443: connect: connection refused"

balenad[1371]: time="2021-10-16T14:59:45.197977420Z" level=error msg="Handler for POST /images/create returned error: Get https://registry2.balena-cloud.com/v2/: dial tcp 54.166.87.228:443: connect: connection refused"

3d8d94b4953c[1371]: [event]   Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/b3069c84e30c1baa86d9277f76485e9b@sha256:8f8bec2d36c48a93e9ebbfe3fd1f26399f279da4615a23f55517008e7dc2b4fa","appId":1872644,"serviceId":1264730,"serviceName":"nginx-proxy","imageId":4139115,"releaseId":1947971,"dependent":0}}

balena-supervisor[1895]: [event]   Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/b3069c84e30c1baa86d9277f76485e9b@sha256:8f8bec2d36c48a93e9ebbfe3fd1f26399f279da4615a23f55517008e7dc2b4fa","appId":1872644,"serviceId":1264730,"serviceName":"nginx-proxy","imageId":4139115,"releaseId":1947971,"dependent":0}}

3d8d94b4953c[1371]: [event]   Event: Image download error {"error":{"message":"(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 54.166.87.228:443: connect: connection refused ","stack":"Error: (HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 54.166.87.228:443: connect: connection refused \n    at /usr/src/app/dist/app.js:10:2333796\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:10:2333683)\n    at IncomingMessage.emit (events.js:322:22)\n    at endReadableNT (_stream_readable.js:1187:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"image":{"name":"registry2.balena-cloud.com/v2/d886532ea1b99adbb0237c2d6f2f06e8@sha256:17e463649c4708feea31945d866bafde3fb448c6e715b6c6bc9f1b0f865ec7fc","appId":1872644,"serviceId":1262445,"serviceName":"metube","imageId":4139120,"releaseId":1947971,"dependent":0}}

balena-supervisor[1895]: [event]   Event: Image download error {"error":{"message":"(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 54.166.87.228:443: connect: connection refused ","stack":"Error: (HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 54.166.87.228:443: connect: connection refused \n    at /usr/src/app/dist/app.js:10:2333796\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:10:2333683)\n    at IncomingMessage.emit (events.js:322:22)\n    at endReadableNT (_stream_readable.js:1187:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"image":{"name":"registry2.balena-cloud.com/v2/d886532ea1b99adbb0237c2d6f2f06e8@sha256:17e463649c4708feea31945d866bafde3fb448c6e715b6c6bc9f1b0f865ec7fc","appId":1872644,"serviceId":1262445,"serviceName":"metube","imageId":4139120,"releaseId":1947971,"dependent":0}}

3d8d94b4953c[1371]: [event]   Event: Image download error {"error":{"message":""},"image":{"name":"registry2.balena-cloud.com/v2/f4772ae9b83aba9f630b89b4c301258b@sha256:ed4b56a834f736c36c37a82d309a1091ddd693448aa557ed453e849ff88c9bb4","appId":1872644,"serviceId":1261893,"serviceName":"portal","imageId":4139118,"releaseId":1947971,"dependent":0}}

balena-supervisor[1895]: [event]   Event: Image download error {"error":{"message":""},"image":{"name":"registry2.balena-cloud.com/v2/f4772ae9b83aba9f630b89b4c301258b@sha256:ed4b56a834f736c36c37a82d309a1091ddd693448aa557ed453e849ff88c9bb4","appId":1872644,"serviceId":1261893,"serviceName":"portal","imageId":4139118,"releaseId":1947971,"dependent":0}}

balenad[1371]: time="2021-10-16T14:59:45.224557409Z" level=warning msg="Error getting v2 registry: Get https://registry2.balena-cloud.com/v2/: dial tcp 3.223.56.254:443: connect: connection refused"

balenad[1371]: time="2021-10-16T14:59:45.224720313Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry2.balena-cloud.com/v2/: dial tcp 3.223.56.254:443: connect: connection refused"

balenad[1371]: time="2021-10-16T14:59:45.224899644Z" level=error msg="Handler for POST /images/create returned error: Get https://registry2.balena-cloud.com/v2/: dial tcp 3.223.56.254:443: connect: connection refused"

3d8d94b4953c[1371]: [event]   Event: Image download error {"error":{"message":"(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 3.223.56.254:443: connect: connection refused ","stack":"Error: (HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 3.223.56.254:443: connect: connection refused \n    at /usr/src/app/dist/app.js:10:2333796\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:10:2333683)\n    at IncomingMessage.emit (events.js:322:22)\n    at endReadableNT (_stream_readable.js:1187:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"image":{"name":"registry2.balena-cloud.com/v2/b3069c84e30c1baa86d9277f76485e9b@sha256:8f8bec2d36c48a93e9ebbfe3fd1f26399f279da4615a23f55517008e7dc2b4fa","appId":1872644,"serviceId":1264730,"serviceName":"nginx-proxy","imageId":4139115,"releaseId":1947971,"dependent":0}}

balena-supervisor[1895]: [event]   Event: Image download error {"error":{"message":"(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 3.223.56.254:443: connect: connection refused ","stack":"Error: (HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: dial tcp 3.223.56.254:443: connect: connection refused \n    at /usr/src/app/dist/app.js:10:2333796\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:10:2333683)\n    at IncomingMessage.emit (events.js:322:22)\n    at endReadableNT (_stream_readable.js:1187:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)"},"image":{"name":"registry2.balena-cloud.com/v2/b3069c84e30c1baa86d9277f76485e9b@sha256:8f8bec2d36c48a93e9ebbfe3fd1f26399f279da4615a23f55517008e7dc2b4fa","appId":1872644,"serviceId":1264730,"serviceName":"nginx-proxy","imageId":4139115,"releaseId":1947971,"dependent":0}}

3d8d94b4953c[1371]: [warn]    Ignoring unsupported or unknown compose fields: containerName

balena-supervisor[1895]: [warn]    Ignoring unsupported or unknown compose fields: containerName

3d8d94b4953c[1371]: [error]   Scheduling another update attempt in 128000ms due to failure:  Error: Failed to apply state transition steps.  Steps:["fetch","fetch","fetch"]

balena-supervisor[1895]: [error]   Scheduling another update attempt in 128000ms due to failure:  Error: Failed to apply state transition steps.  Steps:["fetch","fetch","fetch"]

balena-supervisor[1895]: [error]         at fn (/usr/src/app/dist/app.js:6:8594)

3d8d94b4953c[1371]: [error]         at fn (/usr/src/app/dist/app.js:6:8594)

3d8d94b4953c[1371]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch","fetch","fetch"]

balena-supervisor[1895]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch","fetch","fetch"]

balena-supervisor[1895]: [error]         at fn (/usr/src/app/dist/app.js:6:8594)

3d8d94b4953c[1371]: [error]         at fn (/usr/src/app/dist/app.js:6:8594)

openvpn[1383]: Sat 
E: the current --script-security setting may allow this configuration to call user-defined scripts

openvpn[1383]: Sat 
/UDP: Preserving recently used remote address: [AF_INET6]2600:1f18:6600:7f02:3ef5:d239:9bb4:6036:443

openvpn[1383]: Sat 
ket Buffers: R=[131072->131072] S=[16384->16384]

openvpn[1383]: Sat 
empting to establish TCP connection with [AF_INET6]2600:1f18:6600:7f02:3ef5:d239:9bb4:6036:443 [nonblock]

openvpn[1383]: Sat 
: connect to [AF_INET6]2600:1f18:6600:7f02:3ef5:d239:9bb4:6036:443 failed: Permission denied

openvpn[1383]: Sat 
USR1[connection failed(soft),init_instance] received, process restarting

openvpn[1383]: Sat 
tart pause, 10 second(s)

balena-supervisor[1895]: [error]   LogBackend: unexpected error: Error: connect ECONNREFUSED 3.226.114.49:443

balena-supervisor[1895]: [error]         at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)

3d8d94b4953c[1371]: [error]   LogBackend: unexpected error: Error: connect ECONNREFUSED 3.226.114.49:443

3d8d94b4953c[1371]: [error]         at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16)

openvpn[1383]: Sat 
E: the current --script-security setting may allow this configuration to call user-defined scripts

openvpn[1383]: Sat 
/UDP: Preserving recently used remote address: [AF_INET]35.169.89.252:443

openvpn[1383]: Sat 
ket Buffers: R=[131072->131072] S=[16384->16384]

openvpn[1383]: Sat 
empting to establish TCP connection with [AF_INET]35.169.89.252:443 [nonblock]

openvpn[1383]: Sat 
: connect to [AF_INET]35.169.89.252:443 failed: Connection refused

openvpn[1383]: Sat 
USR1[connection failed(soft),init_instance] received, process restarting

openvpn[1383]: Sat 
tart pause, 20 second(s)
3

We’re almost convinced this might be a problem with our own network.

1 Like
4

Hello @luandro could you please share your docker-compose to see if we find something there to help you?

BTW could you please confirm how the device is connected to the Internet?

7

Hey @mpous, it’s connected thru cable to a mesh node. And that’s probably the problem. We take care of Brazil’s largest community mesh network. We’re experimenting with it all the time, so the possibility of having weird network problems is always high.

The stack we were trying to run was Jurebaba.

8

Hello @luandro could you please confirm if this is working now? :slight_smile:

10

This weekend we deployed the Guarita project (a fork of balena-pihole) to 5 different remote indigenous communities here in Brasil. We had a similar problem with one of the Raspberry Pi 3s, which persisted even after formatting the sd card. After changing the Pi everything worked fine. Makes us think there might be some strange hardware issue involved.

So yes @mpous it’s working now. But we still see similar strange behavior at times. We’ll keep tracking the issue to understand where it originates. Thanks for the support!

1 Like
11

First of all congratulations for installing this @luandro it’s amazing!

Keep us updated with these issues that you are finding!

Let’s stay connected

1 Like