Error network timeout

openBalena
1

Hi, I am getting this timeout error while trying to perform the reboot of an operation:

balena device reboot

Stack of the error:

FetchError: network timeout at: https://api.domain.org/supervisor/v1/reboot
at Timeout._onTimeout (/usr/local/lib/balena-cli/node_modules/node-fetch/index.js:126:13)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10)

Edit:
SUPERVISOR VERSION: 10.2.2
OS VERSION: balenaOS 2.43.0+rev1

What could it be related? Our domain or the balena endpoint?

Thanks for any help or suggestion

Matteo

2

Hi,

I think it’s the endpoint, can you please check if your openBalena API is actually working or is there anything can block the connection that can cause network timeout?

3

I don’t have any problem running a curl command as:

curl -X GET “https://api.balena-cloud.com/v5/application”
-H “Content-Type: application/json”
-H “Authorization: Bearer token”

4

Yeah, that’s the balenaCloud API but from the error messages above, you’re sending the request to your API instead (https://api.domain.org/supervisor/v1/reboot)

7

Sorry, by doing this call

$ curl -X GET “https://api.domain.org/v5/application” -H “Content-Type: application/json” -H “Authorization: Bearer token” --cacert ~/ca.crt

I am not getting at the moment any error

9

Hey there! Glad to hear its working fine. Do tell us if it happens again and we’ll be more than happy to help!

10

Wait, I don’t think is solved, I am still getting the same error while doing

balena device reboot uuid

and, with this call:

curl -X POST --header “Content-Type:application/json” --header “Authorization: Bearer token” --data ‘{“uuid”: “uuid”}’ “https://api.domain.org/supervisor/v1/reboot” --cacert ~/ca.crt

gives me:

curl: (52) Empty reply from server

while the same call, but with this domain “https://api.balena-cloud.com/supervisor/v1/reboot”
seems to work

11

Hey there! api.balena-cloud.com is our hosted instance of balena, and we guarantee that to work. You also have your own separate instance of openBalena running at domain.org, and that is the one that seems to be having problems.

What do you get if you do GET https://api.domain.org/ping?

13

the answer to that call is

OK

Edit:

And what should I do next to understand where is the problem on my instance of openBalena?

Thanks again for the help :slight_smile:

18

@matteopeluso, is it only balena device reboot that fails?

  • Does the output of balena settings and balena whoami show the expected values? E.g. pointing to the correct domain name and user name, rather than balena-cloud.com.
    • By the way, I assume that you are not the owner of the rather generic “domain.org”, and that you’re replacing the actual domain with “domain.org” for privacy. It’s all fine – the only not-fine case is if your actual domain is “somethingelse.com” and you are actually using “domain.org” in the CLI/openBalena configuration. (balena has nothing to do with “domain.org”, as far as I am aware!)
  • Do the following also succeed?

Sorry for the elementary questions, but sometimes that’s where the problem is, and also any errors in some of the other commands above may help us diagnose the problem. By the way, set the DEBUG=1 environment variable for extra debugging information. Thanks!

1 Like
19

And if all the above is fine, perhaps it is the communication link between the device and your openBalena server that is facing an issue. I believe the request to https://api.domain.org/supervisor/v1/reboot goes from the CLI to the openBalena server, and from there to the device where the supervisor is running.

By the way, you could run a few checks on the target device, by opening a ssh shell on it:

  • balena-engine ps (is the supervisor container running?) (share the output if not sure)
  • journalctl -au openvpn (is the VPN link up and running?)
  • journalctl -au resin-supervisor (is the supervisor healthy?)
  • journalctl -au balena
  • journalctl -ab (anything suspicious?)
1 Like
21

Hi, thanks for the answer.
Yes I am hiding the actual domain ^^

All of these operations with the balena cli works fine.
I have tried to open an ssh connection with that specific device but it didn’t let me in

[Error] 127.0.0.1:58102 => 127.0.0.1:22197 ===> 10.240.0.35:22222 :: Device is not listening on port 22222

but by the balena logs I could find this error:

GET https://api.domain.org/pingFailed to download image ‘registry.nerds4life.org/v2/5276f71db744452b35b5bd5bfe24a6c8@sha256:1489e1df188fe78993dec8e572b9ef5fb6043d67c0cfdc64e556bc919b40d4a8’ due to '(HTTP code 500) server error - Get https://registry.domain.org/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

So this error could be specific to this device?

The application I am running on that device is a multicontainer one which is deployed to a fleet of devices, the others don’t have this particular problem.

EDIT:
I am getting the network timeout error also in another device/application.

22

So this error could be specific to this device?

It could be. I suppose you’re using the procedure in the following post to open the ssh shell, right? HowTo: SSH into host device

If you happen to have direct access to the local network whether the device is, it may be possible to ssh directly to the device with the command:

ssh -p 22222 root@<ipaddress>

replacing <ipaddress> with the device’s local IP address such as, say, “192.168.10.10”. (To find the device’s IP local address, commands like arp or balena scan may be helpful.)

If the device is running a development image, no password or ssh key setup is needed. If it is running a production image, an ssh key has to be added to config.json. (I appreciate you may no longer have access to the device to add a key in this way…).

23

Yes, I am following that post to do an ssh connection, I was able to retrieve this errors/logs

error_supervisor.log (3.0 KB) logs_error_balena.log (874 Bytes) logs_error_supervisor.log (1.4 KB) logs_error_vpn.log (5.7 KB)

24

Hi, according to the logs several services on that device are having network problems. Do you have other devices on the same local subnet?

26

And that influences all the balenaCLI calls?

27

Hi,

Some balenaCLI calls are proxied through a tunnel to the device, so if the device has a bad network connection is not reachable balenaCLI calls might fail. Do you have other devices on the same local subnet? If so do they have similar problems?

29

Hi, I don’t have any other devices in that network, also because each device has its own 3g Dongle.

I was experiencing that error also with a test device I have in my office which doesn’t have any bad connection problem.

30

Hello, by “that error” do you mean the same error as in your first post ?
This error does not seem to be related to any device. It is happening between your computer and your openBalena server.
Either your computer or your openBalena instance at domain.org is having network issues.
Or are you running balena device reboot from your balena device ?

32

Ok, I will monitor then my openBalena server and see if this error comes back again.

Thanks to everybody! :slight_smile: