SELF_SIGNED_CERT_IN_CHAIN server error

openBalena
1

We’ve had our openBalena server set up for more than a year now, with 20+ devices working without major errors.
About a week ago, when I tried to login using balena-cli I got an ECONNREFUSED, so I ssh’d into our openBalena server and saw that all the containers were down (using docker-compose ps). I brought the all up again with docker-compose up -d. All containers seemed to be running and I tried to login again.
Now I got a new error SELF_SIGNED_CERT_IN_CHAIN, here is the debug trace:
FetchError: request to https://api.iot-fleet.gen-i.si/login_ failed, reason: self signed certificate in certificate chain
at ClientRequest. (/snapshot/versioned-source/node_modules/node-fetch/index.js:133:11)
at ClientRequest.emit (events.js:203:15)
at ClientRequest.EventEmitter.emit (domain.js:448:20)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at TLSSocket.emit (events.js:198:13)
at TLSSocket.EventEmitter.emit (domain.js:448:20)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
From previous event:
at new Fetch (/snapshot/versioned-source/node_modules/node-fetch/index.js:49:9)
at Fetch (/snapshot/versioned-source/node_modules/node-fetch/index.js:37:10)
at module.exports.fetch.Promise (/snapshot/versioned-source/node_modules/fetch-ponyfill/fetch-node.js:15:12)
at requestAsync (/snapshot/versioned-source/node_modules/balena-request/build/utils.js:327:7)
at exports.getRequestAsync (/snapshot/versioned-source/node_modules/balena-request/build/utils.js:396:12)
at /snapshot/versioned-source/node_modules/balena-request/build/request.js:176:14
From previous event:
at Object.module.exports.exports.send (/snapshot/versioned-source/node_modules/balena-request/build/request.js:175:89)
at authenticate (/snapshot/versioned-source/node_modules/balena-sdk/build/auth.js:137:14)
at login (/snapshot/versioned-source/node_modules/balena-sdk/build/auth.js:174:16)
at addCallbackSupport (/snapshot/versioned-source/node_modules/balena-sdk/build/util/callbacks.js:19:27)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5)
at process.topLevelDomainCallback (domain.js:126:23)
From previous event:
at Object.authenticate (/snapshot/versioned-source/build/utils/patterns.js:25:10)
at LoginCmd.doLogin (/snapshot/versioned-source/build/actions-oclif/login.js:53:33)

I double checked the NODE_EXTRA_CA_CERTS and also did sudo update-ca-certificates again, but the error persisted. I also tried to login from a windows machine and even installed balena-cli on another linux machine, but still I got the same result (self signed error).

I also ssh’d back to openBalena and tried to launch all the containers with the ./scripts/compose up -d, but now I get the following trace:
❯ ./scripts/compose up -d
Creating network “openbalena_default” with the default driver
Starting openbalena_s3_1 …
Starting openbalena_db_1 …
Starting openbalena_redis_1 … error
Starting openbalena_cert-provider_1 …
Starting openbalena_db_1 … error
ERROR: for openbalena_redis_1 Cannot start service redis: b’network d3576ed04dfeb77b596ffc25dd1813511adc1891d25de4dd8540db4f45350c87 not found’

ERROR: for openbalena_db_1 Cannot start service db: b’network d3576ed04dfeb77b5Starting openbalena_s3_1 … error

ERROR: for openbalena_cert-provider_1 Cannot start service cert-provider: b’network d3576ed04dfeb77b596ffc25dd1813511adc1891d25de4dd8540db4f45350c87 not found’

ERROR: for openbalena_s3_1 Cannot start service s3: b’network d3576ed04dfeb77b596ffc25dd1813511adc1891d25de4dd8540db4f45350c87 not found’

ERROR: for redis Cannot start service redis: b’network d3576ed04dfeb77b596ffc25dd1813511adc1891d25de4dd8540db4f45350c87 not found’

ERROR: for db Cannot start service db: b’network d3576ed04dfeb77b596ffc25dd1813511adc1891d25de4dd8540db4f45350c87 not found’

ERROR: for cert-provider Cannot start service cert-provider: b’network d3576ed04dfeb77b596ffc25dd1813511adc1891d25de4dd8540db4f45350c87 not found’

ERROR: for s3 Cannot start service s3: b’network d3576ed04dfeb77b596ffc25dd1813511adc1891d25de4dd8540db4f45350c87 not found’
ERROR: Encountered errors while bringing up the project.

My question is, have I messed up completely? Do I have any chance of logging back into our instance and seeing deployed devices or will I have to start from the begining?

5

Hi,

let’s do some debugging and see how bad the situation really is.

Though these might be related I think we are dealing with two separate issues here:

  • The openBalena containers being down
  • The self-signed certificate error

Could you please elaborate a bit so that we can get a better idea about the current state?

  • You say the openBalena instance has been running fine for over a year and suddenly you got a connection refused, is it something that was working regularly and has just suddenly broken or did you come back after a longer period of time and it just was not working?
  • Were you able to get any error logs from the openBalena containers before they crashed again?
  • The last error seems to be just a lack of cleanup, could you try compose down before the compose up -d?
  • When you say you "double checked the NODE_EXTRA_CA_CERTS " what exactly do you mean? On what machine that was? What were the contents?

Thanks

7

Hello, sorry for the late reply.
Here’s the situation:

  • You were right about the containers, ./scripts/compose down did the trick, all the containers come back up with ./scripts/compose up

  • The login also works again, so the proper running of the containers helped!

I can see al the devices and applications, so nothing is lost. Thank you for your help. Containers have been running smoothly for a long time, and we logged in only sporadically to add a device. It seems that there was an error with the virtual machine on azure, where our instance was running and the containers weren’t running (hence ECONNREFUSED error). Then I improperly started the instance with docker-compomse instead of with included script and then the error with self signed certificates started.
That just goes to show I need to read the manual more carefully next time :sweat_smile: Thank you again for all your help.