Went down again today, still doing so every few days. I’m hoping I am not conflating issues here, this may or may not be related to the registry issue. Will keep them together here for now until we can identify if it is or not.
This time, I managed to locate the following error in the journalctl logs of balena/open-balena-api:v0.119.5. The container is still running, as is every other container, but it is non-responsive on mydomain.com/ping where I usually monitor the server status. It had just been idling, no action taken to trigger the error:
Mar 12 18:21:19 41d36e52e116 api[982]: )) [ 2021-03-12T18:21:19.126Z, 68 ]
Mar 12 18:21:19 41d36e52e116 api[982]: 2021-03-12T18:21:19.135Z 172.20.0.8 s/vpn PATCH /resin/service_instance(68) 200 34.843ms -
Mar 12 18:21:28 41d36e52e116 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=96:00:00:7a:73:5b:d2:74:7f:6e:37:e3:08:00 SRC=103.89.88.99 DST=my-ip LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=22434 PROTO=TCP SPT=51346 DPT=3741 WINDOW=1024 RES=0x00 SYN URGP=0
Mar 12 18:21:29 41d36e52e116 api[982]: Parsing GET /Auth/permission?$select=name&$filter=(is_of__api_key/any(khp:khp/api_key/any(k:k/key eq @apiKey))) or (is_of__role/any(rhp:rhp/role/any(r:r/is_of__api_key/any(khr:khr/api_key/any(k:k/key eq @apiKey)))))&$orderby=name asc&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: Running GET /Auth/permission?$select=name&$filter=(is_of__api_key/any(khp:khp/api_key/any(k:k/key eq @apiKey))) or (is_of__role/any(rhp:rhp/role/any(r:r/is_of__api_key/any(khr:khr/api_key/any(k:k/key eq @apiKey)))))&$orderby=name asc&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT "permission"."name"
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "permission"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE (EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key-has-permission" AS "permission.api key-has-permission"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission"."id" = "permission.api key-has-permission"."permission"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key" AS "permission.api key-has-permission.api key"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.api key-has-permission"."api key" = "permission.api key-has-permission.api key"."id"
Mar 12 18:21:29 41d36e52e116 api[982]: AND ("permission.api key-has-permission.api key"."key") IS NOT NULL AND ("permission.api key-has-permission.api key"."key") = ($1)
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: OR EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "role-has-permission" AS "permission.role-has-permission"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission"."id" = "permission.role-has-permission"."permission"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "role" AS "permission.role-has-permission.role"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.role-has-permission"."role" = "permission.role-has-permission.role"."id"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key-has-role" AS "permission.role-has-permission.role.api key-has-role"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.role-has-permission.role"."id" = "permission.role-has-permission.role.api key-has-role"."role"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key" AS "permission.role-has-permission.role.api key-has-role.api key"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.role-has-permission.role.api key-has-role"."api key" = "permission.role-has-permission.role.api key-has-role.api key"."id"
Mar 12 18:21:29 41d36e52e116 api[982]: AND ("permission.role-has-permission.role.api key-has-role.api key"."key") IS NOT NULL AND ("permission.role-has-permission.role.api key-has-role.api key"."key") = ($1)
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: ))
Mar 12 18:21:29 41d36e52e116 api[982]: ORDER BY "permission"."name" ASC [ 'hidden-the-key' ]
Mar 12 18:21:29 41d36e52e116 api[982]: Parsing GET /Auth/api_key(key=@apiKey)?$select=is_of__actor&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: Running GET /Auth/api_key(key=@apiKey)?$select=is_of__actor&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT "api key"."is of-actor" AS "is_of__actor"
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE ("api key"."key") IS NOT NULL AND ("api key"."key") = ($1) [ 'hidden-the-key' ]
Mar 12 18:21:29 41d36e52e116 api[982]: Parsing PATCH /resin/service_instance(68)
Mar 12 18:21:29 41d36e52e116 api[982]: Parsing GET /Auth/permission?$select=name&$filter=(is_of__api_key/any(khp:khp/api_key/any(k:k/key eq @apiKey))) or (is_of__role/any(rhp:rhp/role/any(r:r/is_of__api_key/any(khr:khr/api_key/any(k:k/key eq @apiKey)))))&$orderby=name asc&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: Running GET /Auth/permission?$select=name&$filter=(is_of__api_key/any(khp:khp/api_key/any(k:k/key eq @apiKey))) or (is_of__role/any(rhp:rhp/role/any(r:r/is_of__api_key/any(khr:khr/api_key/any(k:k/key eq @apiKey)))))&$orderby=name asc&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT "permission"."name"
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "permission"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE (EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key-has-permission" AS "permission.api key-has-permission"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission"."id" = "permission.api key-has-permission"."permission"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key" AS "permission.api key-has-permission.api key"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.api key-has-permission"."api key" = "permission.api key-has-permission.api key"."id"
Mar 12 18:21:29 41d36e52e116 api[982]: AND ("permission.api key-has-permission.api key"."key") IS NOT NULL AND ("permission.api key-has-permission.api key"."key") = ($1)
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: OR EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "role-has-permission" AS "permission.role-has-permission"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission"."id" = "permission.role-has-permission"."permission"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "role" AS "permission.role-has-permission.role"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.role-has-permission"."role" = "permission.role-has-permission.role"."id"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key-has-role" AS "permission.role-has-permission.role.api key-has-role"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.role-has-permission.role"."id" = "permission.role-has-permission.role.api key-has-role"."role"
Mar 12 18:21:29 41d36e52e116 api[982]: AND EXISTS (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT 1
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key" AS "permission.role-has-permission.role.api key-has-role.api key"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE "permission.role-has-permission.role.api key-has-role"."api key" = "permission.role-has-permission.role.api key-has-role.api key"."id"
Mar 12 18:21:29 41d36e52e116 api[982]: AND ("permission.role-has-permission.role.api key-has-role.api key"."key") IS NOT NULL AND ("permission.role-has-permission.role.api key-has-role.api key"."key") = ($1)
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: )
Mar 12 18:21:29 41d36e52e116 api[982]: ))
Mar 12 18:21:29 41d36e52e116 api[982]: ORDER BY "permission"."name" ASC [ 'hidden-the-key' ]
Mar 12 18:21:29 41d36e52e116 api[982]: Parsing GET /Auth/api_key(key=@apiKey)?$select=is_of__actor&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: Running GET /Auth/api_key(key=@apiKey)?$select=is_of__actor&@apiKey='hidden-the-key'
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT "api key"."is of-actor" AS "is_of__actor"
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "api key"
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE ("api key"."key") IS NOT NULL AND ("api key"."key") = ($1) [ 'hidden-the-key' ]
Mar 12 18:21:29 41d36e52e116 api[982]: Running PATCH /resin/service_instance(68)
Mar 12 18:21:29 41d36e52e116 api[982]: UPDATE "service instance"
Mar 12 18:21:29 41d36e52e116 api[982]: SET "last heartbeat" = $1
Mar 12 18:21:29 41d36e52e116 api[982]: WHERE ("service instance"."id") IS NOT NULL AND ("service instance"."id") = ($2)
Mar 12 18:21:29 41d36e52e116 api[982]: AND "service instance"."id" IN ((
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT "service instance"."id"
Mar 12 18:21:29 41d36e52e116 api[982]: FROM (
Mar 12 18:21:29 41d36e52e116 api[982]: SELECT "service instance"."created at", "service instance"."modified at", "service instance"."id", "service instance"."service type", "service instance"."ip address", "service instance"."last heartbeat"
Mar 12 18:21:29 41d36e52e116 api[982]: FROM "service instance"
Mar 12 18:21:29 41d36e52e116 api[982]: ) AS "service instance"
Mar 12 18:21:29 41d36e52e116 api[982]: )) [ 2021-03-12T18:21:29.177Z, 68 ]
Mar 12 18:21:29 41d36e52e116 api[982]: 2021-03-12T18:21:29.183Z 172.20.0.8 s/vpn PATCH /resin/service_instance(68) 200 27.666ms -
Mar 12 18:21:30 41d36e52e116 api[982]: /usr/src/app/src/features/contracts/contracts-directory.ts:123
Mar 12 18:21:30 41d36e52e116 api[982]: new Error(
Mar 12 18:21:30 41d36e52e116 api[982]: ^
Mar 12 18:21:30 41d36e52e116 api[982]: Error: Invalid response while fetching contracts: Internal Server Error
Mar 12 18:21:30 41d36e52e116 api[982]: at Request.handleResponse [as _callback] (/usr/src/app/src/features/contracts/contracts-directory.ts:123:3)
Mar 12 18:21:30 41d36e52e116 api[982]: at Request.self.callback (/usr/src/app/node_modules/request/request.js:185:22)
Mar 12 18:21:30 41d36e52e116 api[982]: at Request.emit (events.js:327:22)
Mar 12 18:21:30 41d36e52e116 api[982]: at Request.EventEmitter.emit (domain.js:467:12)
Mar 12 18:21:30 41d36e52e116 api[982]: at Request.<anonymous> (/usr/src/app/node_modules/request/request.js:1154:10)
Mar 12 18:21:30 41d36e52e116 api[982]: at Request.emit (events.js:327:22)
Mar 12 18:21:30 41d36e52e116 api[982]: at Request.EventEmitter.emit (domain.js:467:12)
Mar 12 18:21:30 41d36e52e116 api[982]: at IncomingMessage.<anonymous> (/usr/src/app/node_modules/request/request.js:1076:12)
Mar 12 18:21:30 41d36e52e116 api[982]: at Object.onceWrapper (events.js:421:28)
Mar 12 18:21:30 41d36e52e116 api[982]: at IncomingMessage.emit (events.js:327:22)
Mar 12 18:21:30 41d36e52e116 api[982]: at IncomingMessage.EventEmitter.emit (domain.js:467:12)
Mar 12 18:21:30 41d36e52e116 api[982]: at endReadableNT (internal/streams/readable.js:1327:12)
Mar 12 18:21:30 41d36e52e116 api[982]: at processTicksAndRejections (internal/process/task_queues.js:80:21)
Mar 12 18:21:30 41d36e52e116 api[982]: Program node index.js exited with code 1