Openbalena crash logs

rlev · February 7, 2019, 10:24pm

I have noted a couple of times that my openbalena backend instance gives http error 503 Service Unavailable when trying to execute commands using balena-cli.
Is there any logs in the openbalena backend side that I can check to see what is causing this?

So far the only workaround I found is to restart openbalena app.

Any help is appreciated.

richbayliss · February 8, 2019, 9:30am

Sure thing @rlev

So if you’re getting a 503 error then it means that one of the backend services, probably the API, has crashed out and isn’t responding. You can check this by finding the container name/ID of the API container, then running the command below:

$ docker exec -it <name or ID> journalctl -u open-balena-api -fn100

This should show some logs and possibly an exception. Screenshot attached for an example.

Screenshot%20from%202019-02-08%2009-23-05

rlev · February 9, 2019, 1:23pm

Thank you for getting back @richbayliss.

Question :

What would be a good way to monitor that the openbalena api service is running OK and in case is not do a service restart automatically?

Thanks

richbayliss · February 11, 2019, 1:50pm

It shouldn’t be getting into a state which caused it to crash; so I would be very interested to see your logs.

rlev · February 16, 2019, 4:16pm

Hi @richbayliss,

Sorry for the delay. I was waiting for the issue to happens again.

Below you will see the logs when I intent to login into my backend instance of openbalena and I get 503 error :

Logging in to mydomain.com
BalenaRequestError: Request error:

503 Service Unavailable

No server is available to handle this request.

If you need help, don’t hesitate in contacting us at:

** GitHub: Sign in to GitHub · GitHub**
** Forums: https://forums.balena.io**

– Logs begin at Tue 2019-02-12 Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa api[906]: Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:17:58 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa api[906]: Feb 12 21:18:00 7c19ee2bf1aa 20:48:27 UTC. –
at runPut (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1357:23)
at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1145:18
api[906]: From previous event:
at runRequest (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1136:51)
at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1043:22
at Promise.try (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:182:55)
api[906]: From previous event:
at promise.tap (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:182:48)
api[906]: From previous event:
at Promise (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:182:25)
api[906]: From previous event:
at Object.transaction (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:174:16)
at runTransaction (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1233:17)
at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1032:18
api[906]: From previous event:
at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1030:13
api[906]: From previous event:
at Promise.each (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/control-flow.js:32:16)
api[906]: From previous event:
at mapTill (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/control-flow.js:31:20)
at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1015:14
api[906]: From previous event:
at handleODataRequest (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1003:8)
at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:912:14
api[906]: From previous event:
at runURI (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:861:12)
at PinejsClient._request (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:814:14)
at PinejsClientCoreFactory.PinejsClientCore.request (/usr/src/app/node_modules/pinejs-client-core/core.js:841:29)
at PinejsClientCoreFactory.PinejsClientCore.patch (/usr/src/app/node_modules/pinejs-client-core/core.js:777:25)
at auth_1.comparePassword.then.res (/usr/src/app/src/routes/session.ts:32:40)
api[906]: From previous event:
at auth_1.findUser.then.user (/usr/src/app/src/routes/session.ts:28:14)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5)
at process.topLevelDomainCallback (domain.js:121:23)
api[906]: From previous event:
at exports.login (/usr/src/app/src/routes/session.ts:23:10)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at next (/usr/src/app/node_modules/express/lib/router/route.js:137:13)
at module.exports. (/usr/src/app/node_modules/express-brute/index.js:142:36)
at Multi.callback (/usr/src/app/node_modules/express-brute-redis/index.js:35:45)
at multi_callback (/usr/src/app/node_modules/redis/lib/multi.js:89:14)
at Command.callback (/usr/src/app/node_modules/redis/lib/multi.js:116:9)
at normal_reply (/usr/src/app/node_modules/redis/index.js:726:21)
at RedisClient.return_reply (/usr/src/app/node_modules/redis/index.js:824:9)
at JavascriptRedisParser.returnReply (/usr/src/app/node_modules/redis/index.js:192:18)
at JavascriptRedisParser.execute (/usr/src/app/node_modules/redis-parser/lib/parser.js:574:12)
at Socket. (/usr/src/app/node_modules/redis/index.js:274:27)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:442:20)
at addChunk (_stream_readable.js:283:12)
at readableAddChunk (_stream_readable.js:264:11)
at Socket.Readable.push (_stream_readable.js:219:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17) message: ‘Connection terminated’, name: ‘DatabaseError’ }
api[906]: resin: Running GET /resin/user(2)
api[906]: Auth: Running GET /Auth/permission?$select=name&$filter=(is_of__user/any(uhp:(uhp/user eq 2) and ((uhp/expiry_date eq null) or (uhp/expiry_date gt now())))) or (is_of__role/any(rhp:rhp/role/any(r:r/is_of__user/any(uhr:(uhr/user eq 2) and ((uhr/expiry_date eq null) or (uhr/expiry_date gt now()))))))&$orderby=name asc
api[906]: resin: SELECT “user”.“created at” AS “created_at”, “user”.“id”, “user”.“actor”, “user”.“username”, “user”.“password”, “user”.“jwt secret” AS “jwt_secret”, “user”.“email”
api[906]: FROM “user”
api[906]: WHERE “user”.“id” = $1 [ 2 ]
api[906]: Auth: SELECT “permission”.“name”
api[906]: FROM “permission”
api[906]: WHERE (EXISTS (
SELECT 1
FROM “user-has-permission” AS “permission.user-has-permission”
WHERE “permission”.“id” = “permission.user-has-permission”.“permission”
AND “permission.user-has-permission”.“user” = $1
AND (“permission.user-has-permission”.“expiry date” IS NULL
OR “permission.user-has-permission”.“expiry date” > CURRENT_TIMESTAMP)
api[906]: )
api[906]: OR EXISTS (
SELECT 1
FROM “role-has-permission” AS “permission.role-has-permission”
WHERE “permission”.“id” = “permission.role-has-permission”.“permission”
AND EXISTS (
SELECT 1
FROM “role” AS “permission.role-has-permission.role”
WHERE “permission.role-has-permission”.“role” = “permission.role-has-permission.role”.“id”
AND EXISTS (
SELECT 1
FROM “user-has-role” AS “permission.role-has-permission.role.user-has-role”
WHERE “permission.role-has-permission.role”.“id” = “permission.role-has-permission.role.user-has-role”.“role”
AND “permission.role-has-permission.role.user-has-role”.“user” = $2
AND (“permission.role-has-permission.role.user-has-role”.“expiry date” IS NULL
OR “permission.role-has-permission.role.user-has-role”.“expiry date” > CURRENT_TIMESTAMP)
)
)
api[906]: ))
api[906]: ORDER BY “permission”.“name” ASC [ 2, 2 ]
api[906]: Error: Connection terminated unexpectedly
at Client. (/usr/src/app/node_modules/pg/lib/client.js:226:28)
at Object.onceWrapper (events.js:273:13)
at Connection.emit (events.js:182:13)
at Connection.EventEmitter.emit (domain.js:442:20)
at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:138:10)
at Socket.emit (events.js:187:15)
at Socket.EventEmitter.emit (domain.js:442:20)
at endReadableNT (_stream_readable.js:1094:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
api[906]: Program node index.js exited with code 1

richbayliss · February 19, 2019, 9:58am

OK so the API died because it lost connection with the Redis & DB services (looking at the errors in the logs) which I have seen in the past if the HAproxy container restarts. The services talk to each other through this service, so that could be the cause.

I will take a look at what you could do to have it restart in the event of an error like this

richbayliss · February 19, 2019, 10:07am

@rlev so if you set the environment variable PRODUCTION_MODE: "true" in the docker-compose.yml for the stack, then you will loose some verbosity on the logs, but you should find that in the event of the API crashing it will be restarted by systemd.

rlev · February 20, 2019, 11:52am

@richbayliss

In the file /open-balena/config/docker-compose.yml
I find only :

 # Project-specific config.
 #
 # All paths must be defined relative to `compose/services.yml` regardless of
 # the location of this file, i.e. refer to `my-open-balena-checkout/somedir`
 # as `../somedir`. This is because of the way docker-compose handles paths
 # when specifying multiple configs and open-balena always specifying
 # `compose/services.yml` as the "base" config.
 #
 # You may view the effective config with `scripts/compose config`.
 version: '2.1'

But in the /open-balena/compose/services.yml

I see :

services:
  api:
    extends:
      file: ./common.yml
      service: component
    image: balena/open-balena-api:${OPENBALENA_API_VERSION_TAG:-master}
    depends_on:
      - db
      - s3
      - redis
    environment:
      API_VPN_SERVICE_API_KEY: ${OPENBALENA_API_VPN_SERVICE_API_KEY}
      BALENA_ROOT_CA: ${OPENBALENA_ROOT_CA}
      COOKIE_SESSION_SECRET: ${OPENBALENA_COOKIE_SESSION_SECRET}
      DB_HOST: db.${OPENBALENA_HOST_NAME}
      DB_PASSWORD: docker
      DB_PORT: 5432
      DB_USER: docker
      .....
      .....
      MIXPANEL_TOKEN: __unused__

PRODUCTION_MODE: ‘${OPENBALENA_PRODUCTION_MODE}’
PUBNUB_PUBLISH_KEY: unused
PUBNUB_SUBSCRIBE_KEY: unused
REDIS_HOST: redis.${OPENBALENA_HOST_NAME}

vpn:
    extends:
      file: ./common.yml
      service: component
    image: balena/open-balena-vpn:${OPENBALENA_VPN_VERSION_TAG:-master}
    depends_on:
      - api
    cap_add:
      - NET_ADMIN
    environment:
      API_SERVICE_API_KEY: ${OPENBALENA_API_VPN_SERVICE_API_KEY}
      BALENA_API_HOST: api.${OPENBALENA_HOST_NAME}
      BALENA_ROOT_CA: ${OPENBALENA_ROOT_CA}
      BALENA_VPN_PORT: 443

PRODUCTION_MODE: ‘${OPENBALENA_PRODUCTION_MODE}’
RESIN_VPN_GATEWAY: 10.2.0.1
SENTRY_DSN:
VPN_HAPROXY_USEPROXYPROTOCOL: ‘true’
VPN_OPENVPN_CA_CRT: ${OPENBALENA_VPN_CA}
VPN_OPENVPN_SERVER_CRT: ${OPENBALENA_VPN_SERVER_CRT}
VPN_OPENVPN_SERVER_DH: ${OPENBALENA_VPN_SERVER_DH}
VPN_OPENVPN_SERVER_KEY: ${OPENBALENA_VPN_SERVER_KEY}
VPN_SERVICE_API_KEY: ${OPENBALENA_VPN_SERVICE_API_KEY}

Is this the file you are refering to?

richbayliss · February 20, 2019, 12:36pm

Yes, sorry, just replace the ${OPENBALENA_PRODUCTION_MODE} for the API service and it should do the trick once you update the running services

rlev · February 20, 2019, 12:48pm

Will do.

Thanks @richbayliss

rlev · February 24, 2019, 10:07pm

@richbayliss I have changed OPENBALENA_PRODUCTION_MODE to ‘true’ for the api service and I see the service still crashing but now looks like it’s restarting.

See logs below :

~$ sudo docker exec -it openbalena_api_1_e7a98861293c journalctl -u open-balena-api -fn500
-- Logs begin at Sun 2019-02-24 21:08:45 UTC. --
Feb 24 21:08:52 fd803e62df47 systemd[1]: Started open-balena-api.
Feb 24 21:08:55 fd803e62df47 api[916]: raven@2.6.4 alert: no DSN provided, error reporting disabled
Feb 24 21:08:55 fd803e62df47 api[916]: Sucessfully executed standard models.
Feb 24 21:08:55 fd803e62df47 api[916]: Successfully executed migrations model.
Feb 24 21:08:56 fd803e62df47 api[916]: Loading application config
Feb 24 21:08:56 fd803e62df47 api[916]: Loading JSON: /usr/src/app/config.json
Feb 24 21:08:56 fd803e62df47 api[916]: Successfully executed resin model.
Feb 24 21:08:57 fd803e62df47 api[916]: Creating superuser account...
Feb 24 21:08:57 fd803e62df47 api[916]: Superuser already exists!
Feb 24 21:08:58 fd803e62df47 api[916]: Server listening in production mode on port 80
Feb 24 21:09:23 fd803e62df47 api[916]: Sun, 24 Feb 2019 21:09:23 GMT express deprecated req.param(name): Use req.params, req.body, or req.query instead at src/routes/services.ts:13:27
Feb 24 21:13:44 fd803e62df47 api[916]: Sun, 24 Feb 2019 21:13:44 GMT express deprecated req.param(name): Use req.params, req.body, or req.query instead at src/routes/devices.ts:186:22
Feb 24 21:50:30 fd803e62df47 api[916]: resin: { DatabaseError: Connection terminated
Feb 24 21:50:30 fd803e62df47 api[916]:     at wrapDatabaseError (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:39:15)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:30 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:30 fd803e62df47 api[916]:  message: 'Connection terminated', name: 'DatabaseError' }
Feb 24 21:50:30 fd803e62df47 api[916]: resin: { DatabaseError: Connection terminated
Feb 24 21:50:30 fd803e62df47 api[916]:     at wrapDatabaseError (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:39:15)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:30 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:30 fd803e62df47 api[916]:  message: 'Connection terminated', name: 'DatabaseError' }
Feb 24 21:50:30 fd803e62df47 api[916]: resin: { DatabaseError: Connection terminated
Feb 24 21:50:30 fd803e62df47 api[916]:     at wrapDatabaseError (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:39:15)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:30 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:30 fd803e62df47 api[916]:  message: 'Connection terminated', name: 'DatabaseError' }
Feb 24 21:50:30 fd803e62df47 api[916]: Error: Transaction has been ended.
Feb 24 21:50:30 fd803e62df47 api[916]:     at PostgresTx.rejectFn (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:73:43)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise.try.tap.tapCatch (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:183:60)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.finallyHandler (/usr/src/app/node_modules/bluebird/js/release/finally.js:57:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:30 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:30 fd803e62df47 api[916]: Error: Transaction has been ended.
Feb 24 21:50:30 fd803e62df47 api[916]:     at PostgresTx.rejectFn (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:73:43)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise.try.tap.tapCatch (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:183:60)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.finallyHandler (/usr/src/app/node_modules/bluebird/js/release/finally.js:57:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:30 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:30 fd803e62df47 api[916]: Error: Transaction has been ended.
Feb 24 21:50:30 fd803e62df47 api[916]:     at PostgresTx.rejectFn (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:73:43)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise.try.tap.tapCatch (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:183:60)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.finallyHandler (/usr/src/app/node_modules/bluebird/js/release/finally.js:57:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:30 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:30 fd803e62df47 api[916]: Error: Transaction has been ended.
Feb 24 21:50:30 fd803e62df47 api[916]:     at PostgresTx.rejectFn (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:73:43)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise.try.tap.tapCatch (/usr/src/app/node_modules/@resin/pinejs/out/database-layer/db.js:183:60)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.finallyHandler (/usr/src/app/node_modules/bluebird/js/release/finally.js:57:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at PassThroughHandlerContext.tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:690:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:30 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:30 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:30 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:30 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:31 fd803e62df47 api[916]: Error JWT auth Transaction has been ended. NotFoundError: Transaction has been ended.
Feb 24 21:50:31 fd803e62df47 api[916]:     at Object.json (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:897:29)
Feb 24 21:50:31 fd803e62df47 api[916]:     at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1076:22
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:694:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:31 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:31 fd803e62df47 api[916]: Error JWT auth Transaction has been ended. NotFoundError: Transaction has been ended.
Feb 24 21:50:31 fd803e62df47 api[916]:     at Object.json (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:897:29)
Feb 24 21:50:31 fd803e62df47 api[916]:     at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1076:22
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:694:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:31 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:31 fd803e62df47 api[916]: Error JWT auth Transaction has been ended. NotFoundError: Transaction has been ended.
Feb 24 21:50:31 fd803e62df47 api[916]:     at Object.json (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:897:29)
Feb 24 21:50:31 fd803e62df47 api[916]:     at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1076:22
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:694:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:31 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:31 fd803e62df47 api[916]: Error JWT auth Transaction has been ended. NotFoundError: Transaction has been ended.
Feb 24 21:50:31 fd803e62df47 api[916]:     at Object.json (/usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:897:29)
Feb 24 21:50:31 fd803e62df47 api[916]:     at /usr/src/app/node_modules/@resin/pinejs/out/sbvr-api/sbvr-utils.js:1076:22
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryCatcher (/usr/src/app/node_modules/bluebird/js/release/util.js:16:23)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromiseFromHandler (/usr/src/app/node_modules/bluebird/js/release/promise.js:512:31)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise (/usr/src/app/node_modules/bluebird/js/release/promise.js:569:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromise0 (/usr/src/app/node_modules/bluebird/js/release/promise.js:614:10)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Promise._settlePromises (/usr/src/app/node_modules/bluebird/js/release/promise.js:694:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueueStep (/usr/src/app/node_modules/bluebird/js/release/async.js:138:12)
Feb 24 21:50:31 fd803e62df47 api[916]:     at _drainQueue (/usr/src/app/node_modules/bluebird/js/release/async.js:131:9)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Async._drainQueues (/usr/src/app/node_modules/bluebird/js/release/async.js:147:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/app/node_modules/bluebird/js/release/async.js:17:14)
Feb 24 21:50:31 fd803e62df47 api[916]:     at runCallback (timers.js:705:18)
Feb 24 21:50:31 fd803e62df47 api[916]:     at tryOnImmediate (timers.js:676:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at processImmediate (timers.js:658:5)
Feb 24 21:50:31 fd803e62df47 api[916]:     at process.topLevelDomainCallback (domain.js:121:23)
Feb 24 21:50:31 fd803e62df47 api[916]: raven@2.6.4 alert: unhandledRejection captured
Feb 24 21:50:31 fd803e62df47 api[916]: Event ID: e7d75124032d4b1081783e2c662e6012
Feb 24 21:50:31 fd803e62df47 api[916]: Reason: getaddrinfo ENOTFOUND db.mydomain.com db.mydomain.com:5432
Feb 24 21:50:31 fd803e62df47 api[916]: raven@2.6.4 alert: unhandledRejection captured
Feb 24 21:50:31 fd803e62df47 api[916]: Event ID: 25ecca65441c46c697d39414005265f7
Feb 24 21:50:31 fd803e62df47 api[916]: Reason: getaddrinfo ENOTFOUND db.mydomain.com db.mydomain.com:5432
Feb 24 21:50:31 fd803e62df47 api[916]: raven@2.6.4 alert: unhandledRejection captured
Feb 24 21:50:31 fd803e62df47 api[916]: Event ID: 4c9cef1830be4c44b1f0dcafd01ea46c
Feb 24 21:50:31 fd803e62df47 api[916]: Reason: getaddrinfo ENOTFOUND db.mydomain.com db.mydomain.com:5432
Feb 24 21:50:31 fd803e62df47 api[916]: raven@2.6.4 alert: unhandledRejection captured
Feb 24 21:50:31 fd803e62df47 api[916]: Event ID: fe06cd93c19f42438726246a0072da9f
Feb 24 21:50:31 fd803e62df47 api[916]: Reason: getaddrinfo ENOTFOUND db.mydomain.com db.mydomain.com:5432
Feb 24 21:50:31 fd803e62df47 api[916]: raven@2.6.4 alert: unhandledRejection captured
Feb 24 21:50:31 fd803e62df47 api[916]: Event ID: 26aed2fbd79b44baba6665a4d7d67d56
Feb 24 21:50:31 fd803e62df47 api[916]: Reason: getaddrinfo ENOTFOUND db.mydomain.com db.mydomain.com:5432
Feb 24 21:50:31 fd803e62df47 api[916]: raven@2.6.4 alert: unhandledRejection captured
Feb 24 21:50:31 fd803e62df47 api[916]: Event ID: d5af757801974cf5b88d094de692354e
Feb 24 21:50:31 fd803e62df47 api[916]: Reason: getaddrinfo ENOTFOUND db.mydomain.com db.mydomain.com:5432
Feb 24 21:50:31 fd803e62df47 api[916]: Error: Connection terminated unexpectedly
Feb 24 21:50:31 fd803e62df47 api[916]:     at Client.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:226:28)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Object.onceWrapper (events.js:273:13)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Connection.emit (events.js:182:13)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Connection.EventEmitter.emit (domain.js:442:20)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:138:10)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Socket.emit (events.js:187:15)
Feb 24 21:50:31 fd803e62df47 api[916]:     at Socket.EventEmitter.emit (domain.js:442:20)
Feb 24 21:50:31 fd803e62df47 api[916]:     at endReadableNT (_stream_readable.js:1094:12)
Feb 24 21:50:31 fd803e62df47 api[916]:     at process._tickCallback (internal/process/next_tick.js:63:19)
Feb 24 21:50:32 fd803e62df47 systemd[1]: open-balena-api.service: Main process exited, code=exited, status=1/FAILURE
Feb 24 21:50:32 fd803e62df47 systemd[1]: open-balena-api.service: Unit entered failed state.
Feb 24 21:50:32 fd803e62df47 systemd[1]: open-balena-api.service: Failed with result 'exit-code'.
Feb 24 21:50:33 fd803e62df47 systemd[1]: open-balena-api.service: Service hold-off time over, scheduling restart.
Feb 24 21:50:33 fd803e62df47 systemd[1]: Stopped open-balena-api.
Feb 24 21:50:34 fd803e62df47 systemd[1]: Started open-balena-api.
Feb 24 21:50:37 fd803e62df47 api[1074]: raven@2.6.4 alert: no DSN provided, error reporting disabled
Feb 24 21:50:37 fd803e62df47 api[1074]: Sucessfully executed standard models.
Feb 24 21:50:38 fd803e62df47 api[1074]: Successfully executed migrations model.
Feb 24 21:50:38 fd803e62df47 api[1074]: Loading application config
Feb 24 21:50:38 fd803e62df47 api[1074]: Loading JSON: /usr/src/app/config.json
Feb 24 21:50:38 fd803e62df47 api[1074]: Successfully executed resin model.
Feb 24 21:50:39 fd803e62df47 api[1074]: Creating superuser account...
Feb 24 21:50:39 fd803e62df47 api[1074]: Superuser already exists!
Feb 24 21:50:39 fd803e62df47 api[1074]: Server listening in production mode on port 80
Feb 24 21:50:46 fd803e62df47 api[1074]: Sun, 24 Feb 2019 21:50:46 GMT express deprecated req.param(name): Use req.params, req.body, or req.query instead at src/routes/services.ts:13:27
Feb 24 21:53:48 fd803e62df47 api[1074]: Sun, 24 Feb 2019 21:53:48 GMT express deprecated req.param(name): Use req.params, req.body, or req.query instead at src/routes/devices.ts:186:22

I am wondering what is causing this crashes? ?

rlev · February 28, 2019, 3:28pm

@richbayliss Two questions :

1- Besides the “./scripts/compose up -d” and “./scripts/compose stop” commands .
Is there any “restart services” command?

2- If I have to write an third appication to monitor the health of the openbalena api service. Which would be a good endpoint api request to do this?

Thanks

rlev · February 28, 2019, 6:57pm

openbalena api service index.js exits suddenly :

– Logs begin at Thu 2019-02-28 17:54:14 UTC. –
Feb 28 18:22:19 986d619be1ae api[1830]: resin: Parsing GET /resin/device?$select=id&$filter=uuid eq ‘a40bd2149d456174f3235d62c51fe82d’
Feb 28 18:22:20 986d619be1ae api[1830]: resin: Running GET /resin/device?$filter=startswith(uuid,%27a40bd2149d456174f3235d62c51fe82d%27)
Feb 28 18:22:20 986d619be1ae api[1830]: resin: Running GET /resin/device?$select=id&$filter=uuid eq ‘a40bd2149d456174f3235d62c51fe82d’
Feb 28 18:22:20 986d619be1ae api[1830]: Auth: Running GET /Auth/permission?$select=name&$filter=(is_of__user/any(uhp:(uhp/user eq 2) and ((uhp/expiry_date eq null) or (uhp/expiry_date gt now())))) or (is_of__role/any(rhp:rhp/role/any(r:r/is_of__user/any(uhr:(uhr/user eq 2) and ((uhr/expiry_date eq null) or (uhr/expiry_date gt now()))))))&$orderby=name asc
Feb 28 18:22:20 986d619be1ae api[1830]: resin: SELECT “device”.“created at” AS “created_at”, “device”.“id”, “device”.“actor”, “device”.“uuid”, “device”.“local id” AS “local_id”, “device”.“device name” AS “device_name”, “device”.“note”, “device”.“device type” AS “device_type”, “device”.“belongs to-application” AS “belongs_to__application”, “device”.“is online” AS “is_online”, “device”.“last connectivity event” AS “last_connectivity_event”, “device”.“is connected to vpn” AS “is_connected_to_vpn”, “device”.“last vpn event” AS “last_vpn_event”, “device”.“is locked until-date” AS “is_locked_until__date”, “device”.“logs channel” AS “logs_channel”, “device”.“public address” AS “public_address”, “device”.“vpn address” AS “vpn_address”, “device”.“ip address” AS “ip_address”, “device”.“is on-commit” AS “is_on__commit”, “device”.“download progress” AS “download_progress”, “device”.“status”, “device”.“os version” AS “os_version”, “device”.“os variant” AS “os_variant”, “device”.“supervisor version” AS “supervisor_version”, “device”.“provisioning progress” AS “provisioning_progress”, “device”.“provisioning state” AS “provisioning_state”, “device”.“api port” AS “api_port”, “device”.“api secret” AS “api_secret”, “device”.“is managed by-service instance” AS “is_managed_by__service_instance”, “device”.“should be running-release” AS “should_be_running__release”, “device”.“is managed by-device” AS “is_managed_by__device”
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “device”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “device”.“uuid” LIKE (REPLACE(REPLACE(REPLACE($1, ‘’, ‘\’), ‘_’, ‘_’), ‘%’, ‘%’) || ‘%’) [ ‘a40bd2149d456174f3235d62c51fe82d’ ]
Feb 28 18:22:20 986d619be1ae api[1830]: resin: SELECT “device”.“id”
Feb 28 18:22:20 986d619be1ae api[1830]: FROM (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT “device”.“created at”, “device”.“id”, “device”.“actor”, “device”.“uuid”, “device”.“local id”, “device”.“device name”, “device”.“note”, “device”.“device type”, “device”.“belongs to-application”, “device”.“is online”, “device”.“last connectivity event”, “device”.“is connected to vpn”, “device”.“last vpn event”, “device”.“is locked until-date”, “device”.“logs channel”, “device”.“public address”, “device”.“vpn address”, “device”.“ip address”, “device”.“is on-commit”, “device”.“download progress”, “device”.“status”, “device”.“os version”, “device”.“os variant”, “device”.“supervisor version”, “device”.“provisioning progress”, “device”.“provisioning state”, “device”.“api port”, “device”.“api secret”, “device”.“is managed by-service instance”, “device”.“should be running-release”, “device”.“is managed by-device”
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “device”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE (“device”.“actor” = $1
Feb 28 18:22:20 986d619be1ae api[1830]: OR EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “application” AS “device.belongs to-application”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “device”.“belongs to-application” = “device.belongs to-application”.“id”
Feb 28 18:22:20 986d619be1ae api[1830]: AND EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “application” AS “device.belongs to-application.depends on-application”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “device.belongs to-application”.“depends on-application” = “device.belongs to-application.depends on-application”.“id”
Feb 28 18:22:20 986d619be1ae api[1830]: AND EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “device” AS “dev.belongs to-application.depends on-application.owns-device”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “device.belongs to-application.depends on-application”.“id” = “dev.belongs to-application.depends on-application.owns-device”.“belongs to-application”
Feb 28 18:22:20 986d619be1ae api[1830]: AND “dev.belongs to-application.depends on-application.owns-device”.“actor” = $1
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: ))
Feb 28 18:22:20 986d619be1ae api[1830]: ) AS “device”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “device”.“uuid” = $2 [ 18, ‘a40bd2149d456174f3235d62c51fe82d’ ]
Feb 28 18:22:20 986d619be1ae api[1830]: Auth: SELECT “permission”.“name”
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “permission”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE (EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “user-has-permission” AS “permission.user-has-permission”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission”.“id” = “permission.user-has-permission”.“permission”
Feb 28 18:22:20 986d619be1ae api[1830]: AND “permission.user-has-permission”.“user” = $1
Feb 28 18:22:20 986d619be1ae api[1830]: AND (“permission.user-has-permission”.“expiry date” IS NULL
Feb 28 18:22:20 986d619be1ae api[1830]: OR “permission.user-has-permission”.“expiry date” > CURRENT_TIMESTAMP)
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: OR EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “role-has-permission” AS “permission.role-has-permission”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission”.“id” = “permission.role-has-permission”.“permission”
Feb 28 18:22:20 986d619be1ae api[1830]: AND EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “role” AS “permission.role-has-permission.role”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission.role-has-permission”.“role” = “permission.role-has-permission.role”.“id”
Feb 28 18:22:20 986d619be1ae api[1830]: AND EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “user-has-role” AS “permission.role-has-permission.role.user-has-role”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission.role-has-permission.role”.“id” = “permission.role-has-permission.role.user-has-role”.“role”
Feb 28 18:22:20 986d619be1ae api[1830]: AND “permission.role-has-permission.role.user-has-role”.“user” = $2
Feb 28 18:22:20 986d619be1ae api[1830]: AND (“permission.role-has-permission.role.user-has-role”.“expiry date” IS NULL
Feb 28 18:22:20 986d619be1ae api[1830]: OR “permission.role-has-permission.role.user-has-role”.“expiry date” > CURRENT_TIMESTAMP)
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: ))
Feb 28 18:22:20 986d619be1ae api[1830]: ORDER BY “permission”.“name” ASC [ 2, 2 ]
Feb 28 18:22:20 986d619be1ae api[1830]: Auth: SELECT “permission”.“name”
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “permission”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE (EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “user-has-permission” AS “permission.user-has-permission”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission”.“id” = “permission.user-has-permission”.“permission”
Feb 28 18:22:20 986d619be1ae api[1830]: AND “permission.user-has-permission”.“user” = $1
Feb 28 18:22:20 986d619be1ae api[1830]: AND (“permission.user-has-permission”.“expiry date” IS NULL
Feb 28 18:22:20 986d619be1ae api[1830]: OR “permission.user-has-permission”.“expiry date” > CURRENT_TIMESTAMP)
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: OR EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “role-has-permission” AS “permission.role-has-permission”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission”.“id” = “permission.role-has-permission”.“permission”
Feb 28 18:22:20 986d619be1ae api[1830]: AND EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “role” AS “permission.role-has-permission.role”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission.role-has-permission”.“role” = “permission.role-has-permission.role”.“id”
Feb 28 18:22:20 986d619be1ae api[1830]: AND EXISTS (
Feb 28 18:22:20 986d619be1ae api[1830]: SELECT 1
Feb 28 18:22:20 986d619be1ae api[1830]: FROM “user-has-role” AS “permission.role-has-permission.role.user-has-role”
Feb 28 18:22:20 986d619be1ae api[1830]: WHERE “permission.role-has-permission.role”.“id” = “permission.role-has-permission.role.user-has-role”.“role”
Feb 28 18:22:20 986d619be1ae api[1830]: AND “permission.role-has-permission.role.user-has-role”.“user” = $2
Feb 28 18:22:20 986d619be1ae api[1830]: AND (“permission.role-has-permission.role.user-has-role”.“expiry date” IS NULL
Feb 28 18:22:20 986d619be1ae api[1830]: OR “permission.role-has-permission.role.user-has-role”.“expiry date” > CURRENT_TIMESTAMP)
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: )
Feb 28 18:22:20 986d619be1ae api[1830]: ))
Feb 28 18:22:20 986d619be1ae api[1830]: ORDER BY “permission”.“name” ASC [ 2, 2 ]
Feb 28 18:22:48 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:22:51 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:22:56 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:23:04 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:23:37 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:23:55 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:24:22 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:24:53 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:25:23 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:25:56 986d619be1ae api[1830]: resin: Parsing GET /resin/user(2)?$select=actor,jwt_secret
Feb 28 18:27:27 986d619be1ae api[1830]: resin: Parsing GET /resin/device/$count?$filter=uuid eq ‘a40bd2149d456174f3235d62c51fe82d’
Feb 28 18:42:32 986d619be1ae api[1830]: Program node index.js exited with code null

rlev · March 2, 2019, 7:38pm

@richbayliss,
Never mind I found out the reason of crashing was because my server had a poor hardware performance. Once upgraded the services runs fine.

Thanks