PM2-RUNTIME receives [ERR_IPC_DISCONNECTED] all the time in Docker

Product support balenaOS
1

I am running a node.js application on a RaspberryPi0-WiFi. The application has two docker containers, one for FTP processing and one for the actually application which is connected to an OLED screen for visual output.

We need to use PM2 because our application collects statistics and those are then being broadcast to the PM2-PLUS service for our engineers to monitor the application with.

My problem, is that if the device is powered-off, then back on again, we suddenly start to receive the following error message.

events.js:170
18.10.19 03:07:34 (+0100) throw er; // Unhandled ‘error’ event
18.10.19 03:07:34 (+0100) ^
18.10.19 03:07:34 (+0100)
18.10.19 03:07:34 (+0100) Error [ERR_IPC_DISCONNECTED]: IPC channel is already disconnected
18.10.19 03:07:34 (+0100) at ChildProcess.target.disconnect (internal/child_process.js:804:26)
18.10.19 03:07:34 (+0100) at Timeout._ [as onTimeout] (/usr/local/lib/node_modules/pm2/node_modules/@pm2/agent/src/InteractorClient.js:233:13)
18.10.19 03:07:34 (+0100) at listOnTimeout (internal/timers.js:535:17)
18.10.19 03:07:34 (+0100) at processTimers (internal/timers.js:479:7)
18.10.19 03:07:34 (+0100) Emitted ‘error’ event at:
18.10.19 03:07:34 (+0100) at ChildProcess.target.disconnect (internal/child_process.js:804:12)
18.10.19 03:07:34 (+0100) at Timeout. [as _onTimeout] (/usr/local/lib/node_modules/pm2/node_modules/@pm2/agent/src/InteractorClient.js:233:13)
18.10.19 03:07:34 (+0100) at listOnTimeout (internal/timers.js:535:17)
18.10.19 03:07:34 (+0100) at processTimers (internal/timers.js:479:7)

Has anyone been successful in running PM2 with Docker.

Only the first time that the application runs after an update does it work. If you power off the device and power it back on again, the error appears. Am I missing something when it restart?

Help appreciated.

I have changed the CMD and I still get the same error.

OLD:
[“pm2-runtime”, “ecosystem.config.js”, “–no-deamon”]

NEW:
CMD [“pm2”, “start”, “FoxyBox.js”,“–no-daemon”]

It worked for the first couple of power-cycles but then the ERR_IPC_DISCONNECTED error happened again and the application then keeps going in to a constant re-start cycle. Any ideas?

2

Hey, my assumption would be that pm2 is not correctly clearing up resources on exit. I would try to catch the SIGTERM that docker sends, and force pm2 to exit cleanly. It would probably require you to add a shell script as your CMD and catch the signal there using trap.

I also found these related issues, which seem to agree with what my assumptions, and they might be of further help: https://github.com/Unitech/pm2/issues/2061 and https://github.com/Unitech/pm2/issues/2043

Let me know if this helps!

4

You are right. After a lot of digging it seems to be resource related. Our device can suddenly loose power so there will be no option to catch the event.

I have changed the CMD as follows:

COPY docker-entrypoint.sh /usr/local/bin/
RUN ln -s /usr/local/bin/docker-entrypoint.sh
&& chmod +x /usr/local/bin/docker-entrypoint.sh
WORKDIR /app
CMD [“docker-entrypoint.sh”]

This runs a script that resets PM2 and starts our application.

#!/bin/bash

echo ‘Resetting PM2 Metadata …’
pm2 reset all
echo ‘Calling pm2-runtime …’
pm2-runtime ecosystem.config.js --no-deamon --name MyApp
echo ‘pm2 now called’

ecosystem.config.js is as follows…

module.exports = [{
script: ‘MyApp.js’,
name: ‘MyApp’,
watch: false,
autorestart: false,
wait_ready: true,
noDaemonMode: true,
exec_mode: ‘fork’,
instances: 1
}]

After doing this, when power is lost while the app is running, when we switch it back on again, everything appears to work as normal. It took a few weeks to understand this, but so far, so good.

Replace “MyApp” with your own names/scripts. I am still playing around with the best PM2 configuration, but for now it solves the resource problem.

Thanks.