I have a simple node app running in balena and the SIGTERM event doesn’t seem to be propagated to my Javascript code.
I am running on a Raspberry Pi 3 with Docker image balenalib/%%BALENA_ARCH%%-debian-node:10.16-jessie-run
I have ENV INITSYSTEM=on in my Dockerfile.template.
I created this project last week, so should have pretty recent versions of the Balena stuff.
Here is the code in my main JS file that doesn’t seem to be hit.
process.on(‘SIGTERM’, () => {
console.log(‘SIGTERM signal received, shutting down…’)
// bunch of other de-init code here
process.exit(0)
}
Just to add some context, how is SIGTERM being sent to the process? For example, typing CTRL-C sends SIGINT rather than SIGTERM, I think. If you run "kill -15 <PID>", where is the process ID that can be list with the “ps” command (perhaps “ps -Af”), does it work? Is SIGTERM being sent “within” the app container, or from the host OS, or from another container? Or are you perhaps trying to intercept the “stop” command on the web dashboard?
Also worth pointing out: if you are running an initsystem (if you need / want it), check the documentation sections “Major Changes” and “Installing your own Initsystem” on the following page:
Perhaps you intended to capture SIGTERM described on the following page?
Actions and Settings - Balena Documentation
" When the containers are stopped, the application is politely asked to stop by sending a SIGTERM and after 10 seconds of wait time a SIGKILL is sent."
To help us reproduce the problem, if you could share a minimum docker-compose.yml (if you have one) and/or Dockerfile.template file that exhibits the issue, it would be great. If they are large-ish, you could share a link to Gist.
I would assume that when Balena restarts/kills a service, my Node application would receive a SIGTERM.
Here is my gist: https://gist.github.com/iheart2code/74c5c6421f4ca9e6ba83c5c597f4a694
Thanks for sharing your docker-compose and Dockerfile. I’ve run some tests (link to my own Gist files) and I think I know what’s going on: the CMD ["npm", "start"] command is creating a chain of 3 processes where one or two of the parent processes don’t propagate the SIGTERM signal to the final node child process. Opening a shell on the app container and running the ps command (on a Raspberry Pi 3) I see:
root@6983a3f:/usr/src/app# ps -Af
UID PID PPID C STIME TTY TIME CMD
root 1 0 8 01:14 pts/0 00:00:03 npm
root 22 1 0 01:14 pts/0 00:00:00 sh -c node index.js
root 23 22 1 01:14 pts/0 00:00:00 node index.js
Clicking the stop or restart buttons on the web dashboard and watching the Logs window indeed shows that the SIGTERM is “missed” by the the node index.js process (3rd process).
However if I replace the CMD instruction with CMD ["/usr/local/bin/node", "index.js"], then there is a single process (rather than the previous 3) and the SIGTERM is captured:
root@6983a3f:/usr/src/app# ps -Af
UID PID PPID C STIME TTY TIME CMD
root 1 0 1 01:19 pts/0 00:00:01 /usr/local/bin/node index.js
# Logs window
06.07.19 02:25:44 (+0100) main hello world
06.07.19 02:25:51 (+0100) Killing service 'main sha256:6b3dbdcce3cfda00673537266746e550567202f7a05a719b10c746d57e178133'
06.07.19 02:25:52 (+0100) main SIGTERM signal received, shutting down…
The issue with npm start may be this npm issue, or it may be the sh process that is not propagating the signal.
By the way, I understand that the ENV INITSYSTEM=on instruction in your Dockerfile.template is unnecessary and is ignored in the balenalib images (unlike the previous resin images). To actually use an init system like systemd (which you may not need anyway), you would actually have to install it and it’s not that straightforward. Unless you have a specific need for an init system like systemd, you’re probably better off without one. If you wanted an init system just to forward signals and reap zombies, conside tini. Note that even with an init system, I think you would still have the "npm start" issue all the same.
This did indeed fix my issue. Replaced CMD with /usr/local/bin/node instead of npm, and was able to remove privileged flag & ENV INITSYSTEM=on from the containers.
Thanks so much for the help and detailed description!