Help to debug Nodejs/Docker/raspberrypi3 error: invalid ELF header

I am new to Balena, and it seems like a super cool platform. But I really hope you can help me out debug this error - I have been stuck for days now.

I am trying to convert a nodejs project to Balena. It uses the rpio library. This library requires GCC support, and in the dockerfile I install this without any issues:

RUN apt-get update && apt-get install -yq \
alsa-utils libasound2-dev gcc-4.8 g++-4.8 && \
apt-get clean && rm -rf /var/lib/apt/lists/*

The dockerfile builds without any problems in local mode. However when the process then starts, I get the following error (invalid ELF header):

17:50:00 - error: uncaughtException: /usr/src/app/node_modules/rpio/build/Release/rpio.node: invalid ELF header

What can I try to fix this problem - I have now been stuck for days trying different things.

What I did to debug the issue
According to my Google searches, this is related to build architecture. I thus added the uname -a command to both the Dockerfile and a prerunner script in the nodeproject:

(Dockerfile)Linux 96cbf2e6ef3d 4.14.68 #1 SMP Tue Nov 27 09:53:28 UTC 2018 armv7l GNU/Linux
(Node  Linux 2cfca32 4.14.68           #1 SMP Tue Nov 27 09:53:28 UTC 2018 armv7l GNU/Linux

But as I can tell this match pretty good. I added the following resources if needed:

Dockerfile

FROM resin/raspberrypi3-node:8

# use apt-get if you need to install dependencies,
# for instance if you need ALSA sound utils, just uncomment the lines below.
RUN apt-get update && apt-get install -yq \
   alsa-utils libasound2-dev gcc-4.8 g++-4.8 && \
   apt-get clean && rm -rf /var/lib/apt/lists/*

RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50 && \
   update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50

RUN gcc --version
# Defines our working directory in container
WORKDIR /usr/src/app

# Copies the package.json first for better cache on later pushes
COPY package.json package.json
# COPY package-lock.json package-lock.json

# This install npm dependencies on the balena build server,
# making sure to clean up the artifacts it creates in order to reduce the image size.
RUN JOBS=MAX npm install --production --unsafe-perm && rm -rf /tmp/*
RUN uname -a

# This will copy all files in our root to the working  directory in the container
COPY . ./

# Enable systemd init system in container
ENV INITSYSTEM on

# server.js will run when container starts up on the device
CMD ["npm", "start"]

package.json

"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "setup-log": "touch logs/app.log",
    "prestart": "bash ./deps.sh",
    "start": "npm run -s setup-log && node skaale/index.js"
},
"dependencies": {
  "cron": "^1.2.1",
  "firebase-admin": "^6.4.0",
  "moment": "^2.18.1",
  "request": "^2.88.0",
  "request-promise": "^4.2.0",
  "rimraf": "^2.6.2",
  "rpio": "^1.0.11",
  "winston": "^2.3.1",
  "winston-daily-rotate-file": "^1.4.6"
}

Full error stack

[ 'Error: /usr/src/app/node_modules/rpio/build/Release/rpio.node: invalid ELF header',
     '    at Object.Module._extensions..node (module.js:681:18)',
     '    at Module.load (module.js:565:32)',
     '    at tryModuleLoad (module.js:505:12)',
     '    at Function.Module._load (module.js:497:3)',
     '    at Module.require (module.js:596:17)',
     '    at require (internal/module.js:11:18)',
     '    at bindings (/usr/src/app/node_modules/bindings/bindings.js:81:44)',
     '    at Object.<anonymous> (/usr/src/app/node_modules/rpio/lib/rpio.js:17:34)',
     '    at Module._compile (module.js:652:30)',
     '    at Object.Module._extensions..js (module.js:663:10)',
     '    at Module.load (module.js:565:32)',
     '    at tryModuleLoad (module.js:505:12)',
     '    at Function.Module._load (module.js:497:3)',
     '    at Module.require (module.js:596:17)',
     '    at require (internal/module.js:11:18)',
     '    at Object.<anonymous> (/usr/src/app/utils/gpioHelper.js:5:12)',
     '    at Module._compile (module.js:652:30)',
     '    at Object.Module._extensions..js (module.js:663:10)',
     '    at Module.load (module.js:565:32)',
     '    at tryModuleLoad (module.js:505:12)',
     '    at Function.Module._load (module.js:497:3)',
     '    at Module.require (module.js:596:17)' ] }

@alexb Make sure in your COPY operations that you donโ€™t take over the node_modules directory from the host as your Mac is x86_64 and so the binaries there are not compatible with the Pi.

@richbayliss sorry for not pointing that out. I have the following file to help ensure that.

.dockerignore

.vscode
node_modules/
node_modules/*

and I also noticed that balena sync file checks for this:

.balena-sync.yml

// Other content of file
ignore:
  - .git
  - node_modules/

@alexb

I tried this myself and found it working after removing the compiler switching:

FROM resin/raspberrypi3-node:8

# Defines our working directory in container
WORKDIR /usr/src/app

# Copies the package.json first for better cache on later pushes
COPY package.json package.json
# COPY package-lock.json package-lock.json

# This install npm dependencies on the balena build server,
# making sure to clean up the artifacts it creates in order to reduce the image size.
RUN JOBS=MAX npm install --production --unsafe-perm && rm -rf /tmp/*
RUN uname -a

# This will copy all files in our root to the working  directory in the container
COPY . ./

# Enable systemd init system in container
ENV INITSYSTEM on

# server.js will run when container starts up on the device
CMD ["npm", "start"]

Hi @robertgzr thanks for your answer. But I am not 100% sure I understand what you mean by:

If I donโ€™t install gcc-4.8 I canโ€™t event install my dependencies in the npm install step?

@alexb , donโ€™t worry, I do not quite understand it either.
Instead I am wondering where your code actually gets compiled. You said you are doing this in โ€˜local modeโ€™ .
So how do you invoke the image creation ? What command do you use ?

I just followed the balena local development guide - so I push a new image with

sudo balena local push -s .

to a raspberry pi 3, on the local network.

@alexb we are wondering if it might be an issue with the compiler. Why are you installing a specific version ?

No particular reason other than the the README of the rpio node project mentions this?

Newer versions of node.js require you to install the GCC 4.8 packages for C++11 support. If you see compilation problems related to C++11, this is the likely cause.

But I am more than open to try a different version - maybe you have one to recommend? I also notice in their readme the following:

By default the module will use /dev/gpiomem when using simple GPIO access. To access this device, your user will need to be a member of the gpio group, and you may need to configure udev with the following rule (as root):

$ cat >/etc/udev/rules.d/20-gpiomem.rules <<EOF
SUBSYSTEM=="bcm2835-gpiomem", KERNEL=="gpiomem", GROUP="gpio", MODE="0660"
EOF

Do you know where I can configure this - or is not relevant for me?

Regarding gcc you could just install plain gcc without version. I am running gcc version 7.3.0 on my system so 4.8 is rather ancient i guess.

Some more suggestions:
try : apt-get install build-essential g++

If that does not help, try non local mode, by just git pushing to your balena app to see if there is any connection to how you push it locally.

Ok build-essential g++ also contains gcc and g++? I will try this, and Iโ€™m currently deploing to production to see if there is any difference there. Thanks for the help :raised_hands:, I will get back when I have results :roll_eyes:

just read up about it:
โ€ฆinstall it by installing the build-essential package, which is a metapackage that installs gcc (for compiling C files), g++ (for compiling C++ files), and the C library headers.
So you probably only need build-essentialsโ€ฆ

https://packages.ubuntu.com/xenial/build-essential

1 Like

@samothx so before trying the build-essential package, I created a production device and pushed the code to this device.

Here the uname -a command returned this:

(docker) Linux 96cbf2e6ef3d 4.15.0-34-generic #37~16.04.1-Ubuntu SMP Tue Aug 28 10:44:16 UTC 2018 armv8l GNU/Linux
(raspberry)  Linux 879dc73 4.14.68 #1 SMP Tue Nov 27 09:16:51 UTC 2018 armv7l GNU/Linux

But it runโ€™s the app - no invalid ELF header error :tada: :partying_face:

So that is super weird if you ask me - since the raspberry reports armv7l, but docker reports armv8l - and then it works :thinking:?

So something is going on in local mode.

I now have to test if GPIO support actually works, audio, volume control etc --> but I guess itโ€™s better to keep the forum post separated. Thanks for the help again.

I would not mind checking that out because it seems weird. Can you provide me with a minimal configuration to reproduce the problem to test this with ?

1 Like

The executables are armv8l because that is what the builders areโ€ฆ

Sure - that would be awesome! I created a repo with what I believe is the minimal exampel to reproduce.

Please notice that I did not have time to test it yet, as I am still waiting for the device to update (I deleted the dev before, when switching to prod). I will let you know when I have tested it out to confirm the minimal example reproduce/work.

thanks, I will take a look and keep you up to date with my findings

Running fine on my system
with balenaOS 2.26.0+rev1

* Streaming application logs..
systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
Detected virtualization 'other'.
Detected architecture 'arm'.
Set hostname to <09989f8>.

> debug-invalid-elf-header@1.0.0 prestart /usr/src/app
> bash ./deps.sh

Linux 09989f8 4.14.68 #1 SMP Tue Nov 6 11:47:26 UTC 2018 armv7l GNU/Linux
gcc (Debian 4.8.4-1) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

g++ (Debian 4.8.4-1) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


> debug-invalid-elf-header@1.0.0 start /usr/src/app
> node main.js

Main program starting up, platform is linux


   โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
   โ”‚                                     โ”‚
   โ”‚   Update available 5.6.0 โ†’ 6.5.0    โ”‚
   โ”‚     Run npm i -g npm to update      โ”‚
   โ”‚                                     โ”‚
   โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ