I have an application running with a remote device that has just been turned on. This device is online, but has only VPN and is not connecting to my azure IoT hub. This connection is done by the node-red flows that are running in the node-red service.
I have spoken to my colleague who was on site earlier on who said the unit did not appear to be doing anything (there should be LEDs controlled by the GPIOs which were all off).
It is almost as if the node-red service is not running at all. I have tried restarting the services but nothing changed.
We do not currently have physical access to the device, but I can access the shell through my balena login.
I’m not sure what I’m best to do in this situation as I have 2 other devices running perfectly on the same application.
If I can work out what the issue is I may be able to schedule a repair.
Connecting to the device and accessing the logs manually I get the following info
$ balena ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8a666a943163 f23c143a3241 "/usr/bin/entry.sh b…" 2 hours ago Up 2 hours 0.0.0.0:80->1880/tcp node-red_1588063_1105193
19c4878fb8ee e79cc5f74ac6 "/usr/bin/entry.sh b…" 2 hours ago Up 17 seconds 0.0.0.0:8888->8888/tcp pigpiod_1588064_1105193
35c6678c216e balena/aarch64-supervisor:v10.6.27 "./entry.sh" 7 months ago Up 2 hours (healthy) resin_supervisor
$ balena logs node-red_1588063_1105193
15 Sep 17:27:45 - [error] [azureiothub:Azure IoT Hub] Could not connect: AMQP Transport: Could not connect
Unhandled rejection Error: AMQP Transport: Could not connect
at translateError (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-amqp-base/lib/amqp_common_errors.js:55:13)
at translateError (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-device-amqp/lib/amqp_device_errors.js:20:13)
at /usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-device-amqp/lib/amqp.js:43:12
at Amqp.<anonymous> (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-amqp-base/lib/amqp.js:111:19)
at tryCatcher (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:547:31)
at Promise._settlePromise (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:604:18)
at Promise._settlePromise0 (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:649:10)
at Promise._settlePromises (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:725:18)
at _drainQueueStep (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:93:12)
at _drainQueue (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:102:5)
at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:15:14)
at processImmediate (internal/timers.js:439:21)
$ balena logs pigpiod_1588064_1105193
2020-09-15 17:29:33 initPeripherals: mmap gpio failed (Operation not permitted)
Can't initialise pigpio library
... daemon stopped unexpectedly.
netstat -tlep | grep 8888
sleeping for 20 sec - we hope that this releases the bound TCP port 8888
end of sleep
netstat -tlep | grep 8888
... start.sh terminated
Checking who is listening/bound to port 8888 ...
...OK, no process is listing/bound to port 8888
remove existing pigpio file
Starting daemon pigpiod ...
2020-09-15 17:29:57 initPeripherals: mmap gpio failed (Operation not permitted)
Can't initialise pigpio library
... daemon stopped unexpectedly.
netstat -tlep | grep 8888
sleeping for 20 sec - we hope that this releases the bound TCP port 8888
end of sleep
netstat -tlep | grep 8888
... start.sh terminated
It looks like the node-red service is failing to connect to Azure and, probably due to this failed connection, the pigpiod service is crash looping, this would explain you not being able to see any activity from the LEDs
Since the application is working on your other devices and we have access to the device through VPN, I would recommend trying to diagnose the Azure connection problems, It might be a firewall issue, or some other network problem.
I’ve managed to solve the Azure connection issue. However, I think the pigpio is still not running as it doesn’t seem to be interfacing with the hardware on site.
The telemetry we have received is suggesting that the pins are not functioning at all.
I have also received a warning that this device has used up 4GB of data in the last 2 days. I cannot see how this is possible unless something is drastically wrong as the other units in the same application are using around 300MB per month.
I’ll re-enable support access if you could check out the data issue.
I managed to solve the issue in the office by creating a new application for Pi3 (32bit) and it works perfectly on that. I remembered that I’d had a similar issue when I was creating the other devices and had solved it by using a 32bit base image.
I’ll recreate the device with a 32bit OS and get it sent out to site.