Device is online, VPN only and services don't appear to be working.

Hi,

I have an application running with a remote device that has just been turned on. This device is online, but has only VPN and is not connecting to my azure IoT hub. This connection is done by the node-red flows that are running in the node-red service.

I have spoken to my colleague who was on site earlier on who said the unit did not appear to be doing anything (there should be LEDs controlled by the GPIOs which were all off).

It is almost as if the node-red service is not running at all. I have tried restarting the services but nothing changed.

We do not currently have physical access to the device, but I can access the shell through my balena login.

I’m not sure what I’m best to do in this situation as I have 2 other devices running perfectly on the same application.

If I can work out what the issue is I may be able to schedule a repair.

The UUID is b14588acf1e139ba61cf05040a014bb0.

I have enabled support access.

Thanks,
Gregor

Hi there, I see the device as online now.

Connecting to the device and accessing the logs manually I get the following info

$ balena ps 
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                 PORTS                    NAMES
8a666a943163        f23c143a3241                         "/usr/bin/entry.sh b…"   2 hours ago         Up 2 hours             0.0.0.0:80->1880/tcp     node-red_1588063_1105193
19c4878fb8ee        e79cc5f74ac6                         "/usr/bin/entry.sh b…"   2 hours ago         Up 17 seconds          0.0.0.0:8888->8888/tcp   pigpiod_1588064_1105193
35c6678c216e        balena/aarch64-supervisor:v10.6.27   "./entry.sh"             7 months ago        Up 2 hours (healthy)                            resin_supervisor

$ balena logs node-red_1588063_1105193
15 Sep 17:27:45 - [error] [azureiothub:Azure IoT Hub] Could not connect: AMQP Transport: Could not connect
Unhandled rejection Error: AMQP Transport: Could not connect
    at translateError (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-amqp-base/lib/amqp_common_errors.js:55:13)
    at translateError (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-device-amqp/lib/amqp_device_errors.js:20:13)
    at /usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-device-amqp/lib/amqp.js:43:12
    at Amqp.<anonymous> (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/azure-iot-amqp-base/lib/amqp.js:111:19)
    at tryCatcher (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/node-red-contrib-azure-iot-hub/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:439:21)

$ balena logs pigpiod_1588064_1105193
2020-09-15 17:29:33 initPeripherals: mmap gpio failed (Operation not permitted)
Can't initialise pigpio library
... daemon stopped unexpectedly.
netstat -tlep | grep 8888
sleeping for 20 sec - we hope that this releases the bound TCP port 8888
end of sleep
netstat -tlep | grep 8888
... start.sh terminated
Checking who is listening/bound to port 8888 ...
...OK, no process is listing/bound to port 8888
remove existing pigpio file
Starting daemon pigpiod ...
2020-09-15 17:29:57 initPeripherals: mmap gpio failed (Operation not permitted)
Can't initialise pigpio library
... daemon stopped unexpectedly.
netstat -tlep | grep 8888
sleeping for 20 sec - we hope that this releases the bound TCP port 8888
end of sleep
netstat -tlep | grep 8888
... start.sh terminated

It looks like the node-red service is failing to connect to Azure and, probably due to this failed connection, the pigpiod service is crash looping, this would explain you not being able to see any activity from the LEDs

Since the application is working on your other devices and we have access to the device through VPN, I would recommend trying to diagnose the Azure connection problems, It might be a firewall issue, or some other network problem.

Hi pipex,

I’ve managed to solve the Azure connection issue. However, I think the pigpio is still not running as it doesn’t seem to be interfacing with the hardware on site.

The telemetry we have received is suggesting that the pins are not functioning at all.

I have also received a warning that this device has used up 4GB of data in the last 2 days. I cannot see how this is possible unless something is drastically wrong as the other units in the same application are using around 300MB per month.

I’ll re-enable support access if you could check out the data issue.

Thanks,
Gregor

Since fixing the Azure connection issue, my data usage appears to have slowed right down.

I am still recieving telemetry data, but it still doesn’t appear that the GPIOs are working.

Hi again Gregor,

It makes sense that the usage was spiking since the device was constantly trying to connect o Azure.

With regards to the gpio, it seems that the pigpio continues to crash with this message

Starting daemon pigpiod ...
2020-09-17 21:59:30 initPeripherals: mmap gpio failed (Operation not permitted)
Can't initialise pigpio library

Have you had any success using the same service version on another device? Could it be a problem with the hardware?

Felipe

Hi pipex,

I managed to solve the issue in the office by creating a new application for Pi3 (32bit) and it works perfectly on that. I remembered that I’d had a similar issue when I was creating the other devices and had solved it by using a 32bit base image.

I’ll recreate the device with a 32bit OS and get it sent out to site.

Thanks,
Gregor