Combining Balena and Azure IoT Edge


#1

Hi!

I’m new to the forums and quite new to Balena as well. I’ve been experimenting with Balena on a Raspberry Pi 2 and so far I’m a fan!

I’m looking in to basing my IoT infrastructure on Balena and at the same time tying in to Azure services using their Azure IoT Edge runtime. Azure IoT Edge is a container-based service building on Azure IoT Hub enabling managed deployment of Azure services as well as custom code to fleets of connected devices. It provides some neat features through their IoT SDK such as agnostic configuration of communication between services and the upstream IoT Hub in the cloud. You can read about it here: https://docs.microsoft.com/en-us/azure/iot-edge/

As nice as that may seem, it is nowhere near the Balena in terms of providing a complete solution to device management (OS updates, connectivity set up, VPN/SSH). Microsoft instructions on updating the IoT Edge runtime itself basically involves running apt commands, which would require one to access the command line through e.g. SSH to actually execute this on every device to be updated - not the most convenient approach. They use systemctl (systemd) to run the IoT Edge runtime itself.

With this in mind I thought an ideal solution could be to:

  • Use Balena to access great functionality for device management, including the ability to update the OS
  • Run Azure IoT Edge runtime in a container deployed and updated through Balena
  • Run and manage Azure-based applications within the Azure IoT Edge runtime container

Has anyone here considered such a set up? Does this seem like a reasonable approach to using Azure IoT Edge with Balena, or am I over-complicating things?

I started out toying around with a Debian stretch base image to run IoT Edge in, installing it as recommended in the Azure documents within a Dockerfile. I made some headway and managed to run the IoT Edge daemon. However, it complains about unix:///var/run/iotedge/mgmt.sock and exits. Anyone more experienced with Docker and such that could point me in a good direction?

Best regards
Daniel


#2

Hey @stenbergd

Glad to hear you’re enjoying balena!

We have some documentation for exactly this, located here: https://www.balena.io/docs/learn/develop/integrations/azure/

Note that a github redirect is not working on that page for some reason, and the link should be https://github.com/balena-io-projects/balena-azure-iot-remote-monitoring for the example code. I’ve notified our docs maintainer and this should be fixed soon.


#3

Thank you for your reply Cameron.

I’m not sure we’re talking about the same thing here. What you’re referring to is integration with Azure IoT Suite (I think its being referred to as Azure IoT Accelerators nowadays), which is a preconfigured Azure IoT solution. Azure IoT Edge is quite a different concept.

I found this old Resin blog post on using Azure IoT Edge V1, which has since been superseeded by V2 and became generally available last year. Since V2 is quite different from V1, the blog post is not of any real use now.
Link: https://www.balena.io/blog/move-to-the-edge-with-azure-iot-and-resin-io/

It would be great with some similar guide on using Azure IoT Edge V2 with Balena.


#4

Hi @stenbergd - we weren’t aware of that IoT Edge V2 release. It looks interesting, and I think your proposed approach of using balena as a way to deploy the runtime and do device management makes a lot of sense.

We’re gonna start looking into building a demo project that does this, but it might take a while. In the meantime, maybe we can help debug your implementation? Could you post the full logs when you get that error message about mgmt.sock? It would also help to see your Dockerfile, and docker-compose.yml if you’re doing multicontainer, if you want to share it.

I suppose you’re either starting another container engine (docker?) inside the container, or accessing the host’s balenaEngine socket?

Cheers!


#5

Hi! No worries, I find it impossible to keep up with all the latest developments on the IoT scene myself! IoT Edge V2 became generally available last summer.

That sounds very promising! I’m no Docker wizz myself, but got some rudimentary understanding and experience of using it. I’m only doing single application from a Balena perspective, but it is as you say: it’s IoT Edge Docker/Moby engine inside Balena Docker/Moby engine :slight_smile:

I think I got it a bit mixed up with my description earlier, it is not an mgmt.sock error (think I got this from some initial research on the issue). Here’s the printout when attempting to start the iotedged daemon.

root@5b1f0a4:/# iotedged -c /etc/iotedge/config.yaml
<6>2019-01-20T16:45:03Z [INFO] - Starting Azure IoT Edge Security Daemon
<6>2019-01-20T16:45:03Z [INFO] - Version - 1.0.5 (d76e0316c6f324345d77c48a83ce836d09392699)
<6>2019-01-20T16:45:03Z [INFO] - Using config file: /etc/iotedge/config.yaml
<6>2019-01-20T16:45:04Z [INFO] - Using runtime network id azure-iot-edge
<3>2019-01-20T16:45:04Z [ERR!] - The daemon could not start up successfully: Could not initialize module runtime
<3>2019-01-20T16:45:04Z [ERR!] -        caused by: Could not initialize module runtime
<3>2019-01-20T16:45:04Z [ERR!] -        caused by: Invalid URL "unix:///var/run/docker.sock": Socket file could not be found

I defined a Dockerfile.template as:

# Dockerfile to run Azure IoT Edge on Balena OS devices

FROM balenalib/%%BALENA_MACHINE_NAME%%-debian:stretch

RUN apt-get update && apt-get install -y \
  iptables \
 && rm -rf /var/lib/apt/lists/*

# Download and install the moby-engine
RUN curl -L https://aka.ms/moby-engine-armhf-latest -o moby_engine.deb && sudo dpkg -i ./moby_engine.deb

# Download and install the moby-cli
RUN curl -L https://aka.ms/moby-cli-armhf-latest -o moby_cli.deb && sudo dpkg -i ./moby_cli.deb

# Run apt-get fix
RUN sudo apt-get install -f

# To avoid error message
RUN echo exit 0 > /usr/sbin/policy-rc.d

# Download and install the standard libiothsm implementation
RUN curl -L https://aka.ms/libiothsm-std-linux-armhf-latest -o libiothsm-std.deb && sudo dpkg -i ./libiothsm-std.deb

# Download and install the IoT Edge Security Daemon
RUN curl -L https://aka.ms/iotedged-linux-armhf-latest -o iotedge.deb && sudo dpkg -i ./iotedge.deb

# Run apt-get fix
RUN sudo apt-get install -f

ENV INITSYSTEM on

CMD sleep infinity

As I’m using a Raspberry Pi 2, I’ve created the Dockerfile.template based on the installation instructions for Linux ARM32 found here: https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-linux-arm

I inserted the sleep infinity at the end so that I could SSH in to the running container and play around with the iotedge CLI and daemon manually.

Within the IoT Edge Dev Tool repository, I actually found some files that may help me in my use case. I haven’t had the time to try it out yet. I also found some issues where other developers have asked about running IoT Edge in a Docker container, but it seems Microsoft are stressing that it should not be used in production scenarios (don’t know the exact reason for this…)

Here’s the files I was referring to, will come back with more info when I’ve tried it out myself: https://github.com/Azure/iotedgedev/tree/master/docker/runtime/arm

I also found a Yocto layer for IoT Edge which could (technically) be an option. Not sure about Balena support for adding custom Yocto layers to a base OS and I’m a complete Yocto novice.


#6

I’ve put some more effort in to this trying to adapt the Microsoft instructions to Balena. However, I’m still getting the docker.sock error: Invalid URL "unix:///var/run/docker.sock": Socket file could not be found

Looking at the suggested launch script for the Docker container, it seems I need to volume mount the *.sock file (from host /var/run/balena.sock ?) to the running container, forward a couple of ports, run as a network bridge, set env variables etc.

Is there an equivalent to achieving this in Balena, exposing the same functionality as docker run. Environment variables seem easy from the Balena dashboard, but what about the rest?

This is what I’m looking to do with Balena instead of Docker:

docker run \
    -i \
    -t \
    --rm \
    -v //var//run//docker.sock://var//run//docker.sock \
    -p 15580:15580 \
    -p 15581:15581 \
    --network bridge \
    --name iotedgec \
    -e IOT_DEVICE_CONNSTR="$IOT_DEVICE_CONNSTR" \
    iot-edge-c

#7

Also Interested in this Solution, since we have a Similar request at work