Combining Balena and Azure IoT Edge

Project help
1

Hi!

I’m new to the forums and quite new to Balena as well. I’ve been experimenting with Balena on a Raspberry Pi 2 and so far I’m a fan!

I’m looking in to basing my IoT infrastructure on Balena and at the same time tying in to Azure services using their Azure IoT Edge runtime. Azure IoT Edge is a container-based service building on Azure IoT Hub enabling managed deployment of Azure services as well as custom code to fleets of connected devices. It provides some neat features through their IoT SDK such as agnostic configuration of communication between services and the upstream IoT Hub in the cloud. You can read about it here: https://docs.microsoft.com/en-us/azure/iot-edge/

As nice as that may seem, it is nowhere near the Balena in terms of providing a complete solution to device management (OS updates, connectivity set up, VPN/SSH). Microsoft instructions on updating the IoT Edge runtime itself basically involves running apt commands, which would require one to access the command line through e.g. SSH to actually execute this on every device to be updated - not the most convenient approach. They use systemctl (systemd) to run the IoT Edge runtime itself.

With this in mind I thought an ideal solution could be to:

  • Use Balena to access great functionality for device management, including the ability to update the OS
  • Run Azure IoT Edge runtime in a container deployed and updated through Balena
  • Run and manage Azure-based applications within the Azure IoT Edge runtime container

Has anyone here considered such a set up? Does this seem like a reasonable approach to using Azure IoT Edge with Balena, or am I over-complicating things?

I started out toying around with a Debian stretch base image to run IoT Edge in, installing it as recommended in the Azure documents within a Dockerfile. I made some headway and managed to run the IoT Edge daemon. However, it complains about unix:///var/run/iotedge/mgmt.sock and exits. Anyone more experienced with Docker and such that could point me in a good direction?

Best regards
Daniel

1 Like
2

Hey @stenbergd

Glad to hear you’re enjoying balena!

We have some documentation for exactly this, located here: https://www.balena.io/docs/learn/develop/integrations/azure/

Note that a github redirect is not working on that page for some reason, and the link should be https://github.com/balena-io-projects/balena-azure-iot-remote-monitoring for the example code. I’ve notified our docs maintainer and this should be fixed soon.

3

Thank you for your reply Cameron.

I’m not sure we’re talking about the same thing here. What you’re referring to is integration with Azure IoT Suite (I think its being referred to as Azure IoT Accelerators nowadays), which is a preconfigured Azure IoT solution. Azure IoT Edge is quite a different concept.

I found this old Resin blog post on using Azure IoT Edge V1, which has since been superseeded by V2 and became generally available last year. Since V2 is quite different from V1, the blog post is not of any real use now.
Link: https://www.balena.io/blog/move-to-the-edge-with-azure-iot-and-resin-io/

It would be great with some similar guide on using Azure IoT Edge V2 with Balena.

4

Hi @stenbergd - we weren’t aware of that IoT Edge V2 release. It looks interesting, and I think your proposed approach of using balena as a way to deploy the runtime and do device management makes a lot of sense.

We’re gonna start looking into building a demo project that does this, but it might take a while. In the meantime, maybe we can help debug your implementation? Could you post the full logs when you get that error message about mgmt.sock? It would also help to see your Dockerfile, and docker-compose.yml if you’re doing multicontainer, if you want to share it.

I suppose you’re either starting another container engine (docker?) inside the container, or accessing the host’s balenaEngine socket?

Cheers!

1 Like
5

Hi! No worries, I find it impossible to keep up with all the latest developments on the IoT scene myself! IoT Edge V2 became generally available last summer.

That sounds very promising! I’m no Docker wizz myself, but got some rudimentary understanding and experience of using it. I’m only doing single application from a Balena perspective, but it is as you say: it’s IoT Edge Docker/Moby engine inside Balena Docker/Moby engine :slight_smile:

I think I got it a bit mixed up with my description earlier, it is not an mgmt.sock error (think I got this from some initial research on the issue). Here’s the printout when attempting to start the iotedged daemon.

root@5b1f0a4:/# iotedged -c /etc/iotedge/config.yaml
<6>2019-01-20T16:45:03Z [INFO] - Starting Azure IoT Edge Security Daemon
<6>2019-01-20T16:45:03Z [INFO] - Version - 1.0.5 (d76e0316c6f324345d77c48a83ce836d09392699)
<6>2019-01-20T16:45:03Z [INFO] - Using config file: /etc/iotedge/config.yaml
<6>2019-01-20T16:45:04Z [INFO] - Using runtime network id azure-iot-edge
<3>2019-01-20T16:45:04Z [ERR!] - The daemon could not start up successfully: Could not initialize module runtime
<3>2019-01-20T16:45:04Z [ERR!] -        caused by: Could not initialize module runtime
<3>2019-01-20T16:45:04Z [ERR!] -        caused by: Invalid URL "unix:///var/run/docker.sock": Socket file could not be found

I defined a Dockerfile.template as:

# Dockerfile to run Azure IoT Edge on Balena OS devices

FROM balenalib/%%BALENA_MACHINE_NAME%%-debian:stretch

RUN apt-get update && apt-get install -y \
  iptables \
 && rm -rf /var/lib/apt/lists/*

# Download and install the moby-engine
RUN curl -L https://aka.ms/moby-engine-armhf-latest -o moby_engine.deb && sudo dpkg -i ./moby_engine.deb

# Download and install the moby-cli
RUN curl -L https://aka.ms/moby-cli-armhf-latest -o moby_cli.deb && sudo dpkg -i ./moby_cli.deb

# Run apt-get fix
RUN sudo apt-get install -f

# To avoid error message
RUN echo exit 0 > /usr/sbin/policy-rc.d

# Download and install the standard libiothsm implementation
RUN curl -L https://aka.ms/libiothsm-std-linux-armhf-latest -o libiothsm-std.deb && sudo dpkg -i ./libiothsm-std.deb

# Download and install the IoT Edge Security Daemon
RUN curl -L https://aka.ms/iotedged-linux-armhf-latest -o iotedge.deb && sudo dpkg -i ./iotedge.deb

# Run apt-get fix
RUN sudo apt-get install -f

ENV INITSYSTEM on

CMD sleep infinity

As I’m using a Raspberry Pi 2, I’ve created the Dockerfile.template based on the installation instructions for Linux ARM32 found here: https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-linux-arm

I inserted the sleep infinity at the end so that I could SSH in to the running container and play around with the iotedge CLI and daemon manually.

Within the IoT Edge Dev Tool repository, I actually found some files that may help me in my use case. I haven’t had the time to try it out yet. I also found some issues where other developers have asked about running IoT Edge in a Docker container, but it seems Microsoft are stressing that it should not be used in production scenarios (don’t know the exact reason for this…)

Here’s the files I was referring to, will come back with more info when I’ve tried it out myself: https://github.com/Azure/iotedgedev/tree/master/docker/runtime/arm

I also found a Yocto layer for IoT Edge which could (technically) be an option. Not sure about Balena support for adding custom Yocto layers to a base OS and I’m a complete Yocto novice.

6

I’ve put some more effort in to this trying to adapt the Microsoft instructions to Balena. However, I’m still getting the docker.sock error: Invalid URL "unix:///var/run/docker.sock": Socket file could not be found

Looking at the suggested launch script for the Docker container, it seems I need to volume mount the *.sock file (from host /var/run/balena.sock ?) to the running container, forward a couple of ports, run as a network bridge, set env variables etc.

Is there an equivalent to achieving this in Balena, exposing the same functionality as docker run. Environment variables seem easy from the Balena dashboard, but what about the rest?

This is what I’m looking to do with Balena instead of Docker:

docker run \
    -i \
    -t \
    --rm \
    -v //var//run//docker.sock://var//run//docker.sock \
    -p 15580:15580 \
    -p 15581:15581 \
    --network bridge \
    --name iotedgec \
    -e IOT_DEVICE_CONNSTR="$IOT_DEVICE_CONNSTR" \
    iot-edge-c
7

Also Interested in this Solution, since we have a Similar request at work

2 Likes
8

@stenbergd @pcarranzav do you have any news about IoT Edge v2 ? Thanks

10

@stenbergd we have a docker compose label that you can add, that will cause the supervisor to mount the balena-engine socker into the container, and provide an address with the DOCKER_HOST environment variable.

@lpierrat I’m not aware on any progress on this, but I’ll reach out to the team to see if I can get anymore information for you.

14

Unfortunately, I haven’t been able to pursue this further. My next step would probably be to to experiment with docker compose as @CameronDiver suggests. However, I don’t feel there’s a clear way forward. It seems Microsoft assumes that one should run their IoT Edge solution directly in the host OS and currently only have “production level” support for Raspbian OS for ARM32v7 platforms such as Raspberry Pi (https://docs.microsoft.com/en-us/azure/iot-edge/support)

I’m glad more people seem interested in this and I hope this fact can move things along.

17

Azure integration is a reason why i would go for Balena. I like OS management. But for me is key the integration with Azure Edge V2. Is there a doc I can follow to try it? From that tutorial i do not understand where the IoT Edge connection string to IoT Hub is configured- Please advice.

18

Between I am trying to follow the tutorial. When I push for building i get this error:

…
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease Unable to find expected entry ‘main/binary-armhf/Packages’ in Release file (Wrong sources.list entry or malformed file)
[main] E: Some index files failed to download. They have been ignored, or old ones used instead.
[main]
[main] Removing intermediate container 4d1f13d07938
[Info] Uploading images
[main] The command ‘/bin/sh -c apt-get update’ returned a non-zero code: 100
[Success] Successfully uploaded images
[Error] Some services failed to build:
[Error] Service: main
[Error] Error: The command ‘/bin/sh -c apt-get update’ returned a non-zero code: 100
[Error] Not deploying release.
Remote build failed

Any hints?

20

@banto_78 it looks like the problem that you’re having there is because of debian dropping the jessie-updates repository: https://github.com/balena-io-library/base-images/issues/532

You can fix it by updating to a newer base image, or alternatively running RUN sed -i '/jessie-updates/{s/^/#/}' /etc/apt/sources.list before performing an apt update - but I’d certainly recommend updating instead.

There currently isn’t an Azure Edge v2 integration document or tutorial at the moment.

22

Many thanks. I think I’ve anyway resolved by updating to:

FROM balenalib/raspberrypi3-python:3.6-jessie-build

I have built and deployed. However I do not see any message uploaded in Azure IoT Hub…and by the way where can i find the `image.png’ used in the script? This is in the log view:

01.07.19 18:05:10 (+0200) main convert: improper image header image.png' @ error/png.c/ReadPNGImage/3943. 01.07.19 18:05:10 (+0200) main convert: no images definedrotated.png’ @ error/convert.c/ConvertImageCommand/3210.
01.07.19 18:05:11 (+0200) main using “DejaVu Sans Mono-16”, pixelsize=16.67 file=/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf

31

@banto_78 That is just any image, and is used as a small example, I’d just add something simple yourself.

This also may be of interest, but I’ve not used it, so YMMV https://github.com/maxtheaviator/resin-azure-iotedge-multicontainer

33

ok. The script start.sh has different bugs. For example " replaced.json" it is produced as empty file. Any idea?

34

My assumption would be that the file it reads this from cat /usr/src/app/iot-edge/samples/simulated_device_cloud_upload/src/simulated_device_cloud_upload_lin.json is empty, can you confirm?

35

it is not empty that file. I can prove that the result of this:

echo "$J_FILE" | \  
  jq '.modules[0].args.IoTHubName="<IoTHubName>"' | \
  jq '.modules[0].args.IoTHubSuffix="azure-devices.net"' | \
  jq '.modules[0].args.Transport="AMQP"' | \
  jq '.modules[1].args[0].deviceId="'$DEVICE1'"' | \
  jq '.modules[1].args[0].deviceKey="'$DEVICE1KEY'"' | \
  jq '.modules[1].args[1].deviceId="'$DEVICE2'"' | \
  jq '.modules[1].args[1].deviceKey="'$DEVICE2KEY'"' \
  > replaced.json

is an empty file!

37

Hey, so it turns out that the repo structure for the iot-edge example app from Azure has changed since the article’s script was written. This is why the files show up as non-existent. You should replace instances of the iot-edge directory with iot-edge/v1 in the Dockerfile and start.sh.

41

Yes, I already found out the issue with “V1” in the path. Still, that script produces a wrong (not replacing the values) or empty file. I am wondering because the structure of the JSON (https://github.com/Azure/iot-edge-v1/blob/master/v1/samples/simulated_device_cloud_upload/src/simulated_device_cloud_upload_lin.json ) should be fine… Any thoughts?