Dockerize IoT Edge v2 for BalenaOS

Hi,

I’m trying to Dockerize Iot Edge v2 so that I can run it on BelenaOS (something similar has been done for v1: https://github.com/maxtheaviator/resin-azure-iotedge-multicontainer/)

The instruction I have are :

you can create a base container using one of the OSes in https://docs.microsoft.com/en-us/azure/iot-edge/support#tier-1 , install the package into it, and set up the entrypoint + config to run it without systemd (*). Also ensure the Docker socket from the host is mounted inside the container.

() Specifically, you want to mount the homedir on the host so that it can be remounted inside modules, and you want to use UDS unix:// URIs for the listen. endpoints instead of fd:// since it won’t be running under systemd.

And this is the Dockerfile I’m building:

FROM amd64/debian

RUN apt-get update
RUN apt-get install --yes curl wget

# install balena
RUN wget --no-check-certificate https://github.com/balena-os/balena-engine/releases/download/v18.9.7/balena-engine-v18.$
RUN tar -xzf balena-engine-v18.9.7-armv7.tar.gz

# create docker "symlink"
COPY docker /usr/local/bin
RUN chmod +x /usr/local/bin/docker

# install iotedge.deb
RUN curl -L https://github.com/Azure/azure-iotedge/releases/download/1.0.8-rc3/libiothsm-std_1.0.8.rc3-1_amd64.deb -o libiothsm-std.deb
RUN dpkg -i ./libiothsm-std.deb
RUN curl -L https://github.com/Azure/azure-iotedge/releases/download/1.0.8-rc3/iotedge_1.0.8.rc3-1_amd64.deb -o iotedge.deb
RUN dpkg -i ./iotedge.deb

# edit config file for iotedge
WORKDIR /etc/iotedge
RUN sed -i '/device_connection_string/c\  device_connection_string : \"'$DEVICE_CONNECTION_STRING'\"' ./config.yaml
RUN sed -i '/docker.sock/c\  uri: \"unix:///var/run/balena.sock\"' ./config.yaml

What I’m doing here is to install the balena engine (instead of moby-engine as specified in the Microsoft documentation), install the iotedge.deb package and then update the config file with the DEVICE_CONNECTION_STRING and also use balena.sock instead of docker.sock.

I don’t know if this beginning of Dockerfile is correct. Please correct it if needed.

Can someone please explain to me particularly what do I need to do for this:

set up the entrypoint + config to run it without systemd (*). Also ensure the Docker socket from the host is mounted inside the container.

() Specifically, you want to mount the homedir on the host so that it can be remounted inside modules, and you want to use UDS unix:// URIs for the listen. endpoints instead of fd:// since it won’t be running under systemd.

I have also a `docker-compose.yml’ (taken from https://github.com/maxtheaviator/resin-azure-iotedge-multicontainer) :

version: '2.1'
services:
  iotedge:
    build:
      context: ./iotedge
    restart: 'yes'
    pid: "host"
    volumes: 
        - 'resin-data:/data'
        - /var/run/:/var/run/
    labels:
      io.resin.features.balena-socket: '1'

# persistent for reboot:
volumes: 
    resin-data:

Where I added - /var/run/:/var/run/

But I have the error:

[Error]    Could not parse compose file
[Error]      Bind mounts are not allowed
[Error]    Not deploying release.

If I don’t use the docker-compose file (just push the Dockerfile), I have the following error:

[main]     !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[main]      ERROR: No container runtime detected.
[main]      Please install a container runtime and run this install again.
[main]     !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I guess there is something to do to tell iotedge to use balena-engine instead of moby, but I don’t know how to do it.

Could you please help?
Thanks

Hi,

So when you push your compose file to balenaCloud, the label io.resin.features.balena-socket instructs the Supervisor to bind-mount the path /var/run/balena.sock in your container at runtime. It may be that the IoT edge software want’s to see /var/run/docker.sock instead, so you should symlink this in the containers entry.sh.

Please try this out and then we can try to tackle any remaining issues :+1:

Actually in my Dockerfile, the docker mentioned by the lines

COPY docker /usr/local/bin
RUN chmod +x /usr/local/bin/docker

refers to this file:

$ cat docker 
#!/bin/sh
exec /balena-engine/balena-engine "$@"

Does it help you to understand in more detail what I do?

Sure, but that is the docker binary and not the socket - the error in your message up top Bind mounts are not allowed is because you’re trying to mount /var/run from the host to get access to the docker socket, which you don’t need to do since the label instructs the supervisor to make this available in the filesystem.

Ok. So in my Dockerfile I need to add something like RUN ln -s /var/run/balena.sock /var/run/docker.sock ?

Hi,

That might work, but I suspect you’ll instead need to create a link at runtime startup as at build time, the socket descriptor won’t exist. This could be part of the entry script you run when the container starts.

Best regards, Heds

Ok. So at the end of my Dockerfile I added

COPY ./entry.sh /entry.sh
ENTRYPOINT ["/bin/bash","/entry.sh"]

and entry.sh:

#!/bin/bash
ln -s /var/run/balena.sock /var/run/docker.sock

Is that ok?

And by the way, here are all the logs I have:

$ balena push IotEdge_Aaeon
[Info]     Starting build for IotEdge_Aaeon, user antonio_bruno
[Info]     Dashboard link: https://dashboard.balena-cloud.com/apps/1490224/devices
[Info]     Building on x64_01
[Info]     Pulling previous images for caching purposes...
[Success]  Successfully pulled cache images
[iotedge]  Step 1/14 : FROM amd64/debian
[iotedge]   ---> 00bf7fdd8baf
[iotedge]  Step 2/14 : RUN apt-get update
[iotedge]  Using cache
[iotedge]   ---> 4226513b27cd
[iotedge]  Step 3/14 : RUN apt-get install --yes curl wget
[iotedge]  Using cache
[iotedge]   ---> 15c512d709a6
[iotedge]  Step 4/14 : RUN wget --no-check-certificate https://github.com/balena-os/balena-engine/releases/download/v18.9.7/balena-engine-v18.9.7-armv7.tar.gz
[iotedge]  Using cache
[iotedge]   ---> 5487f3e3f797
[iotedge]  Step 5/14 : RUN tar -xzf balena-engine-v18.9.7-armv7.tar.gz
[iotedge]  Using cache
[iotedge]   ---> 1c09dbafdd18
[iotedge]  Step 6/14 : COPY docker /usr/local/bin
[iotedge]  Using cache
[iotedge]   ---> 22689f2ade12
[iotedge]  Step 7/14 : RUN chmod +x /usr/local/bin/docker
[iotedge]  Using cache
[iotedge]   ---> 53256761e312
[iotedge]  Step 8/14 : RUN curl -L https://github.com/Azure/azure-iotedge/releases/download/1.0.8-rc3/iotedge_1.0.8.rc3-1_amd64.deb -o iotedge.deb
[iotedge]   ---> Running in 3bedf1e70a61
[iotedge]    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[iotedge]                                   Dload  Upload   Total   Spent    Left  Speed
  0     0    0 
[iotedge]      0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   620    0   620    0     0   5740      0 --:--:-- --:--:-- --:--:--  5740
[iotedge]  
  0 4928k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 4928k  100 4928k    0     0  18.6M      0 --:--:-- --:--:-- --:--:--  120M
[iotedge]  
[iotedge]   ---> 0b5196a8da61
[iotedge]  Removing intermediate container 3bedf1e70a61
[iotedge]  Step 9/14 : RUN dpkg -i ./iotedge.deb
[iotedge]   ---> Running in ab9a19649aba
[iotedge]  Selecting previously unselected package iotedge.
[iotedge]  (Reading database ... 7217 files and directories currently installed.)
[iotedge]  Preparing to unpack ./iotedge.deb ...
[iotedge]  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[iotedge]   ERROR: No container runtime detected.
[iotedge]   Please install a container runtime and run this install again.
[iotedge]  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[iotedge]  dpkg: error processing archive ./iotedge.deb (--install):
[iotedge]   new iotedge package pre-installation script subprocess returned error exit status 1
[iotedge]  
[iotedge]  Errors were encountered while processing:
[iotedge]   ./iotedge.deb
[iotedge]  
[iotedge]  Removing intermediate container ab9a19649aba
[Info]     Uploading images
[iotedge]  The command '/bin/sh -c dpkg -i ./iotedge.deb' returned a non-zero code: 1
[Success]  Successfully uploaded images
[Error]    Some services failed to build:
[Error]      Service: iotedge
[Error]        Error: The command '/bin/sh -c dpkg -i ./iotedge.deb' returned a non-zero code: 1
[Error]    Not deploying release.
Remote build failed

Hi again,

So the entry script looks right. However, it looks the builder is failing because some checks are carried out when the iotedge.deb package is installed. It looks like it may be looking for evidence of docker or LXC or the like as part of the install. I’d suggest the best way to go here is to look at the instructions for installing this package under Linux seeing what dependencies it looks for and how. This may allow you to create a workaround to ensure it successfully installs.

Best regards, Heds

Here you can see the instruction to install it : https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-linux

There is nothing more than the container engine (moby) and the security deamon (which is the iotedge.deb package I’m trying to install)

Hi,

The instructions on that page state Azure IoT Edge relies on an OCI-compatible container runtime. Unfortunately it doesn’t state how it checks for the compatible container runtime. It’s possible that it doesn’t recognise balenaEngine based on paths or something similar. I would recommend decomposing the installer file (the .deb) and seeing how the runtime check is carried out. Also, the previous project you pointed to may have hints on how they ensure the package is successfully installed.

Best regards, Heds

Well following this thread:

I understand they say that balena should be supported:

Just to re-state my problem: I am trying to containerize Edge on runtime that is NOT Moby but balena-engine…

-> Sure. As long as balena-engine exposes the same HTTP API as Moby does, the Security Daemon will not care.

Hello @banto-78, from your message it is not clear what you’re trying to do and what issue you’re facing. Can you please clarify?

Sure. I was point out to a thread where it is stated that iotedge should be compatible with balena-engine. So it was a reply to the doubt of @hedss. However now we are trying now to install moby in the container and see if it is compatible with balena-engine.

Hi,

I’ve try to make a container with moby-engine inside it. Here is the Dockerfile:

FROM amd64/ubuntu

RUN apt-get update
RUN apt-get install --yes curl wget iptables libssl1.0.0

RUN curl -L https://github.com/Azure/azure-iotedge/releases/download/1.0.7/moby-engine_3.0.5_amd64.deb -o moby-engine.deb
RUN dpkg -i ./moby-engine.deb

RUN curl -L https://github.com/Azure/azure-iotedge/releases/download/1.0.8-rc3/libiothsm-std_1.0.8.rc3-1_amd64.deb -o libiothsm-std.deb
RUN dpkg -i ./libiothsm-std.deb

RUN curl -L https://github.com/Azure/azure-iotedge/releases/download/1.0.8-rc3/iotedge_1.0.8.rc3-1_amd64.deb -o iotedge.deb
RUN dpkg -i ./iotedge.deb

I can push it to balena. When I start a terminal session on balenaCloud to set the connection_string and start the service, I have:

root@75738f6a7481:/etc/iotedge# service iotedge start
root@75738f6a7481:/etc/iotedge# service iotedge status
 * iotedge is not running

Also, if I try to set the connection string with the Dockerfile, with
RUN sed -i '/device_connection_string/c\ device_connection_string : \"'$DEVICE_CONNECTION_STRING'\"' /etc/iotedge/config.yaml, the string is not set. I have the env variable set in balenacloud. Any idea?

Thanks

Hi, the env vars configured in the balena-cloud dashboard are only available on the containers when they start on the device, not in the cloud builder during the build.
If you need to use env vars in your dockerfile itself, then have to use built time secrets/variables. Let me point you to the respective documentation page.

@thgreasi Thank you, that’s exactly what I was looking for.

@hedss the answer from microsoft:

That particular error is because the package install wants to enforce that you have some kind of container runtime running, and it does that by checking for the existence of the docker group. The group gets created when you install a Docker package, but of course you don’t want to do that inside the iotedged container (since it’s supposed to use Docker running on the host). So instead just create a dummy group with that name.

So I added RUN groupadd docker in my Dockerfile and I’m now able to deploy iotedge on my device.

Now I have to make it running. I think all the instructions are there:

you can create a base container using one of the OSes in https://docs.microsoft.com/en-us/azure/iot-edge/support#tier-1 , install the package into it, and set up the entrypoint + config to run it without systemd (*). Also ensure the Docker socket from the host is mounted inside the container.

(*) Specifically, you want to mount the homedir on the host so that it can be remounted inside modules, and you want to use UDS unix:// URIs for the listen.* endpoints instead of fd:// since it won’t be running under systemd.

But I don’t understand concretely what I have to do, especially to configure it to run without systemd and mount the docker (balena?) socket inside the container. Would you be so kind to help me?
Thanks.

Hi Everyone, I had a play around with this today and put together a repo here https://github.com/balena-io-playground/azure-iot-edge-v2 , it seems to have the security agent running and pulls down the edgeAgent container, etc.

However, for some reason the edgeAgent container if failing with the following:

2019-07-12 12:31:54.029 +00:00 [INF] - Started operation refresh twin config
2019-07-12 12:31:54.211 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
2019-07-12 12:31:58.129 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
2019-07-12 12:31:58.621 +00:00 [FTL] - Fatal error starting Agent.
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (111): Connection refused /var/run/iotedge/mgmt.sock
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state, Boolean flowContext)
   at System.Net.Sockets.Socket.BeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.GetConnectedSocketAsync() in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessage7
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Mic3
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_01_30.GeneratedCode.EdgeletHttpClient.GetSystemInfoAsync(String api_version, CancellationToken cancellationToken) in /h7
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_01_30.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-age3
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Micro3
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2019_01_30.ModuleManagementHttpClient.GetSystemInfoAsync() in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edg5
   at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerEnvironmentProvider.CreateAsync(IRuntimeInfoProvider runtimeInfoProvider, IEntityStore`2 store, IRestartPolicyManager restartPo9
   at Microsoft.Azure.Devices.Edge.Agent.Service.Modules.EdgeletModule.<>c.<<Load>b__11_6>d.MoveNext() in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Servi6
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Agent.Service.Modules.AgentModule.<>c.<<Load>b__14_14>d.MoveNext() in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Servic3
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Agent.Service.Program.MainAsync(IConfiguration configuration) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Service/Pro2
2019-07-12 12:31:58.832 +00:00 [ERR] - Unexpected error in periodic operation refresh twin config
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.Azure.Devices.Edge.Util.PeriodicTask.DoWork() in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/PeriodicTask.cs:line 83
2019-07-12 12:31:58.842 +00:00 [INF] - Termination requested, initiating shutdown.
2019-07-12 12:31:58.842 +00:00 [INF] - Waiting for cleanup to finish
2019-07-12 12:31:58.842 +00:00 [INF] - Done with cleanup. Shutting down.

I don’t really know much about the Azure IoT edge stack, so it might be that I have misconfigured something, but getting closer to having it running.

1 Like

Oh, from inspecting the edgeAgent container, it seems to need to create a bind mount to the host OS at /var/run/iotedge/mgmt.sock location, which obviously doesn’t exist, since this is in our balena managed container. This looks like quite an issue as balenaCloud doesn’t allow bind mounts to the hostOS. Perhaps its possible to do it with some kinda of named volume, but we would probably need to have the edgeAgent defined in the docker-compose.yml rather than it being pulled in.

This is the mount for those that are interested:

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/run/iotedge/mgmt.sock",
                "Destination": "/var/run/iotedge/mgmt.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run/iotedge/workload.sock",
                "Destination": "/var/run/iotedge/workload.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ]
1 Like

@shaunmulligan can you share the Dockerfile/docker-compose you used for that ?
Thanks

EDIT
Ok I hadn’t seen the repo you shared