Dockerize IoT Edge v2 for BalenaOS

Project help
43

@lpierrat its all in here https://github.com/balena-io-playground/azure-iot-edge-v2

44

@shaunmulligan when you say that

balenaCloud doesn’t allow bind mounts to the hostOS

you mean that it supports only managed volume?

thanks

45

Hi @banto_78 , yes, balenaCloud only allows named volumes, but its not permitted to do random bind-mounts to the hostOS. So you couldn’t define something like :

volume:
    - /etc/os-release:/etc/os-release

the reason for not allowing random bind mounts is that we need to maintain a constant interface and can’t commit to keeping the hostOS filesystem naming and layout the same forever. Allowing those bindmounts opens the flood gates to incompatibilities when doing OS upgrades, so we need to keep the interface as well defined as possible and as loosely coupled with the OS as possible.

46

@shaunmulligan I am bit confused. Is for you managed volume == named volume ?

i.e. something managed completely by docker/balena-engine as:

/var/lib/docker/vfs/dir/“some volume ID”

Whereas

volume:
- /etc/os-release:/etc/os-release

is a bind mount. Are we aligned?

thanks

48

@banto_78, balena supports named volumes in a configuration such as the following, where “resin-data” is the name of the named volume:

Sample docker-compose.yml file:

version: '2'
volumes: 
    resin-data:
services:
    example:
        build: ./example
        volumes: 
            - 'resin-data:/data'

Above, note how the volumes: key appears twice: first at the top of the file, where the named volume is declared. Then for each service, to make the previously declared volume available for use by that service.

As for bind mounts, balena’s platform does not allow arbitrary bind mounts to be specified in a docker-compose file. Only a few pre-defined bind mounts are available, through the use of “labels”, as listed and explained in the Labels section of the following page:

Reading Shaun’s message, I think he meant that "/var/run/iotedge/mgmt.sock" could not be used as a bind mount because of the limitations explained above, plus the fact that "/var/run/iotedge/mgmt.sock" does not exist in the balena host OS. I think Shaun alluded to the possibility of creating a named volume for use with an edge-agent service that would be declared in the docker-compose.yml file, such that "/var/run/iotedge/mgmt.sock" would exist in the named volume rather than in the host OS.

50

@pdcastro thanks. So a named volumes in compose file is only possible if it is resin-data: ? If so, a test we can try is to see if we can change where edgeAgent is binding-mounting the container in a way that is going to bind in the resin-data location. What do you think?

51

iotedged can use IP addresses instead of UDS. See: https://github.com/Azure/iotedgedev/blob/master/docker/runtime/rund.sh as an example.

This can be used for development and testing but not a supported production configuration.

52

Do you mean this part here?

listen:
management_uri: “http://:15580$IP”
workload_uri: “http://:15581$IP”

53

@banto-78 the restriction only applies to bind mounts. Named volumes are fully supported, see here: https://www.balena.io/docs/reference/supervisor/docker-compose/

54

@shaunmulligan @robertgzr @pdcastro thanks for all your answers.
But I need to order the sequence of questions since i am bit lost now.

Questions:

  1. Can you please explain in Balena what’s the difference between named volume and bind mount? And how are they configured? Because I do not find a way to compare it if i just look at Docker documentation.

  2. A named volumes in compose file is only possible if it is resin-data: ?

iotedged can use IP addresses instead of UDS.

Are you referring to this part in the script https://github.com/Azure/iotedgedev/blob/master/docker/runtime/rund.sh to this part:

…
listen:
management_uri: “http://:15580$IP”
workload_uri: “http://:15581$IP”
…

Sorry I just would like to be aligned with you 100%. And sorry if i mixed up questions.

thanks again.

55

Sorry for being confusing…

  1. On a balenaOS device you can’t use bind-mounts because they mount files/directories from the host system into your container.

    Named voluems refer to docker volumes, which are storage managed by the docker daemon that is independent from the host file system. You would typically use the to persist some app data during the lifetime of your container.

    It’s confusing because in docker both concepts are mixed in with each-other by overloading the term volume.

  2. no. there is no naming restriction, you can call them anything you want :slight_smile:

1 Like
56

I can’t say anything about 3. that one would be for @vyalla - welcome to the forums btw o/

59

Ok I understand the definition now.
But where I still need your help with is to understand their syntax in the compose of Balena.

Named Volume:

volumes:
- ‘resin-data:/data’

Bind Mount:

volume:
- /etc/os-release:/etc/os-release

Where are they different? Is because in the first form resin-data is just a reference name or alias and NOT a real path? And of course I understand that the second form is not supported in BalenaOS.

thanks.

60

Hi @banto-78,

That’s exactly correct. The Supervisor does not currently allow bind-mounts (which would mount a real path on balenaOS), whereas it does allow named-volumes, where it will dynamically create a volume for each volume defined which will then be attached to the service that references it.

Best regards, Heds

1 Like
62

Hi @lpierrat,

I saw in another post that you had some success with Iot Edge v2, are you able to provide some assistance on the steps to get this up and running?

Thanks in advance!

63

hi @ShepherdDev welcome to the forums! Have you tried https://github.com/balena-io-playground/azure-iot-edge-v2 its a bit old now but it basically does a docker-in-balena approach to solve the problem and was working in September last year, so worth a shot

65

Hi @shaunmulligan thanks! Just getting started on Balena and still looking into it’s implementation.

In regards to the docker in docker approach, what are the drawbacks or inherent issues before we go down this path. (i.e longer device reboot / container reboot ect.)

Thanks

66

For more on docker-in-docker I would recommend looking at this blog post by the inventor of it, http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

68

@shaunmulligan
I am seeing you used a named volume here - https://github.com/balena-io-playground/azure-iot-edge-v2/blob/master/docker-compose.yml#L11

Are there any other named volumes we can use? Besides resin-data & docker-data?

69

Guys

I have a similar issue.

I am trying to use KubeEdge as a balena service, but it fails as it needs /var/lib/docker directory:

Hi, @mbalamat @jviotti Yes, I was able to get KubeEdge running as a balena service on balenaOS. slight_smile But if I try to deploy a simple application, it fails because KubeEdge tries to mount some directories on hostOS: State: Terminated Reason: ContainerCannotRun Message: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \“rootfs_linux.go:58: mounting \\\”/var/lib/edged/pods…

Any idea?