Goal: Get KubeEdge working as a service on BalenaOS
Approach: I have bind mounted /var/run/balena.sock of the hostOS to /var/run/docker.sock of the service OS by io.balena.features.balena-socket lable here - https://www.balena.io/docs/learn/develop/multicontainer/#labels
What I want is to bind mount some other directories including /var/lib/, /etc/balena/, etc.
Is this strictly not allowed?
Cheers.
Hello, even though I haven’t seen KubeEdge before do you know what requirements does it have? Just balena-engine’s socket didn’t work? Do you see any errors that you can share with us? Also regarding your question, one can’t bind mount directories other than the allowed ones, but maybe there are other ways to make KubeEdge work.
@mbalamat
Thanks for your quick reply!
Yeah, here is a full log:
I0323 10:00:47.151419 331 docker_service.go:240] Hairpin mode set to "hairpin-veth"
W0323 10:00:47.151688 331 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
I0323 10:00:47.180046 331 docker_service.go:255] Docker cri networking managed by kubernetes.io/no-op
I0323 10:00:47.214302 331 docker_service.go:260] Docker Info: &{ID:JMIZ:GL6J:3HXT:Z3HA:VF56:6JUK:WUHB:ZBR6:HRYH:G4DU:JUV7:SVP2
Containers:5 ContainersRunning:3 ContainersPaused:0 ContainersStopped:2 Images:924 Driver:aufs
DriverStatus:[[Root Dir /var/lib/docker/aufs] [Backing Filesystem extfs] [Dirs 499] [Dirperm1 Supported true]] SystemStatus:[]
Plugins:{Volume:[local] Network:[bridge host ipvlan null] Authorization:[] Log:[journald json-file local]}
MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:false CPUCfsPeriod:true
CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:false IPv4Forwarding:true BridgeNfIptables:true
BridgeNfIP6tables:true Debug:false NFd:95 OomKillDisable:true NGoroutines:99 SystemTime:2020-03-23T10:00:47.181615959Z
LoggingDriver:journald CgroupDriver:systemd NEventsListener:1 KernelVersion:5.2.10-yocto-standard
OperatingSystem:balenaOS 2.44.0+rev1 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/
RegistryConfig:0xc000414230 NCPU:4 MemTotal:8262914048 GenericResources:[] DockerRootDir:/var/lib/docker
HTTPProxy: HTTPSProxy: NoProxy: Name:balena Labels:[] ExperimentalBuild:true ServerVersion:18.09.10-dev
ClusterStore: ClusterAdvertise: Runtimes:map[bare:{Path: Args:[]} runc:{Path:balena-engine-runc Args:[]}]
DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState: ControlAvailable:false Error: RemoteManagers:[]
Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false
Isolation: InitBinary:balena-engine-init ContainerdCommit:{ID: Expected:} RuncCommit:{ID:b3d5036b6520060db00c81352d533a5dd919aaaf Expected:b3d5036b6520060db00c81352d533a5dd919aaaf} InitCommit:{ID:949e6fa-dirty Expected:fec3683b971d9} SecurityOptions:[] ProductLicense: Warnings:[WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface]}
E0323 10:00:47.214401 331 edged.go:234] init new edged error, misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
I had to create a symlink with ln -s /var/run/balena.sock /var/run/docker.sock command before launching the keadm join command…
Cheers
Hey there! I never did this myself, but apparently you should be able to change the cgroup driver configuration to match the one that Docker provides: https://serverfault.com/questions/891994/kubernetes-cgroup-driver-misconfiguration (basically cgroup-driver=systemd)
Let us know if that worked!
Yes, I was able to get KubeEdge running as a balena service on balenaOS. 
But if I try to deploy a simple application, it fails because KubeEdge tries to mount some directories on hostOS:
State: Terminated
Reason: ContainerCannotRun
Message: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/edged/pods/c7ba96b4-6514-4d8a-873a-dcdd9063e575/etc-hosts\\\" to rootfs \\\"/var/lib/docker/aufs/mnt/6956ff094c3c1fdbcf573da844135a685315b447c1b3528eb735ae317677015a\\\" at \\\"/var/lib/docker/aufs/mnt/6956ff094c3c1fdbcf573da844135a685315b447c1b3528eb735ae317677015a/etc/hosts\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
It failed because /var/lib/ cannot be mount.
Is there anyway to bind /var/lib of the hostOS to a service?
Cheers
hey @scarlyon,
unfortunately there is no way to bind mount arbitrary host directories and I can’t recommend any alternatives.
Thanks,
Cheers