Edgex on RPi4 using Balena fail to run MQTT&device-random services

I was trying to deploy Edgex on RPi4 following the https://www.balena.io/blog/make-a-web-frame-with-raspberry-pi-in-30-minutes/ to set up balena CLI, balena cloud etc on my PC, balena Eatcher;
i get the docker-compose file for the arm64 and push it into my RPi4, untill this everything was fine, but i noticied what the MQTT and device-random services are remains in the status “Installed” and even when i try to run them from the dashboard nothing happening. No particular error log, except one strange thing i noticied what the edgex-config-seed service is restarting from time to time… and when i go to the myIP:8500/v1/agent/services i get a list of all except those services. Any suggestions how i can fix this?
Thanks a lot in advance,
Liudmila

Hi, is it possible to share your code with us (if it’s up on Github, for example)? Would be able to see better what are you doing. Not sure what Edgex is, and hard to comment on that.

Do you have any Raspberry Pi 3 boards to try things? I wonder if it’s a 32-bit/64-bit issue (Pi3 balenaOS is by default 32-bit, while the Pi4 is 64-bit). If it runs fine on Raspberry Pi3, the could try switching th base image from the Pi4 one to raspberrypi3 (ie. replacing %%BALENA_MACHINE_NAME%% with explicit raspberrypi3) , and see if that works.

If you try these and learn anything new, and still doesn’t work, we might be able to suggest other things to try

Hi, no, i don’t have RPi3 to try.
i loaded a 64bit OS because EdgeX require it. So after this i made push of the docker-compose.yaml, i discovered partially the problem rootcause, it was the field names uncompatible with the version 2 of docker-composer, such as container_name, hostname and networks, commenting them out all services are started, but i still have a problems with the mqtt service, it fails to connect to broker. The part responsable for installing mqtt: device-mqtt:
image: edgexfoundry/docker-device-mqtt-go-arm64:1.0.0
ports:
- “49982:49982”

container_name: edgex-device-mqtt

hostname: edgex-device-mqtt

networks:

- edgex-network

volumes:
  - db-data:/data/db
  - log-data:/edgex/logs
  - consul-config:/consul/config
  - consul-data:/consul/data
depends_on:
  - edgex-core-data
  - edgex-core-command

and i get the error from the log:
evice-mqtt level=INFO ts=2019-10-14T13:39:25.327173463Z app=edgex-device-mqtt source=driver.go:279 msg="Create MQTT client and connection: uri=tcp://admin:public@0.0.0.0:1883 clientID=CommandResponseSubscriber "
14.10.19 15:39:25 (+0200) device-mqtt panic: start command response Listener failed, please check MQTT broker settings are correct, Network Error : dial tcp 0.0.0.0:1883: connect: connection refused
14.10.19 15:39:25 (+0200) device-mqtt
i belive what this is a matter of configuration, but have no idea why it go to 0.0.0.0 and not getting the device IP…and how i can change it.

@Miliks1975 did you find this tutorial, which may help: EdgeX + Balena Walkthrough Tutorial

Hi @chrisys, yes, but i didn’t find solution there. it says what networking is not working fine, and i commented out the hostname line from mqtt definition, so how i can define the hostname for this service?

BTW, commenting out the same fields for device-random container definition solved the issue. Now this service looks fine.

@Miliks1975 the hostname by default is the service name, so you would rename the service name to the hostname desired - it mentions this in the guide:

Due to (6), hostname doesn’t work either. The services are accessible using only the service name as defined in the docker compose . Since EdgeX services are hard-coded to look for certain hostnames (that are defined in the original delhi compose file), we rename the service name to match the original hostname . The services are now discoverable, bypassing the bug.

so, i changed it like this:
edgex-device-mqtt:
image: edgexfoundry/docker-device-mqtt-go-arm64:1.0.0
ports:
- “49982:49982”
volumes:
- db-data:/data/db
- log-data:/edgex/logs
- consul-config:/consul/config
- consul-data:/consul/data
depends_on:
- edgex-core-data
- edgex-core-command


no luck anyway

Let’s ping @odyslam here and see if he can offer any advice since he wrote the tutorial!

1 Like

Hey,

The mqtt device service appears to have a panic error because it can’t connect to the mqtt broker. This question is more edgex related and not balena (I think).

You should configure the device-service to point to the broker, either using the right ip or the correct container name.

but i changed the service name to make it equal to the “suppose-to-be” host name, what else i can modify? another issue i notcied, i can’t connect to any service throught SSH terminal… it keep responding me:
Connecting to 37458559f0f49d81681634e1e230de3e…
Spawning shell…
OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused “exec: “/bin/sh”: stat /bin/sh: no such file or directory”: unknown
OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused “exec: “bash”: executable file not found in $PATH”: unknown
SSH session disconnected
SSH reconnecting…
Spawning shell…

@Miliks1975 it would help if you could share your project too - if you can put everything on GitHub others will be able to take a look and try to reproduce what you’re seeing.

it’s just a .yml i’m trying to deploy.
I put it there: https://github.com/Miliks/BalenaEdgeX.git

This error is due to the fact that the containers have no shell installed in them, this is to minimise the container size as much as possible. Without a shell, you can 't spawn an ssh.

Regarding our issue, the service appears to be trying to connecto to 0.0.0.0:1883, which is a wront ip. Try using an online mqtt broker and configuring the url instead of an ip, try this one MQTT Websocket Client

Ok, so basically u r saying what there is no way to have an SSH access to any of containers loaded by default with this yml, but i shoulbuil my own directly… is there any chance to make it in easier way? Can u give me a hint where i can find the Docker files on RPi so i can modify it? Or it is not possible? Sorry for the questions, but i’m new in this.
Thanks a lot for your help!

Hi @miliks1975 what odys is saying is that the image you are trying to ssh into doesn’t have the/bin/sh binary in it, so its not possible to start a shell session. Do you have the source code for the image that you are trying to ssh into, it would be good to see what OS it is based on, some docker images don’t have a file system and are just do binaries, so you can’t ssh into them.

No, i don’t. I probably can google for them. I’m trying to follow the tutorial to deploy EdgeX on RPi4, and the default docker-compose.yml is having this issues. I put it here: https://github.com/Miliks/BalenaEdgeX.git.
If the only way is to build all from the skretch i do not see all this advantages of having the lightweght OS.
I can access to the Host in SSH though… at least something :slight_smile:

@miliks1975 the idea is not to build it all from scratch, its just good to have the source to fully understand the code one is running. It makes things much more difficult to debug code if you have no idea what the code does or is :stuck_out_tongue: .

I will setup a pi4 here and try recreate the issue.

1 Like

@miliks1975 I pushed this to a pi3 (64bit) and indeed many of the surfaces are just small binaries, so its expected to not be able to ssh into those containers. However i can ssh into the edgex-mongo container because it is based of of ubuntu. on my device the edgex-device-mqtt and edgex-export-distro services seem to keep restarting, but inspecting the containers its not producing any logs, so there isn’t much to help debug there. Maybe @odyslam can help us troubleshoot a bit more here as he has more edgeX knowledge :slight_smile:

2 Likes