BalenaOS - RPi 4 - Kafka ( exec user process caused "exec format error")

Product support balenaOS
1

Hello, I’m trying to setup zookeeper and kafka in a docker container running on my Raspberry 4 with BalenaOS.

Here is the zookeeper and kafka sectio of my docker-compose.yml:

zoo1:
image: zookeeper:3.4.9
hostname: zoo1
ports:
- “2181:2181”
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zoo1:2888:3888
kafka1:
image: confluentinc/cp-kafka:5.3.1
hostname: kafka1
ports:
- “9092:9092”
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: “zoo1:2181”
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: “kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO”
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
depends_on:
- zoo1

If I’ll try to push this to mydevice zookeeper and kafka are restarting all the time with the following error message:

standard_init_linux.go:207: exec user process caused "exec format error"

Does anybody know how to setup kafka on a raspberry with BalenaOS?
I really like this OS and would prefer to not switch back to rasbian.

If relevant, I’m using following versions:
Balena-Cli = 11.18.3
Balena-Engine = Docker version 18.09.8-dev
BalenaOS = BalenaOS 2.41.0+rev4

Any help is highly appreciated. Thank you in advance. Cheers Raphael

2

Hey, so that error means that the image you’re trying to run is for the incorrect architecture. You’ll need to find images which are for aarch64 if you’re running on a pi4. It looks like https://hub.docker.com/r/simonwalk/kafka-pi and https://hub.docker.com/r/arm64v8/zookeeper should work.

1 Like
3

Thanks, I’ll try it out. Thank you again for also searching out fitting images!

4

Not a problem, let us know how it goes!

6

I’m not entirely sure if it’s working. I can deploy and kafka looks like it’s up and running.
But if I try to connect to this address (with java) I get the exception:
java.io.IOException: Can't resolve address: kafka1:9092

I tried to connect with mydevice.local and with the ip-address of the raspberry.

I don’t understand why there is the name of my service, specified in docker-compose file in this error message.

I’ve also tried out to push data into kafka using a python producer and it seemed to work with `mydevice.local:9092’. So it’s even more strange that my java code isn’t working and that it tries to connect to this weird address.

7

Where does that connection error come from? Is it the zookeeper service or are you trying to connect to the device from your computer? Typically containers will communicate with each out using the contianer names, so in this case kafka1, but I’m uncertain how exactly the error originated.

8

The error occurred when i tried to run a java application on my computer. I’m setting up a kafka-consumer in there, and this consumer can’t connect.

Thank you for your support! I highly appreciate it.

9

I’m not too familiar with kafka, but just given the information I have, I’d try changing: KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092 to point to 127.0.0.1 instead of kafka1. I don’t believe containers can resolve their own names.

11

Finally got it working by using the following images:

owlab/zookeeper-arm64
owlab/kafka-arm64

Compose section:

zookeeper:
            image: owlab/zookeeper-arm64
            ports:
                - 2181:2181
kafka:
            image: owlab/kafka-arm64
            ports:
                - 9092:9092
                - 7203:7203
            environment:
                - KAFKA_ADVERTISED_HOST_NAME={IP-OF-DEVICE}
                - ZOOKEEPER_IP={IP-OF-DEVICE}

I didn’t manage to get other images running (I’ve the feeling that I tried all of them).

2 Likes