Cellular Modem connectivity issues on boot

Did not help,

Is that modem problem?

[update: the same setup works fine with ZTE MF823, any recommendations?]

journalctrl -u NetworkManager

modem-broadband[cdc-wdm0]: failed to connect modem: Couldn’t create client for service ‘wds’: QMI protocol error (5): ‘Clien
tIdsExhausted’[[0m

Huge log with log-level=debuging, for all devices

Hi All,
Unfortunately It seems I have kind of the same problem with a Sitcom 7100E modem.

I have setup the latest img: Resin OS 2.7.8+rev1 (dev) with this network manager config:

[connection]
id=resin-cellular
type=gsm
autoconnect=true

[gsm]
apn=apn-name

When running it in local mode it is connecting fine.

# nmcli c    
NAME                UUID                                  TYPE            DEVICE   
Wired connection 1  ae641dae-f70c-3691-98c5-888c26cd3f8b  802-3-ethernet  eth0     
resin-cellular      d811a35a-70fc-3716-84e1-a3578d10cc1e  gsm             cdc-wdm0

But when running with an app like: simple-server-python or resin-node-red, the modem is not recognized. Sometimes it does, mostly it doesn’t:

# nmcli c    
NAME                UUID                                  TYPE            DEVICE 
Wired connection 1  ae641dae-f70c-3691-98c5-888c26cd3f8b  802-3-ethernet  eth0   
resin-cellular      d811a35a-70fc-3716-84e1-a3578d10cc1e  gsm             --  

The modem manager isn’t recognizing the modem:

# mmcli -L
No modems were found

Then i tried to use an empty app like below and that does seem to work.

FROM armhf/busybox
CMD ["echo", "hello"]

Another thing I noticed is that the modem is sometimes recognized as ttyUSB2 and in other times as cdc-wdm0. When running in local mode with the empty app its most of the time setup as cdc-wdm0 if i am right.

Is there a solution for this problem, that the modem is not recognized? Because when this happens and I manually scan with modem manager it is not showing.

I tried to edit this file from within the running container:


And commended out rule 59 and 60.

When i rebooted the device multiple times the connection stays up and all the time with cdc-wdm0. So that looks like it works.

But then when i added this rule

COPY entry.sh /usr/bin/entry.sh

to the docker template i get errors when running the application. It just won’t start.
Is there something i am missing?

I just copied the whole original entry.sh file to entry.sh in my app so it can be copied over to the container on start. I even tried to keep the exact copy of the file and do copy that over (so without the disabled rule 59 and 60). But still this results in an error starting the application like the log:

19.01.18 14:53:16 (+0100) Application is already stopped and the container removed 'registry2.resin.io/my-app-name/def011c10ea2ee9e95592766dc2d9851df5c1e70'
19.01.18 14:53:16 (+0100) Killed application 'registry2.resin.io/my-app-name/def011c10ea2ee9e95592766dc2d9851df5c1e70'
19.01.18 14:53:16 (+0100) Installing application 'registry2.resin.io/my-app-name/def011c10ea2ee9e95592766dc2d9851df5c1e70'
19.01.18 14:53:17 (+0100) Installed application 'registry2.resin.io/my-app-name/def011c10ea2ee9e95592766dc2d9851df5c1e70'
19.01.18 14:53:17 (+0100) Starting application 'registry2.resin.io/my-app-name/def011c10ea2ee9e95592766dc2d9851df5c1e70'
19.01.18 14:53:18 (+0100) Failed to start application 'registry2.resin.io/my-app-name/def011c10ea2ee9e95592766dc2d9851df5c1e70' due to '(HTTP code 500) server error - invalid character 'c' looking for beginning of value '
19.01.18 14:53:18 (+0100) Failed to update application 'registry2.resin.io/my-app-name/def011c10ea2ee9e95592766dc2d9851df5c1e70' due to '(HTTP code 500) server error - invalid character 'c' looking for beginning of value '

What can i do to make this stable?

Hi All, I think I have fixed it by chmod the entry.sh file as -x. Also I disabled the ENV INITSYSTEM bij adding:
ENV INITSYSTEM=off to the docker file.
I will test it some more if this stays stable.

@RDB Thanks for your posts. We are using the same Sitcom 7100E modem. Maybe even the same device(embedded pi)? Using your fixes I got the modem working. But not reliably though. Do you have any other tips?

@sdehaes i had the same problem. When restarting the container or reboot the device the modem was sometimes not recognized anymore.

When running no app container (local mode) the connection and modem where stable and kept being recognized. But when an app is started the containers base img is triggering this command:

udevadm trigger &> /dev/null

So you should make sure to disable that line as a fix for now until Resin has a more stable solution.
What you should do is to find the file entry.sh in you base img. This can be found in /usr/bin/entry.sh normally. You should copy that entry file and place it in your project with the line commented out. After that you have to add a extra rule in your docker file with:

COPY entry.sh /usr/bin/entry.sh

This makes sure the file is overwritten on build and the udevadm trigger &> /dev/null is not triggered. This means the host OS will not touch the usb devices or any other devices when the container is starting or restarting. Because i made a new file that wasn’t executable i run into the problems described above. But when i made the file executable it did startup correctly and wasn’t triggering the udevadm anymore and so the modem became untouched on every restart.

@RDB

So you left the following line?

This does mean I can not use udev rules in the app container?

@sdehaes True i left that as i think it wouldn’t hurt. But maybe i will leave it out aswell since that checks if new devices where added after the container is booted. But since this isn’t possible with the embeddedpi we can leave it out.

Also its true you can’t use the udev rules. Since they will only work with the udevadm trigger command.
But that breaks the modem functionality :(. Iam struggling with this aswell. If you or a Resin engineer has a good working suggestion, please let me know :).

@shaunmulligan The best thing i could think of is that it would be great if we can add the custom udev rules to the hostOS. Maybe even with a folder like system-connections solution on the resinOS, which will copied into the hostOS folder: /etc/udev/rules.d/ on boot. Hopefully Resin will implement such feature to have the custom device udev rules applied from the host already.

@RDB

There is an issue open on github for the udev rules :slight_smile: So we could explain our use case there.

@sdehaes Nice! I only see its already open for 1 year :open_mouth:
Hopefully they will give it more prio since it would be really helpful.

Hello! I did that and still get the failed to start application, can I take a look at your docker file?

Mine goes like this:


# Set the working directory
WORKDIR /usr/src/app

# Copy everything into the container
COPY . ./

RUN chmod -x entry.sh

COPY entry.sh /usr/bin/entry.sh

# Start application
CMD ["bash", "start.sh"]

@carlosLabrado hm strange.
Well i also tried to make the file executable on my OSX file system before pushing it to the git repo.
Maybe that will work insted of adding it to the docker file.
I have removed the lines since the file was executable and it wasn’t need to set this each time the container started.

For anyone struggling with modem issues, I have created a little test project here: https://github.com/resin-io-projects/cellular-test

This project disables all the udevd stuff and should allow you to validate whether your modem is having issues or not, without the App causing any weird effects.

1 Like

I was unable to get my (Hologram Nova) modem to show up with mmcli -L in my app, but your cellular-test app worked, and it showed up (woo!).

What steps should I take to get the modem working in my app?

Thanks!

Hi @jingman,

The relevant portion is the modified entry.sh file that disables the udev instance running in the application container. Add this file to your project, then add these steps in your Dockerfile.template file.

COPY entry.sh /usr/bin/entry.sh 
RUN chmod +x /usr/bin/entry.sh

This should work assuming you are using a Resin base image.

Is this still needed?
I have similiar situation that cellular devices goes offline when stopping container.

@thomasw as this thread is quite old, are you able to share more information about the issue you are seeing at the moment (perhaps starting with sharing the balenaOS version and supervisor version, as well as any logs you may have)?

I was able to fix it by ENV UDEV=off in dockerfile.
Issue was cellular connection interrupted each time I have restarted container and had to hard reset raspberry by disconnect power to reconnect cellular again.
balenaOS 2.29.2+rev2

@thomasw if you don’t mind me asking, what base image are you using in your containers? the newer balenalib base images should run udev in the container in network namespace to avoid this issue, so im interested to know if it was these images?


What should be used instead?