Bug GPIO balenaOS 6.5.1+rev2

Product support
1

Hello,

I’m working on raspberry pi4 pinned on balenaOS 6.5.1+rev2
I’m experiencing a potential issue with BalenaOS 6.5.1+rev2.
When I upgrade to this version and include the following configuration parameters:

"18=op,dh"

"sc16is752-i2c,int_pin=17,addr=0x4d,xtal=1843200"

I’m unable to instantiate the SC16IS752 UART. In previous versions, this configuration worked as expected, and I could access ttySC0 and ttySC1. However, with version 6.5.1+rev2, no UART devices appear.

Is this related to a GPIO handling bug introduced in this release?
It would also be very appreciated to add a feature to go back to the OS versions, especially in the event of a problem :slight_smile:

BR
Firat

2

Hello @Firat thanks for your message!

We are working actively to fix this. Looks like it’s related with this Github Issue. Looks like the gpio it’s deprecated. You can read more here.

We will keep you posted, however let us know if you can fix this yourself as well!

3

@mpous can you clarify where this would have been introduced? I’m also seeing this issue starting in version 6.5.x. The affected section of code was working fine in v6.4.1 rev 1.
The GitHub issue you linked is from a while ago, not clear why it would affect only new versions of Balena OS (and it’s not clear from the changelog)?

I’m using gpiozero and RPi.GPIO

Like in the GitHub issue, my code also now crashes with:

 my-service    File "/usr/local/lib/python3.9/site-packages/gpiozero/pins/pi.py", line 319, in _set_when_changed
 my-service      self._enable_event_detect()
 my-service    File "/usr/local/lib/python3.9/site-packages/gpiozero/pins/rpigpio.py", line 226, in _enable_event_detect
 my-service      GPIO.add_event_detect(
 my-service  RuntimeError: Failed to add edge detection
4

@mpous ,

What would be greatly appreciated from Balena would be the ability to roll back to a previous version of BalenaOS
A very concrete example:
After updating several machines to version 6.5, we are currently experiencing several issues in the GPIO system

5

@Firat @yardstick1 thanks for your messages!

Due to changes with the new Raspberry Pi OS, the RPi.GPIO library commonly used has been deprecated, so it is not compatible with the newer OS versions.

Check out Raspberry Pi’s History of GPIO usage document (PDF) for tips on GPIO libraries compatible with the Raspberry Pi including libgpiod.

Is it possible to change the GPIO library from your projects?

6

@mpous,
Thanks for your reply.
In our case, we want to pull up a GPIO at boot, not after in some sevice
What solutions do you suggest?

7

We are facing issues with this as well, it would be really great if we could have a downgrade option for OS… Has anyone found a way to work around it?
in our case this command is throwing an error:
echo 7 > /sys/class/gpio/export
This prevents us from resetting our lora concentrator on startup at the moment.
libgpiod doesn’t seem to be available either since gpiodetect is not available. Or do we have to install this ourselves in the container?

8

The only way I’ve managed to “downgrade” is to switch to the previous boot partition by updating the file /mnt/boot/resinOS_uEnv.txt and changing resin_root_part=2 to resin_root_part=3 (or vice versa depending on which was the last working partition).

9

Update: Root cause found and workaround confirmed

After extensive debugging, we’ve identified the root cause of the SC16IS752 (/dev/ttySC*) devices disappearing on newer balenaOS versions (kernel 6.x+).

Root cause:

Starting with Linux kernel ~6.8+, the sc16is7xx driver was split into three separate kernel modules:

Module Role
sc16is7xx.ko Core driver (serial logic)
sc16is7xx_i2c.ko I2C bus registration
sc16is7xx_spi.ko SPI bus registration

On older kernels (5.15 in balenaOS 6.1.24), the single sc16is7xx.ko module handled everything. On kernel 6.12 (balenaOS 6.10.24), the DT overlay sc16is752-i2c correctly loads sc16is7xx.ko via modalias matching, but does not trigger loading of sc16is7xx_i2c.ko, which is the module that actually registers the I2C driver.

The result: the device tree node is correctly populated (/proc/device-tree/soc/i2c@7e804000/sc16is752@4d exists), the I2C device is detected in sysfs (/sys/bus/i2c/devices/1-004d exists), the core module is loaded (lsmod | grep sc16is shows sc16is7xx), but no I2C driver is registered (/sys/bus/i2c/drivers/sc16is7xx/ does not exist), so the device never gets probed and /dev/ttySC* never appears.

Workaround:

modprobe sc16is7xx_i2c

This immediately creates /dev/ttySC0 and /dev/ttySC1. Confirmed on a Raspberry Pi 4 running balenaOS 6.10.24 (kernel 6.12.61-v8).

However this is volatile and lost on reboot. We’re currently looking for a way to make this persistent.

Suggested fix for balenaOS:

The sc16is7xx_i2c.ko (and sc16is7xx_spi.ko) modules should have proper modules.dep entries so that modprobe sc16is7xx automatically pulls in the bus-specific sub-modules, or the DT overlay modalias resolution should trigger loading of the correct sub-module. This seems like a module dependency / depmod issue that should be addressed at the OS level.

Diagnostic comparison (same fleet, same DT overlay config):

balenaOS 6.1.24 (works) balenaOS 6.10.24 (broken)
Kernel 5.15.92-v8 6.12.61-v8
sc16is7xx.ko single module, registers I2C driver core only, no bus registration
sc16is7xx_i2c.ko does not exist exists but not auto-loaded
/dev/ttySC* present missing
After modprobe sc16is7xx_i2c n/a fixed

Could the balena team look into ensuring proper module dependency resolution for split kernel drivers in newer OS versions?

BR,

Firat.