I have a custom RPi CM4 product and I’ve been using the IOboard device type. It seems that it’s been stuck at BalenaOS version 4.0.16 for the longest time. I don’t really have any issues with it, but I would like to get on a later version of the Linux kernel. I tried to add a new device to my fleet of the standard RPi4 device-type and it works fine on my CM4 (I just have a custom overlay which I added). And now I can install BalenaOS 5.2.8. But I was surprised to see that it also contains Linux kernel 5.15.92, which is the same as in 4.0.16. Is there a newer version of BalenaOS that uses the v6 kernel of Linux? Also is there an official list of the release versions/notes and which Linux kernel they’re using? I found an outdated Github repo that hasn’t been updated for 3 years, that’s obviously not it.
1 Like
Hello @HakanL thanks for your questions. I asked internally to the OS team to help here.
In the meantime, is it possible to understand what features from the v6 kernel of Linux do you need on your project?
I had some display hardware issues that I brought up to the RPi Linux developers and they told me there has been a lot of changes in the 6.x kernel to address these (for example, unreliable output from DSI0 on RPi CM4).
I ended up re-designing my hardware to use DSI1 instead, to workaround the issue, so it’s not an urgent need for me, but more that fixes like that aren’t back-ported to v5, so having access to a v6 kernel would allow for more options on the hardware side.
Hi,
As you probably know, the v5.15 kernel is a long term release that will not become EOL until the end of 2026. The meta-raspberrypi layer only updated to v6.1 on mickledore, which was not a Yocto LTS release, but is likely they will use v6.6 on Scarthgap, which we are currently working on supporting.
So I would expect an update to the v6.6 longterm kernel with the Yocto Scarthgap LTS release.
2 Likes
Thanks, that’s helpful. Will there be a way to select the 6.6 vs 5.15 track when selecting OS in Balena, or will it just be a new release like today?
Hi, we do not have release tracks available, so once the kernel update happens there will be a new release that supports the new kernel.
2 Likes
Could this also be a problem if we intend to use an external SD card using SD0 interface?
I have an rpi cm4 running raspbian (debian bullseye with Linux Kernel v6.1 hosted in eMMC memory) and I was able to compile a .dts file and add the sd0.dbo file in /boot/overlays and add the dtoverlay=sd0 parameter in /boot/config.txt file and then use an external SD card as external memory.
However, on the same rpi cm4 running BalenaOS I did the same steps and added the file sd0.dtbo in mnt/boot/overlays and added the parameter dtoverlay=sd0 in /mnt/boot/config.txt but I am receiving the following error executing the command dmesg:
mmc2: sdhost-bcm2835 loaded - DMA enabled (>1)
mmc2: invalid bus width
mmc2: error -22 when initializing SD card
mmc1: new high-speed SDIO card at address 0001
mmc0: SDHCI controller in fe340000.mmc [fe340000.mmc] using ADMA
mmc0: new DDR MMC card at address 0001
mmcblk0: mmc0:0001 BJTD4R 29.1 GiB
mmcblk0: p1 p2 p3 p4 < p5 p6 >
mmcblk0: mmc0:0001 BJTD4R 29.1 GiB
mmcblk0boot0: mmc0:0001 BJTD4R 4.00 MiB
mmcblk0boot1: mmc0:0001 BJTD4R 4.00 MiB
mmcblk0rpmb: mmc0:0001 BJTD4R 4.00 MiB, chardev (242:0)
mmc2: invalid bus width
mmc2: error -22 when initializing SD card
mmc2: invalid bus width
mmc2: error -22 when initializing SD card
Can this problem be related to the Linux Kernel of BalenaOS? Why in BalenaOS I can’t use an external SD Card?
@embedded I’m not sure about your specific issue, but in my design I have an external SD card, plus the internal eMMC, and it works fine using BalenaOS. However I purposely put the external SD card on different pins so both could be accessible concurrently. Could it be that your overlay isn’t loaded properly, I know it can be hard to catch those errors.
Hi
Thanks for your response. Can you check out this post I wrote explaining my process? I used the exact same .dts file to compile the .dtob file and I also used the same dtoverlay. It works on Raspbian. It doesn’t work on BalenaOS.
Hmm, I never tried to have a container modify the host, I didn’t think it was possible (but I may be wrong on that), I manually updated the host image before I wrote it to the eMMC on my devices, and it seems to survive OS upgrades. Your sd0-overlay file seems to match mine. Have you confirmed that the issue isn’t with updating the host via container, ie if you just copy the file into the overlay folder and update the config via the Balena Dashboard? Is your SD card actually wired to the pins in the overlay file (22-27), I assume they are?
It was a while ago now, but I believe there are overlay tools where you have it query the current running tree (in /proc I think), maybe that can give you hints if the overlay is actually loaded? I know I struggled quite a bit with having incorrect overlays and no reliable way to see the errors.
I’m not updating the host in the container as you mentioned, Balena doesn’t allow it. That was just an explanation of what I was trying to do first but then I change the approach. So what I did:
1) Write .dts file
/dts-v1/;
/plugin/;
/{
compatible = "brcm,bcm2711";
fragment@0 {
target = <&sdhost>;
frag0: __overlay__ {
pinctrl-names = "default";
pinctrl-0 = <&sdhost_pins>;
bus-width = <4>;
brcm,overclock-50 = <0>;
brcm,pio-limit = <1>;
status = "okay";
};
};
fragment@1 {
target = <&gpio>;
__overlay__ {
sdhost_pins: sdhost_pins {
brcm,pins = <22 23 24 25 26 27>;
brcm,function = <4 4 4 4 4 4>; /* ALT0: SD0 */
brcm,pull = <0 2 2 2 2 2>; /* pull up all except clk */
};
};
};
__overrides__ {
overclock_50 = <&frag0>,"brcm,overclock-50:0";
force_pio = <&frag0>,"brcm,force-pio?";
pio_limit = <&frag0>,"brcm,pio-limit:0";
debug = <&frag0>,"brcm,debug?";
};
};
2. Compile the .dts file in my own laptop running the same image as BalenaOS
dtc -@ -I dts -O dtb -o ./sd0.dtbo /sd0-overlay.dts
3. Using scp send the sd0.dtbo file to the device running BalenaOS. Put it in the /mnt/boot/overlays directory
4. Add dtoverlay=sd0 to /mnt/boot/config.txt file
5. Reboot the device
6. Check dmesg
mmc2: sdhost-bcm2835 loaded - DMA enabled (>1)
mmc2: invalid bus width
mmc2: error -22 when initializing SD card
mmc1: new high-speed SDIO card at address 0001
mmc0: SDHCI controller in fe340000.mmc [fe340000.mmc] using ADMA
mmc0: new DDR MMC card at address 0001
mmcblk0: mmc0:0001 BJTD4R 29.1 GiB
mmcblk0: p1 p2 p3 p4 < p5 p6 >
mmcblk0: mmc0:0001 BJTD4R 29.1 GiB
mmcblk0boot0: mmc0:0001 BJTD4R 4.00 MiB
mmcblk0boot1: mmc0:0001 BJTD4R 4.00 MiB
mmcblk0rpmb: mmc0:0001 BJTD4R 4.00 MiB, chardev (242:0)
mmc2: invalid bus width
mmc2: error -22 when initializing SD card
mmc2: invalid bus width
mmc2: error -22 when initializing SD card
Is your SD card actually wired to the pins in the overlay file (22-27), I assume they are?
Yes. The same .dts file to generate the .dtbo file and the same dtoverlay was used in rpiOS (Debian bullseye) and works correctly. In rpiOS I can access the SD card. Only on BalenaOS I can’t do it.
Ok, I just wanted to confirm. Your file is identical to mine. I do one more step, I merge several overlays into one dts, but that shouldn’t matter. You wrote that you add overlay directly into the config.txt, but it’s my understanding that the supervisor will revert this and use the settings from the dashboard, but that may not be your issue, I just wanted to mention that.
Here’s my boot dmesg, maybe you can see if there are any obvious differences:
Spawning shell...
=============================================================
Welcome to balenaOS
=============================================================
root@beaac95:~# dmesg
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 5.15.92-v8 (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Wed Feb 8 16:47:50 UTC 2023
[ 0.000000] random: crng init done
[ 0.000000] Machine model: Raspberry Pi Compute Module 4 Rev 1.1
[ 0.000000] efi: UEFI not found.
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000001ac00000, size 320 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x000000003fffffff]
[ 0.000000] DMA32 [mem 0x0000000040000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000003e5fffff]
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[ 0.000000] On node 0, zone DMA32: 6656 pages in unavailable ranges
[ 0.000000] On node 0, zone Normal: 16384 pages in unavailable ranges
[ 0.000000] percpu: Embedded 29 pages/cpu s78104 r8192 d32488 u118784
[ 0.000000] pcpu-alloc: s78104 r8192 d32488 u118784 alloc=29*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: detected: Spectre-v2
[ 0.000000] CPU features: detected: Spectre-v3a
[ 0.000000] CPU features: detected: Spectre-v4
[ 0.000000] CPU features: detected: Spectre-BHB
[ 0.000000] CPU features: kernel page table isolation forced ON by KASLR
[ 0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[ 0.000000] CPU features: detected: ARM erratum 1742098
[ 0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 2041344
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=E4:5F:01:DF:C5:10 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000 dwc_otg.lpm_enable=0 rootfstype=ext4 rootwait dwc_otg.lpm_enable=0 rootwait console=null quiet splash vt.global_cursor_default=0 consoleblank=0 cgroup_enable=memory video=HDMI-A-1:800x480M-32@60D root=UUID=ba1eadef-01cc-47d5-999b-bc0d8c256fb9 rootwait
[ 0.000000] cgroup: Enabling memory control group subsystem
[ 0.000000] Unknown kernel command line parameters "splash", will be passed to user space.
[ 0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] software IO TLB: mapped [mem 0x000000003a600000-0x000000003e600000] (64MB)
[ 0.000000] Memory: 7691752K/8296448K available (10304K kernel code, 1702K rwdata, 3944K rodata, 33152K init, 995K bss, 277016K reserved, 327680K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] ftrace: allocating 40658 entries in 159 pages
[ 0.000000] ftrace: allocated 159 pages with 6 groups
[ 0.000000] trace event string verifier disabled
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] Root IRQ handler: gic_handle_irq
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7ef00100, parent irq: 10)
[ 0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[ 0.000001] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[ 0.000425] Console: colour dummy device 80x25
[ 0.000522] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=216000)
[ 0.000554] pid_max: default: 32768 minimum: 301
[ 0.000727] LSM: Security Framework initializing
[ 0.001042] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 0.001161] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 0.005942] rcu: Hierarchical SRCU implementation.
[ 0.007308] EFI services will not be available.
[ 0.008064] smp: Bringing up secondary CPUs ...
[ 0.009528] Detected PIPT I-cache on CPU1
[ 0.009622] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[ 0.011239] Detected PIPT I-cache on CPU2
[ 0.011308] CPU2: Booted secondary processor 0x0000000002 [0x410fd083]
[ 0.012982] Detected PIPT I-cache on CPU3
[ 0.013051] CPU3: Booted secondary processor 0x0000000003 [0x410fd083]
[ 0.013296] smp: Brought up 1 node, 4 CPUs
[ 0.013322] SMP: Total of 4 processors activated.
[ 0.013338] CPU features: detected: 32-bit EL0 Support
[ 0.013350] CPU features: detected: 32-bit EL1 Support
[ 0.013366] CPU features: detected: CRC32 instructions
[ 0.017815] CPU: All CPU(s) started at EL2
[ 0.017898] alternatives: patching kernel code
[ 0.020033] devtmpfs: initialized
[ 0.035999] Enabled cp15_barrier support
[ 0.036111] Enabled setend support
[ 0.036139] KASLR enabled
[ 0.036554] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.036595] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.047652] pinctrl core: initialized pinctrl subsystem
[ 0.048721] DMI not present or invalid.
[ 0.049530] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 0.054058] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[ 0.054400] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[ 0.055459] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[ 0.055578] audit: initializing netlink subsys (disabled)
[ 0.055947] audit: type=2000 audit(0.052:1): state=initialized audit_enabled=0 res=1
[ 0.056741] thermal_sys: Registered thermal governor 'step_wise'
[ 0.057032] cpuidle: using governor menu
[ 0.057512] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.057779] ASID allocator initialised with 32768 entries
[ 0.057966] Serial: AMBA PL011 UART driver
[ 0.074555] bcm2835-mbox fe00b880.mailbox: mailbox enabled
[ 0.112576] raspberrypi-firmware soc:firmware: Attached to firmware from 2023-05-09T12:16:50, variant start_cd
[ 0.116597] raspberrypi-firmware soc:firmware: Firmware hash is 30aa0d70ab280427ba04ebc718c81d4350b9d394
[ 0.182898] bcm2835-dma fe007000.dma: DMA legacy API manager, dmachans=0x1
[ 0.188152] vgaarb: loaded
[ 0.188717] SCSI subsystem initialized
[ 0.189066] usbcore: registered new interface driver usbfs
[ 0.189139] usbcore: registered new interface driver hub
[ 0.189222] usbcore: registered new device driver usb
[ 0.189690] usb_phy_generic phy: supply vcc not found, using dummy regulator
[ 0.189962] usb_phy_generic phy: dummy supplies not allowed for exclusive requests
[ 0.190334] pps_core: LinuxPPS API ver. 1 registered
[ 0.190350] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.190382] PTP clock support registered
[ 0.190681] Advanced Linux Sound Architecture Driver Initialized.
[ 0.192420] clocksource: Switched to clocksource arch_sys_counter
[ 0.269097] VFS: Disk quotas dquot_6.6.0
[ 0.269246] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.269462] FS-Cache: Loaded
[ 0.269747] CacheFiles: Loaded
[ 0.285116] NET: Registered PF_INET protocol family
[ 0.286081] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[ 0.292119] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[ 0.292219] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.292257] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[ 0.292785] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[ 0.293707] TCP: Hash tables configured (established 65536 bind 65536)
[ 0.294053] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[ 0.294204] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[ 0.294683] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 0.294743] PCI: CLS 0 bytes, default 64
[ 0.306855] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[ 0.307268] kvm [1]: IPA Size Limit: 44 bits
[ 0.308803] kvm [1]: vgic interrupt IRQ9
[ 0.309172] kvm [1]: Hyp mode initialized successfully
[ 0.314114] Initialise system trusted keyrings
[ 0.314628] workingset: timestamp_bits=46 max_order=21 bucket_order=0
[ 0.323982] zbud: loaded
[ 0.326407] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.328246] Key type asymmetric registered
[ 0.328269] Asymmetric key parser 'x509' registered
[ 0.328374] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[ 0.328821] io scheduler mq-deadline registered
[ 0.328841] io scheduler kyber registered
[ 0.338533] gpio-507 (ant1): hogged as output/high
[ 0.340024] gpio-511 (ant2): hogged as output/low
[ 0.342177] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 0.342236] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 0.342337] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[ 0.342433] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x01ffffffff -> 0x0400000000
[ 0.660463] brcm-pcie fd500000.pcie: link down
[ 0.674663] iproc-rng200 fe104000.rng: hwrng registered
[ 0.675268] vc-mem: phys_addr:0x00000000 mem_base=0x3f000000 mem_size:0x3f600000(1014 MiB)
[ 0.677164] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000
[ 0.695317] brd: module loaded
[ 0.707660] loop: module loaded
[ 0.708511] zram: Added device: zram0
[ 0.709514] Loading iSCSI transport class v2.0-870.
[ 0.715845] tun: Universal TUN/TAP device driver, 1.6
[ 0.717086] bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
[ 0.784628] unimac-mdio unimac-mdio.-19: Broadcom UniMAC MDIO bus
[ 0.786688] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[ 0.787091] dwc_otg: FIQ enabled
[ 0.787106] dwc_otg: NAK holdoff enabled
[ 0.787118] dwc_otg: FIQ split-transaction FSM enabled
[ 0.787136] Module dwc_common_port init
[ 0.787870] usbcore: registered new interface driver uas
[ 0.787951] usbcore: registered new interface driver usb-storage
[ 0.788082] usbcore: registered new interface driver usbserial_generic
[ 0.788127] usbserial: USB Serial support registered for generic
[ 0.788441] mousedev: PS/2 mouse device common for all mice
[ 0.795058] sdhci: Secure Digital Host Controller Interface driver
[ 0.795079] sdhci: Copyright(c) Pierre Ossman
[ 0.795895] sdhci-pltfm: SDHCI platform and OF driver helper
[ 0.801705] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.802001] hid: raw HID events driver (C) Jiri Kosina
[ 0.802115] usbcore: registered new interface driver usbhid
[ 0.802123] usbhid: USB HID core driver
[ 0.802318] ashmem: initialized
[ 0.810453] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[ 0.810527] IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
[ 0.810682] IPVS: ipvs loaded.
[ 0.811238] NET: Registered PF_INET6 protocol family
[ 0.812555] Segment Routing with IPv6
[ 0.812587] In-situ OAM (IOAM) with IPv6
[ 0.812663] NET: Registered PF_PACKET protocol family
[ 0.812704] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 0.812737] Key type dns_resolver registered
[ 0.813647] registered taskstats version 1
[ 0.813671] Loading compiled-in X.509 certificates
[ 0.814577] Key type .fscrypt registered
[ 0.814588] Key type fscrypt-provisioning registered
[ 0.823876] uart-pl011 fe201000.serial: there is not valid maps for state default
[ 0.824112] uart-pl011 fe201000.serial: cts_event_workaround enabled
[ 0.824230] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 19, base_baud = 0) is a PL011 rev2
[ 0.826595] fe201a00.serial: ttyAMA1 at MMIO 0xfe201a00 (irq = 19, base_baud = 0) is a PL011 rev2
[ 0.835303] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[ 0.835638] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[ 0.837973] mmc-bcm2835 fe300000.mmcnr: mmc_debug:0 mmc_debug2:0
[ 0.837989] mmc-bcm2835 fe300000.mmcnr: DMA channel allocated
[ 0.862577] sdhost: log_buf @ 000000008854c97d (c3400000)
[ 0.934784] mmc2: sdhost-bcm2835 loaded - DMA enabled (>1)
[ 0.938439] of_cfs_init
[ 0.938529] of_cfs_init: OK
[ 0.971688] mmc1: new high speed SDIO card at address 0001
[ 0.975566] mmc0: SDHCI controller on fe340000.mmc [fe340000.mmc] using ADMA
[ 0.976351] ALSA device list:
[ 0.976366] No soundcards found.
[ 0.976474] Warning: unable to open an initial console.
[ 1.007781] Freeing unused kernel memory: 33152K
[ 1.020493] Run /init as init process
[ 1.020501] with arguments:
[ 1.020506] /init
[ 1.020511] splash
[ 1.020516] with environment:
[ 1.020521] HOME=/
[ 1.020526] TERM=linux
[ 1.040580] mmc0: new DDR MMC card at address 0001
[ 1.041390] mmcblk0: mmc0:0001 AJTD4R 14.6 GiB
[ 1.044565] mmcblk0: p1 p2 p3 p4 < p5 p6 >
[ 1.045657] mmcblk0: mmc0:0001 AJTD4R 14.6 GiB
[ 1.046109] mmcblk0boot0: mmc0:0001 AJTD4R 4.00 MiB
[ 1.048179] mmcblk0boot1: mmc0:0001 AJTD4R 4.00 MiB
[ 1.050181] mmcblk0rpmb: mmc0:0001 AJTD4R 4.00 MiB, chardev (242:0)
[ 2.222182] zram0: detected capacity change from 0 to 8052584
[ 3.912351] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 3.960760] EXT4-fs (mmcblk0p5): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 4.452676] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none.
[ 4.474105] EXT4-fs (mmcblk0p6): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 4.636463] systemd[1]: System time before build time, advancing clock.
[ 4.814830] systemd[1097]: /lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 1.
[ 4.988947] systemd[1]: /lib/systemd/system/balena-host.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/balena-host.sock \xe2\x86\x92 /run/balena-host.sock; please update the unit file accordingly.
[ 5.012743] systemd[1]: /lib/systemd/system/balena-engine.socket:7: ListenStream= references a path below legacy directory /var/run/, updating /var/run/balena-engine.sock \xe2\x86\x92 /run/balena-engine.sock; please update the unit file accordingly.
[ 5.406856] fuse: init (API version 7.34)
[ 8.135961] mc: Linux media interface: v0.10
[ 8.151060] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.152102] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
[ 8.152125] [vc_sm_connected_init]: start
[ 8.176179] vc_sm_cma_vchi_init: failed to open VCHI service (-1)
[ 8.176200] [vc_sm_connected_init]: failed to initialize shared memory service
[ 8.203837] videodev: Linux video capture interface: v2.00
[ 8.211172] brcmstb-i2c fef04500.i2c: @97500hz registered in polling mode
[ 8.237475] i2c i2c-22: Added multiplexed i2c bus 0
[ 8.239012] i2c i2c-22: Added multiplexed i2c bus 10
[ 8.249299] brcmstb-i2c fef09500.i2c: @97500hz registered in polling mode
[ 8.253114] rpivid_hevc: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.256022] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.288051] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.288104] rtc-pcf85063 1-0051: registered as rtc0
[ 8.289286] rtc-pcf85063 1-0051: setting system clock to 2024-08-12T02:10:49 UTC (1723428649)
[ 8.301672] rpivid feb10000.codec: Device registered as /dev/video19
[ 8.312577] dwc2 fe980000.usb: supply vusb_d not found, using dummy regulator
[ 8.324474] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.356579] dwc2 fe980000.usb: supply vusb_a not found, using dummy regulator
[ 8.361614] bcm2835_isp: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.362248] bcm2835_codec: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.367421] bcm2835_v4l2: module is from the staging directory, the quality is unknown, you have been warned.
[ 8.378249] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 8.404067] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 8.412272] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 8.412788] cfg80211: Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[ 8.427723] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 8.428525] : Unregister from media controller
[ 8.428542] (efault): Unregistering node (null)[0] device node /dev/video0
[ 8.428551] (efault): Unregistering node (null)[0] device node /dev/video0
[ 8.428558] (efault): Unregistering node (null)[0] device node /dev/video0
[ 8.428565] (efault): Unregistering node (null)[0] device node /dev/video0
[ 8.432098] dwc2 fe980000.usb: DWC OTG Controller
[ 8.432139] dwc2 fe980000.usb: new USB bus registered, assigned bus number 1
[ 8.432180] dwc2 fe980000.usb: irq 23, io mem 0xfe980000
[ 8.432667] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.15
[ 8.432682] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 8.432691] usb usb1: Product: DWC OTG Controller
[ 8.432699] usb usb1: Manufacturer: Linux 5.15.92-v8 dwc2_hsotg
[ 8.432706] usb usb1: SerialNumber: fe980000.usb
[ 8.438092] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 8.440284] hub 1-0:1.0: USB hub found
[ 8.440547] hub 1-0:1.0: 1 port detected
[ 8.521554] vc4-drm gpu: bound fe400000.hvs (ops vc4_hvs_ops [vc4])
[ 8.522015] [drm] forcing HDMI-A-1 connector on
[ 8.531797] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 8.538928] Registered IR keymap rc-cec
[ 8.539082] rc rc0: vc4 as /devices/platform/soc/fef00700.hdmi/rc/rc0
[ 8.539218] input: vc4 as /devices/platform/soc/fef00700.hdmi/rc/rc0/input0
[ 8.543546] vc4-drm gpu: bound fef00700.hdmi (ops vc4_hdmi_ops [vc4])
[ 8.547161] [drm] Initialized v3d 1.0.0 20180419 for fec00000.v3d on minor 0
[ 8.551258] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 8.553056] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-compute-module.bin failed with error -2
[ 8.553806] Registered IR keymap rc-cec
[ 8.555466] rc rc1: vc4 as /devices/platform/soc/fef05700.hdmi/rc/rc1
[ 8.556889] input: vc4 as /devices/platform/soc/fef05700.hdmi/rc/rc1/input1
[ 8.566837] vc4-drm gpu: bound fef05700.hdmi (ops vc4_hdmi_ops [vc4])
[ 8.567146] vc4-drm gpu: bound fe004000.txp (ops vc4_txp_ops [vc4])
[ 8.567365] vc4-drm gpu: bound fe206000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 8.567562] vc4-drm gpu: bound fe207000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 8.567904] vc4-drm gpu: bound fe20a000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 8.568082] vc4-drm gpu: bound fe216000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 8.568259] vc4-drm gpu: bound fec12000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 8.597659] usbcore: registered new interface driver brcmfmac
[ 8.637701] zram: Can't change algorithm for initialized device
[ 8.691815] [drm] Initialized vc4 0.0.0 20140616 for gpu on minor 1
[ 8.800195] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 8.800371] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 8.806397] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Nov 1 2021 00:37:25 version 7.45.241 (1a2f2fa CY) FWID 01-703fd60
[ 8.816459] Console: switching to colour frame buffer device 100x30
[ 8.837620] vc4-drm gpu: [drm] fb0: vc4drmfb frame buffer device
[ 9.061839] edt_ft5x06 10-0038: supply vcc not found, using dummy regulator
[ 9.062041] edt_ft5x06 10-0038: supply iovcc not found, using dummy regulator
[ 9.068035] input: generic ft5x06 (79) as /devices/platform/soc/fe205000.i2c/i2c-22/i2c-10/10-0038/input/input2
[ 9.970747] Adding 4026288k swap on /dev/zram0. Priority:-2 extents:1 across:4026288k SSFS
[ 10.227466] uart-pl011 fe201000.serial: no DMA platform data
[ 11.848623] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[ 11.849774] bcmgenet fd580000.ethernet eth0: Link is Down
[ 11.911128] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[ 12.459589] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[ 12.636647] i2c_dev: i2c /dev entries driver
[ 16.945085] Bluetooth: Core ver 2.22
[ 16.945197] NET: Registered PF_BLUETOOTH protocol family
[ 16.945205] Bluetooth: HCI device and connection manager initialized
[ 16.945223] Bluetooth: HCI socket layer initialized
[ 16.945233] Bluetooth: L2CAP socket layer initialized
[ 16.945268] Bluetooth: SCO socket layer initialized
[ 16.953645] Bluetooth: HCI UART driver ver 2.3
[ 16.953667] Bluetooth: HCI UART protocol H4 registered
[ 16.953749] Bluetooth: HCI UART protocol Three-wire (H5) registered
[ 16.953967] Bluetooth: HCI UART protocol Broadcom registered
[ 17.291156] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 17.291181] Bluetooth: BNEP filters: protocol multicast
[ 17.291205] Bluetooth: BNEP socket layer initialized
[ 17.312359] NET: Registered PF_ALG protocol family
[ 25.156835] bcmgenet fd580000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[ 25.156920] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 31.254332] Bridge firewalling registered
[ 31.434796] Initializing XFRM netlink socket
[ 31.716469] cam-dummy-reg: disabling
[ 43.332731] systemd-journald[1218]: Time jumped backwards, rotating.
[ 45.316561] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 58.537081] uart-pl011 fe201a00.serial: no DMA platform data
[ 417.710224] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[ 824.727325] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[ 1231.710005] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
Related to this, my balena dashboard also has the dtoverlay=dt0 setting.
If you run the command lsblk you can see the mmc2 related to your external Micro SD Card?
I think I’ll try mounting the image on my laptop, adding and editing the files, and then using the image to flash the device instead of just adding the .dtbo file and dtoverlay.
Ok, good, I just wanted to confirm.
Yes, mmc2 is my micro SD card and it shows up in lsblk and I can mount it (I had to create the device nodes in my start up script for the container, but I think that’s related to the container distro). Shouldn’t matter if you copy the files over scp or locally on your laptop, I copied the files using scp while I was debugging and it worked fine.
Does your dmesg change if you remove the overlay? I assume it won’t even attempt mmc2 then? Which BalenaOS version are you on?
If I remove dtoverlay=sd0 from the /mnt/boot/config.txt file my ‘dmesg’ does not show the errors I described before related to mmc2:
mmc2: sdhost-bcm2835 loaded - DMA enabled (>1)
mmc2: invalid bus width
mmc2: error -22 when initializing SD card
Which BalenaOS version are you on?
I’m using balenaOS 5.4.0 This is my dmesg (without dtoverlay=sd0):
Spawning shell...
=============================================================
Welcome to balenaOS
=============================================================
root@59afb12:~# dmesg
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 5.15.92-v8 (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Wed Feb 8 16:47:50 UTC 2023
[ 0.000000] random: crng init done
[ 0.000000] Machine model: Raspberry Pi Compute Module 4 Rev 1.1
[ 0.000000] efi: UEFI not found.
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000001ac00000, size 320 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x000000003fffffff]
[ 0.000000] DMA32 [mem 0x0000000040000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x0000000037ffffff]
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[ 0.000000] On node 0, zone Normal: 16384 pages in unavailable ranges
[ 0.000000] percpu: Embedded 29 pages/cpu s78104 r8192 d32488 u118784
[ 0.000000] pcpu-alloc: s78104 r8192 d32488 u118784 alloc=29*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: detected: Spectre-v2
[ 0.000000] CPU features: detected: Spectre-v3a
[ 0.000000] CPU features: detected: Spectre-v4
[ 0.000000] CPU features: detected: Spectre-BHB
[ 0.000000] CPU features: kernel page table isolation forced ON by KASLR
[ 0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[ 0.000000] CPU features: detected: ARM erratum 1742098
[ 0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 2015744
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=E4:5F:01:F3:72:3F vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 dwc_otg.lpm_enable=0 rootfstype=ext4 rootwait dwc_otg.lpm_enable=0 rootwait console=null quiet splash vt.global_cursor_default=0 consoleblank=0 cgroup_enable=memory root=UUID=ba1eadef-d82d-4c8b-8c8e-0fdb99591f69 rootwait
[ 0.000000] cgroup: Enabling memory control group subsystem
[ 0.000000] Unknown kernel command line parameters "splash", will be passed to user space.
[ 0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] software IO TLB: mapped [mem 0x0000000034000000-0x0000000038000000] (64MB)
[ 0.000000] Memory: 7591724K/8192000K available (10240K kernel code, 1702K rwdata, 3944K rodata, 31040K init, 995K bss, 272596K reserved, 327680K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] ftrace: allocating 40658 entries in 159 pages
[ 0.000000] ftrace: allocated 159 pages with 6 groups
[ 0.000000] trace event string verifier disabled
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] Root IRQ handler: gic_handle_irq
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7ef00100, parent irq: 10)
[ 0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[ 0.000001] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[ 0.000382] Console: colour dummy device 80x25
[ 0.000476] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=216000)
[ 0.000509] pid_max: default: 32768 minimum: 301
[ 0.000670] LSM: Security Framework initializing
[ 0.000955] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 0.001052] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 0.005529] rcu: Hierarchical SRCU implementation.
[ 0.006845] EFI services will not be available.
[ 0.007474] smp: Bringing up secondary CPUs ...
[ 0.008907] Detected PIPT I-cache on CPU1
[ 0.009000] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[ 0.010499] Detected PIPT I-cache on CPU2
[ 0.010568] CPU2: Booted secondary processor 0x0000000002 [0x410fd083]
[ 0.012104] Detected PIPT I-cache on CPU3
[ 0.012172] CPU3: Booted secondary processor 0x0000000003 [0x410fd083]
[ 0.012409] smp: Brought up 1 node, 4 CPUs
[ 0.012434] SMP: Total of 4 processors activated.
[ 0.012450] CPU features: detected: 32-bit EL0 Support
[ 0.012463] CPU features: detected: 32-bit EL1 Support
[ 0.012478] CPU features: detected: CRC32 instructions
[ 0.016569] CPU: All CPU(s) started at EL2
[ 0.016650] alternatives: patching kernel code
[ 0.018596] devtmpfs: initialized
[ 0.034796] Enabled cp15_barrier support
[ 0.034841] Enabled setend support
[ 0.034865] KASLR enabled
[ 0.035254] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.035293] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.045701] pinctrl core: initialized pinctrl subsystem
[ 0.046636] DMI not present or invalid.
[ 0.047424] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 0.051744] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[ 0.052076] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[ 0.053114] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[ 0.053224] audit: initializing netlink subsys (disabled)
[ 0.053577] audit: type=2000 audit(0.052:1): state=initialized audit_enabled=0 res=1
[ 0.054296] thermal_sys: Registered thermal governor 'step_wise'
[ 0.054561] cpuidle: using governor menu
[ 0.055028] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.055289] ASID allocator initialised with 32768 entries
[ 0.055465] Serial: AMBA PL011 UART driver
[ 0.072445] bcm2835-mbox fe00b880.mailbox: mailbox enabled
[ 0.112559] raspberrypi-firmware soc:firmware: Attached to firmware from 2023-05-09T12:17:27, variant start_x
[ 0.116580] raspberrypi-firmware soc:firmware: Firmware hash is 30aa0d70ab280427ba04ebc718c81d4350b9d394
[ 0.183850] bcm2835-dma fe007000.dma: DMA legacy API manager, dmachans=0x1
[ 0.189208] vgaarb: loaded
[ 0.189741] SCSI subsystem initialized
[ 0.189997] usbcore: registered new interface driver usbfs
[ 0.190064] usbcore: registered new interface driver hub
[ 0.190141] usbcore: registered new device driver usb
[ 0.190635] usb_phy_generic phy: supply vcc not found, using dummy regulator
[ 0.190870] usb_phy_generic phy: dummy supplies not allowed for exclusive requests
[ 0.191227] pps_core: LinuxPPS API ver. 1 registered
[ 0.191245] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.191287] PTP clock support registered
[ 0.191568] Advanced Linux Sound Architecture Driver Initialized.
[ 0.193332] clocksource: Switched to clocksource arch_sys_counter
[ 0.269992] VFS: Disk quotas dquot_6.6.0
[ 0.270126] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.270328] FS-Cache: Loaded
[ 0.270600] CacheFiles: Loaded
[ 0.285583] NET: Registered PF_INET protocol family
[ 0.286382] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[ 0.292280] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[ 0.292367] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.292404] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[ 0.292761] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[ 0.293729] TCP: Hash tables configured (established 65536 bind 65536)
[ 0.294050] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[ 0.294169] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[ 0.294564] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 0.294619] PCI: CLS 0 bytes, default 64
[ 0.303609] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[ 0.304000] kvm [1]: IPA Size Limit: 44 bits
[ 0.305546] kvm [1]: vgic interrupt IRQ9
[ 0.305915] kvm [1]: Hyp mode initialized successfully
[ 0.320239] Initialise system trusted keyrings
[ 0.320695] workingset: timestamp_bits=46 max_order=21 bucket_order=0
[ 0.330006] zbud: loaded
[ 0.332274] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.334289] Key type asymmetric registered
[ 0.334318] Asymmetric key parser 'x509' registered
[ 0.334410] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[ 0.334814] io scheduler mq-deadline registered
[ 0.334834] io scheduler kyber registered
[ 0.344912] gpio-507 (ant1): hogged as output/high
[ 0.346339] gpio-511 (ant2): hogged as output/low
[ 0.348132] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[ 0.348185] brcm-pcie fd500000.pcie: No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[ 0.348283] brcm-pcie fd500000.pcie: MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[ 0.348379] brcm-pcie fd500000.pcie: IB MEM 0x0000000000..0x01ffffffff -> 0x0400000000
[ 0.665380] brcm-pcie fd500000.pcie: link down
[ 0.674431] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[ 0.678153] iproc-rng200 fe104000.rng: hwrng registered
[ 0.678740] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
[ 0.680617] gpiomem-bcm2835 fe200000.gpiomem: Initialised: Registers at 0xfe200000
[ 0.698040] brd: module loaded
[ 0.709954] loop: module loaded
[ 0.710785] zram: Added device: zram0
[ 0.711756] Loading iSCSI transport class v2.0-870.
[ 0.717930] tun: Universal TUN/TAP device driver, 1.6
[ 0.719099] bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
[ 0.785539] unimac-mdio unimac-mdio.-19: Broadcom UniMAC MDIO bus
[ 0.787525] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[ 0.787926] dwc_otg: FIQ enabled
[ 0.787941] dwc_otg: NAK holdoff enabled
[ 0.787953] dwc_otg: FIQ split-transaction FSM enabled
[ 0.787971] Module dwc_common_port init
[ 0.788615] usbcore: registered new interface driver uas
[ 0.788692] usbcore: registered new interface driver usb-storage
[ 0.788816] usbcore: registered new interface driver usbserial_generic
[ 0.788858] usbserial: USB Serial support registered for generic
[ 0.789129] mousedev: PS/2 mouse device common for all mice
[ 0.795628] sdhci: Secure Digital Host Controller Interface driver
[ 0.795652] sdhci: Copyright(c) Pierre Ossman
[ 0.796449] sdhci-pltfm: SDHCI platform and OF driver helper
[ 0.800735] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.801029] hid: raw HID events driver (C) Jiri Kosina
[ 0.801132] usbcore: registered new interface driver usbhid
[ 0.801139] usbhid: USB HID core driver
[ 0.801334] ashmem: initialized
[ 0.808995] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[ 0.809059] IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
[ 0.809203] IPVS: ipvs loaded.
[ 0.809882] NET: Registered PF_INET6 protocol family
[ 0.810793] Segment Routing with IPv6
[ 0.810815] In-situ OAM (IOAM) with IPv6
[ 0.810889] NET: Registered PF_PACKET protocol family
[ 0.810922] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 0.810944] Key type dns_resolver registered
[ 0.811814] registered taskstats version 1
[ 0.811837] Loading compiled-in X.509 certificates
[ 0.812629] Key type .fscrypt registered
[ 0.812638] Key type fscrypt-provisioning registered
[ 0.820569] uart-pl011 fe201000.serial: there is not valid maps for state default
[ 0.822085] uart-pl011 fe201000.serial: cts_event_workaround enabled
[ 0.822212] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 19, base_baud = 0) is a PL011 rev2
[ 0.828889] bcm2835-aux-uart fe215040.serial: there is not valid maps for state default
[ 0.829487] fe215040.serial: ttyS0 at MMIO 0xfe215040 (irq = 22, base_baud = 62500000) is a 16550
[ 0.830526] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[ 0.832169] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[ 0.836245] mmc-bcm2835 fe300000.mmcnr: mmc_debug:0 mmc_debug2:0
[ 0.836266] mmc-bcm2835 fe300000.mmcnr: DMA channel allocated
[ 0.869633] of_cfs_init
[ 0.869768] of_cfs_init: OK
[ 0.901438] mmc0: SDHCI controller on fe340000.mmc [fe340000.mmc] using ADMA
[ 0.902430] ALSA device list:
[ 0.902444] No soundcards found.
[ 0.902516] Warning: unable to open an initial console.
[ 0.937824] Freeing unused kernel memory: 31040K
[ 0.937939] Run /init as init process
[ 0.937945] with arguments:
[ 0.937950] /init
[ 0.937955] splash
[ 0.937960] with environment:
[ 0.937965] HOME=/
[ 0.937970] TERM=linux
[ 0.952909] mmc1: new high speed SDIO card at address 0001
[ 0.967794] mmc0: new DDR MMC card at address 0001
[ 0.968657] mmcblk0: mmc0:0001 BJTD4R 29.1 GiB
[ 0.971882] mmcblk0: p1 p2 p3 p4 < p5 p6 >
[ 0.972995] mmcblk0: mmc0:0001 BJTD4R 29.1 GiB
[ 0.973507] mmcblk0boot0: mmc0:0001 BJTD4R 4.00 MiB
[ 0.975682] mmcblk0boot1: mmc0:0001 BJTD4R 4.00 MiB
[ 0.977618] mmcblk0rpmb: mmc0:0001 BJTD4R 4.00 MiB, chardev (242:0)
[ 2.072757] zram0: detected capacity change from 0 to 7950448
[ 3.591598] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 3.639155] EXT4-fs (mmcblk0p5): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 4.109790] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none.
[ 4.134655] EXT4-fs (mmcblk0p6): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 4.297243] systemd[1]: System time before build time, advancing clock.
[ 4.455727] systemd[1063]: /lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 1.
[ 4.628509] systemd[1]: /lib/systemd/system/balena-host.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/balena-host.sock \xe2\x86\x92 /run/balena-host.sock; please update the unit file accordingly.
[ 4.649227] systemd[1]: /lib/systemd/system/balena-engine.socket:7: ListenStream= references a path below legacy directory /var/run/, updating /var/run/balena-engine.sock \xe2\x86\x92 /run/balena-engine.sock; please update the unit file accordingly.
[ 5.032790] fuse: init (API version 7.34)
[ 6.811864] mc: Linux media interface: v0.10
[ 6.867665] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.917656] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
[ 6.917688] [vc_sm_connected_init]: start
[ 6.934442] [vc_sm_connected_init]: installed successfully
[ 6.943504] i2c 0-0036: Fixing up cyclic dependency with fe800000.csi
[ 6.943905] i2c i2c-22: Added multiplexed i2c bus 0
[ 6.948995] i2c 10-0036: Fixing up cyclic dependency with fe801000.csi
[ 6.949521] i2c i2c-22: Added multiplexed i2c bus 10
[ 6.953226] videodev: Linux video capture interface: v2.00
[ 6.978385] brcmstb-i2c fef04500.i2c: @97500hz registered in polling mode
[ 6.978883] brcmstb-i2c fef09500.i2c: @97500hz registered in polling mode
[ 7.018374] rpivid_hevc: module is from the staging directory, the quality is unknown, you have been warned.
[ 7.020924] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 7.027233] rpivid feb10000.codec: Device registered as /dev/video19
[ 7.028826] bcm2835_isp: module is from the staging directory, the quality is unknown, you have been warned.
[ 7.030091] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 7.032226] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 7.034565] bcm2835-isp bcm2835-isp: Device node output[0] registered as /dev/video13
[ 7.036883] bcm2835_codec: module is from the staging directory, the quality is unknown, you have been warned.
[ 7.040558] bcm2835_v4l2: module is from the staging directory, the quality is unknown, you have been warned.
[ 7.052044] bcm2835-isp bcm2835-isp: Device node capture[0] registered as /dev/video14
[ 7.075658] bcm2835-codec bcm2835-codec: Device registered as /dev/video10
[ 7.075706] bcm2835-codec bcm2835-codec: Loaded V4L2 decode
[ 7.091774] bcm2835-isp bcm2835-isp: Device node capture[1] registered as /dev/video15
[ 7.093618] bcm2835-isp bcm2835-isp: Device node stats[2] registered as /dev/video16
[ 7.093644] bcm2835-isp bcm2835-isp: Register output node 0 with media controller
[ 7.093656] bcm2835-isp bcm2835-isp: Register capture node 1 with media controller
[ 7.093666] bcm2835-isp bcm2835-isp: Register capture node 2 with media controller
[ 7.093684] bcm2835-isp bcm2835-isp: Register capture node 3 with media controller
[ 7.106326] bcm2835-codec bcm2835-codec: Device registered as /dev/video11
[ 7.106405] bcm2835-codec bcm2835-codec: Loaded V4L2 encode
[ 7.108146] bcm2835-isp bcm2835-isp: Device node output[0] registered as /dev/video20
[ 7.118018] bcm2835-isp bcm2835-isp: Device node capture[0] registered as /dev/video21
[ 7.129376] bcm2835-codec bcm2835-codec: Device registered as /dev/video12
[ 7.129445] bcm2835-codec bcm2835-codec: Loaded V4L2 isp
[ 7.136728] bcm2835-codec bcm2835-codec: Device registered as /dev/video18
[ 7.136776] bcm2835-codec bcm2835-codec: Loaded V4L2 image_fx
[ 7.143468] bcm2835-codec bcm2835-codec: Device registered as /dev/video31
[ 7.143507] bcm2835-codec bcm2835-codec: Loaded V4L2 encode_image
[ 7.147089] bcm2835-isp bcm2835-isp: Device node capture[1] registered as /dev/video22
[ 7.165108] vc4-drm gpu: bound fe400000.hvs (ops vc4_hvs_ops [vc4])
[ 7.179897] Registered IR keymap rc-cec
[ 7.183970] bcm2835-isp bcm2835-isp: Device node stats[2] registered as /dev/video23
[ 7.184004] bcm2835-isp bcm2835-isp: Register output node 0 with media controller
[ 7.184018] bcm2835-isp bcm2835-isp: Register capture node 1 with media controller
[ 7.184032] bcm2835-isp bcm2835-isp: Register capture node 2 with media controller
[ 7.184057] bcm2835-isp bcm2835-isp: Register capture node 3 with media controller
[ 7.185584] bcm2835-isp bcm2835-isp: Loaded V4L2 bcm2835-isp
[ 7.202161] rc rc0: vc4 as /devices/platform/soc/fef00700.hdmi/rc/rc0
[ 7.265187] input: vc4 as /devices/platform/soc/fef00700.hdmi/rc/rc0/input0
[ 7.279838] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 7.280446] [drm] Initialized v3d 1.0.0 20180419 for fec00000.v3d on minor 1
[ 7.307523] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 7.308175] cfg80211: Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[ 7.374462] zram: Can't change algorithm for initialized device
[ 7.429768] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 7.441482] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 7.442266] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-compute-module.bin failed with error -2
[ 7.463480] vc4-drm gpu: bound fef00700.hdmi (ops vc4_hdmi_ops [vc4])
[ 7.480673] usbcore: registered new interface driver brcmfmac
[ 7.483607] Registered IR keymap rc-cec
[ 7.500430] rc rc1: vc4 as /devices/platform/soc/fef05700.hdmi/rc/rc1
[ 7.589539] input: vc4 as /devices/platform/soc/fef05700.hdmi/rc/rc1/input1
[ 7.671630] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 7.671781] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 7.676758] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Nov 1 2021 00:37:25 version 7.45.241 (1a2f2fa CY) FWID 01-703fd60
[ 7.799793] ov5647 0-0036: Consider updating driver ov5647 to match on endpoints
[ 7.810373] vc4-drm gpu: bound fef05700.hdmi (ops vc4_hdmi_ops [vc4])
[ 7.810775] vc4-drm gpu: bound fe004000.txp (ops vc4_txp_ops [vc4])
[ 7.811013] vc4-drm gpu: bound fe206000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 7.811222] vc4-drm gpu: bound fe207000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 7.811418] vc4-drm gpu: bound fe20a000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 7.811577] vc4-drm gpu: bound fe216000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 7.811772] vc4-drm gpu: bound fec12000.pixelvalve (ops vc4_crtc_ops [vc4])
[ 7.880090] ov5647 10-0036: Consider updating driver ov5647 to match on endpoints
[ 7.881890] [drm] Initialized vc4 0.0.0 20140616 for gpu on minor 0
[ 7.882481] vc4-drm gpu: [drm] Cannot find any crtc or sizes
[ 8.586118] Adding 3975220k swap on /dev/zram0. Priority:-2 extents:1 across:3975220k SSFS
[ 8.836068] uart-pl011 fe201000.serial: no DMA platform data
[ 10.594654] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[ 10.595820] bcmgenet fd580000.ethernet eth0: Link is Down
[ 10.649051] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[ 11.200830] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[ 11.382188] i2c_dev: i2c /dev entries driver
[ 13.352305] Bridge firewalling registered
[ 13.555874] Initializing XFRM netlink socket
[ 15.075394] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[ 15.182127] br-3191b3157077: port 1(vethf180a12) entered blocking state
[ 15.182149] br-3191b3157077: port 1(vethf180a12) entered disabled state
[ 15.182322] device vethf180a12 entered promiscuous mode
[ 15.182616] br-3191b3157077: port 1(vethf180a12) entered blocking state
[ 15.182629] br-3191b3157077: port 1(vethf180a12) entered forwarding state
[ 15.186379] br-3191b3157077: port 1(vethf180a12) entered disabled state
[ 15.590741] Bluetooth: Core ver 2.22
[ 15.590859] NET: Registered PF_BLUETOOTH protocol family
[ 15.590866] Bluetooth: HCI device and connection manager initialized
[ 15.590886] Bluetooth: HCI socket layer initialized
[ 15.590896] Bluetooth: L2CAP socket layer initialized
[ 15.590915] Bluetooth: SCO socket layer initialized
[ 15.601065] Bluetooth: HCI UART driver ver 2.3
[ 15.601089] Bluetooth: HCI UART protocol H4 registered
[ 15.601176] Bluetooth: HCI UART protocol Three-wire (H5) registered
[ 15.605532] Bluetooth: HCI UART protocol Broadcom registered
[ 16.084913] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 16.084939] Bluetooth: BNEP filters: protocol multicast
[ 16.084960] Bluetooth: BNEP socket layer initialized
[ 16.112275] NET: Registered PF_ALG protocol family
[ 16.280524] supervisor0: port 1(vethac9ec76) entered blocking state
[ 16.280547] supervisor0: port 1(vethac9ec76) entered disabled state
[ 16.280749] device vethac9ec76 entered promiscuous mode
[ 16.281055] supervisor0: port 1(vethac9ec76) entered blocking state
[ 16.281071] supervisor0: port 1(vethac9ec76) entered forwarding state
[ 16.282741] supervisor0: port 1(vethac9ec76) entered disabled state
[ 18.032155] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 19.154811] eth0: renamed from vethae21b67
[ 19.186697] IPv6: ADDRCONF(NETDEV_CHANGE): vethf180a12: link becomes ready
[ 19.186847] br-3191b3157077: port 1(vethf180a12) entered blocking state
[ 19.186860] br-3191b3157077: port 1(vethf180a12) entered forwarding state
[ 19.186999] IPv6: ADDRCONF(NETDEV_CHANGE): br-3191b3157077: link becomes ready
[ 19.218884] eth1: renamed from vethdc2505b
[ 19.253049] IPv6: ADDRCONF(NETDEV_CHANGE): vethac9ec76: link becomes ready
[ 19.253203] supervisor0: port 1(vethac9ec76) entered blocking state
[ 19.253216] supervisor0: port 1(vethac9ec76) entered forwarding state
[ 19.253419] IPv6: ADDRCONF(NETDEV_CHANGE): supervisor0: link becomes ready
[ 31.709428] cam-dummy-reg: disabling
[ 36.830969] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
Ok, so at least you know it’s attempting to load the overlay, that’s good. I’m also on 5.4.0 (tried with earlier versions and they worked as well). Sorry, I’m afraid I don’t have any other ideas, “it works here” only goes so far 
Let me know if you need me to check on anything in my system for comparison.
Could there be any other overlay/driver using the same IO pin?
Just to clarify a few points:
-
Did you compile the .dts file on raspbian and then transfer the .dtbo file from raspbian to BalenaOS or did you run BalenaOS in a docker container to compile the .dts file in the same environment as the .dtbo will be?
-
In which balenaOS directory did you place your sd0.dtbo file? In mnt/boot/overlays how did I do it?
-
I just noticed that your dmesg has the start_cd variant:
raspberrypi-firmware soc:firmware: Attached to firmware from 2023-05-09T12:16:50, variant start_cdWhile I have variant start_x:
raspberrypi-firmware soc:firmware: Attached to firmware from 2023-05-09T12:16:50, variant start_xHowever, my application needs to work with two MIPI-CSI OV5647 cameras on BalenaOS. Without this start_x parameter, cameras are not detected. I will remove this variable and check if BalenaOS can now detect the SD card. Also note that in Raspbian I can work with both MIPI-CSI cameras and also with the external Micro SD card. @alexgg Can the variant start_x be an issue to the External MicroSD Card?
Also I did reverse engineering to the .dtbo file to check if the compilation worked as expected.
I ran the command:
dtc -I dtb -O dts -o output.dts sd0.dtbo
And the result is:
/dts-v1/;
/ {
compatible = "brcm,bcm2711";
fragment@0 {
target = <0xffffffff>;
__overlay__ {
pinctrl-names = "default";
pinctrl-0 = <0x01>;
bus-width = <0x04>;
brcm,overclock-50 = <0x00>;
brcm,pio-limit = <0x01>;
status = "okay";
phandle = <0x02>;
};
};
fragment@1 {
target = <0xffffffff>;
__overlay__ {
sdhost_pins {
brcm,pins = <0x16 0x17 0x18 0x19 0x1a 0x1b>;
brcm,function = <0x04 0x04 0x04 0x04 0x04 0x04>;
brcm,pull = <0x00 0x02 0x02 0x02 0x02 0x02>;
phandle = <0x01>;
};
};
};
__overrides__ {
overclock_50 = <0x02 0x6272636d 0x2c6f7665 0x72636c6f 0x636b2d35 0x303a3000>;
force_pio = <0x02 0x6272636d 0x2c666f72 0x63652d70 0x696f3f00>;
pio_limit = [00 00 00 02 62 72 63 6d 2c 70 69 6f 2d 6c 69 6d 69 74 3a 30 00];
debug = <0x02 0x6272636d 0x2c646562 0x75673f00>;
};
__symbols__ {
frag0 = "/fragment@0/__overlay__";
sdhost_pins = "/fragment@1/__overlay__/sdhost_pins";
};
__fixups__ {
sdhost = "/fragment@0:target:0";
gpio = "/fragment@1:target:0";
};
__local_fixups__ {
fragment@0 {
__overlay__ {
pinctrl-0 = <0x00>;
};
};
__overrides__ {
overclock_50 = <0x00>;
force_pio = <0x00>;
pio_limit = <0x00>;
debug = <0x00>;
};
};
};
Could you spend 5 minutes to reverse engineer your .dtbo file just to check if the result is the same as mine?
Could there be any other overlay/driver using the same IO pin?
Related to interface conflicts that could use the same gpios, I believe there is no problem with this since the external MicroSD card on the sd0 interface works correctly on raspbian with the same overlays in config.txt that I am using on balenaOS.
Thanks again for spending some time helping me debug my issue. At least I know it’s possible.
@HakanL Well… I finally solved the problem.
GPIO23 which is used by the SD0 interface is damaged. Someone may have been careless and used 5V instead of 3.3V on this pin in the past and damaged it. Luckily I had another cm4 here, I replaced it and now it works.
Thanks a lot for your time and help.
Best regards.
1 Like
Great you solved it!
Just for clarity I’ll still answer the questions.
- I can’t remember at this point, I may have compiled it on another Raspberry Pi, but to my knowledge it doesn’t matter where you compile it, I believe it’s just a binary serialization of the source file.
- /mnt/boot/overlays
- Hmm, I’m actually not sure where that’s coming from, I can’t remember that being something I’ve specified, but I don’t have any cameras, only a HDMI touch screen.
The reason I asked about another overlay/driver using the same IO pin was if your Raspbian distro perhaps didn’t enable a particular driver that used a pin, but then you’d probably get a different error, I was just throwing out ideas