I am deploying my first balena project and wanted to record the problem and solution to a weird issue I had. I bought a couple of Raspberry Pi Zero 2 W’s and it turns out that they’re a bit underpowered for what I want to do, but they do work… after this fix!
They would boot fine, but then would not finish updating. They would make it so far, then the green light on the board would flash 4 times quickly then pause. This apparently indicates a network issue. I was able to ssh in to one while it was starting the update and captured the kernel log before the wifi completely died on me:
Jul 27 22:00:25 0acb7ae kernel: brcmfmac: brcmf_sdio_isr: failed backplane access
Jul 27 22:00:25 0acb7ae kernel: brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -110
Jul 27 22:00:25 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK
Jul 27 22:00:25 0acb7ae kernel: brcmfmac: brcmf_sdio_hdparse: HW superframe header length error
Jul 27 22:00:25 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame
Jul 27 22:00:25 0acb7ae kernel: brcmfmac: brcmf_sdio_hdparse: HW header length too long
Jul 27 22:00:25 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: terminate frame
Jul 27 22:00:26 0acb7ae kernel: brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -110
Jul 27 22:00:26 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK
Jul 27 22:00:26 0acb7ae kernel: brcmfmac: brcmf_sdio_rxglom: descriptor len 3 bad: 0
Jul 27 22:00:26 0acb7ae kernel: ------------[ cut here ]------------
Jul 27 22:00:26 0acb7ae kernel: WARNING: CPU: 2 PID: 1388 at drivers/mmc/host/bcm2835-mmc.c:495 bcm2835_mmc_transfer_dma+0x2c/0x1a8
Jul 27 22:00:26 0acb7ae kernel: Modules linked in: ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_filter ip6_tables xt_MASQUERADE nf_conntrack_netlink nfnetlink br_netfilter xt_owner i2c_dev cmac algif_hash aes_arm64 algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc spidev brcmfmac vc4 brcmutil sha256_generic snd_soc_hdmi_codec cec cfg80211 drm_kms_helper bcm2835_codec(C) bcm2835_v4l2(C) syscopyarea bcm2835_isp(C) sysfillrect rfkill bcm2835_mmal_vchiq(C) v4l2_mem2mem sysimgblt raspberrypi_hwmon fb_sys_fops videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common videodev sdhci_iproc mc snd_bcm2835(C) vc_sm_cma(C) spi_bcm2835 uio_pdrv_genirq uio drm sch_fq_codel drm_panel_orientation_quirks backlight fuse
Jul 27 22:00:26 0acb7ae kernel: CPU: 2 PID: 1388 Comm: kworker/u9:1 Tainted: G C 5.15.92-v8 #1
Jul 27 22:00:26 0acb7ae kernel: Hardware name: Raspberry Pi Zero 2 W Rev 1.0 (DT)
Jul 27 22:00:26 0acb7ae kernel: Workqueue: brcmf_wq/mmc1:0001:1 brcmf_sdio_dataworker [brcmfmac]
Jul 27 22:00:26 0acb7ae kernel: pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Jul 27 22:00:26 0acb7ae kernel: pc : bcm2835_mmc_transfer_dma+0x2c/0x1a8
Jul 27 22:00:26 0acb7ae kernel: lr : bcm2835_mmc_request+0x90/0xac
Jul 27 22:00:26 0acb7ae kernel: sp : ffffffc008c0b960
Jul 27 22:00:26 0acb7ae kernel: x29: ffffffc008c0b960 x28: 0000000000000000 x27: 0000000000000000
Jul 27 22:00:26 0acb7ae kernel: x26: 0000000020000000 x25: 0000000000080000 x24: 0000000001000000
Jul 27 22:00:26 0acb7ae kernel: x23: 0000000000000001 x22: ffffffc008c0bb08 x21: 0000000000000000
Jul 27 22:00:26 0acb7ae kernel: x20: ffffffc008c0bb50 x19: ffffff800381ad80 x18: 0000000000000000
Jul 27 22:00:26 0acb7ae kernel: x17: ffffffa3c71f9000 x16: ffffffdc55d09d20 x15: 0000000000004000
Jul 27 22:00:26 0acb7ae kernel: x14: 0000000000000000 x13: 802e7e8100010126 x12: a193c01dd6ee0800
Jul 27 22:00:26 0acb7ae kernel: x11: 00a42f2500900026 x10: 3706d80439470b60 x9 : ffffffdc55d17534
Jul 27 22:00:26 0acb7ae kernel: x8 : ffffff800381ae50 x7 : 0000000000000000 x6 : 000000000000003f
Jul 27 22:00:26 0acb7ae kernel: x5 : 0000000000000040 x4 : 0000000000000008 x3 : 0000000000000000
Jul 27 22:00:26 0acb7ae kernel: x2 : ffffff80049e9e40 x1 : 0000000000000000 x0 : 0000000000000000
Jul 27 22:00:26 0acb7ae kernel: Call trace:
Jul 27 22:00:26 0acb7ae kernel: bcm2835_mmc_transfer_dma+0x2c/0x1a8
Jul 27 22:00:26 0acb7ae kernel: bcm2835_mmc_request+0x90/0xac
Jul 27 22:00:26 0acb7ae kernel: __mmc_start_request+0xa8/0x110
Jul 27 22:00:26 0acb7ae kernel: mmc_start_request+0x90/0xb0
Jul 27 22:00:26 0acb7ae kernel: mmc_wait_for_req+0x80/0xc4
Jul 27 22:00:26 0acb7ae kernel: mmc_io_rw_extended+0x1e0/0x2c4
Jul 27 22:00:26 0acb7ae kernel: sdio_io_rw_ext_helper+0xd4/0x148
Jul 27 22:00:26 0acb7ae kernel: sdio_readsb+0x28/0x34
Jul 27 22:00:26 0acb7ae kernel: brcmf_sdiod_skbuff_read+0x90/0x94 [brcmfmac]
Jul 27 22:00:26 0acb7ae kernel: brcmf_sdiod_recv_pkt+0x50/0x64 [brcmfmac]
Jul 27 22:00:26 0acb7ae kernel: brcmf_sdio_dataworker+0xff4/0x1a6c [brcmfmac]
Jul 27 22:00:26 0acb7ae kernel: process_one_work+0x1d4/0x298
Jul 27 22:00:26 0acb7ae kernel: worker_thread+0x1e0/0x278
Jul 27 22:00:26 0acb7ae kernel: kthread+0xf8/0x108
Jul 27 22:00:26 0acb7ae kernel: ret_from_fork+0x10/0x20
Jul 27 22:00:26 0acb7ae kernel: ---[ end trace 5f6fea7f8f6a1b70 ]---
Jul 27 22:00:34 0acb7ae kernel: net_ratelimit: 58 callbacks suppressed
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: count never zeroed: last 0x04e0
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_hdparse: HW header checksum error
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: terminate frame
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_htclk: Failed access turning clock off: -110
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_readframes: RXHEADER FAILED: -110
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: abort command, terminate frame, send NAK
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_rxglom: descriptor len 3 bad: 0
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_hdparse: HW header length too long
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_rxfail: terminate frame
Jul 27 22:00:34 0acb7ae kernel: brcmfmac: brcmf_sdio_hdparse: HW superframe header length error
I found [someone|https://forums.raspberrypi.com/viewtopic.php?t=322591] who suggested this might be due to overheating so I applied those config options as variables to my balena fleet:
BALENA_HOST_CONFIG_arm_freq=600
BALENA_HOST_CONFIG_gpu_freq=300
BALENA_HOST_CONFIG_sdram_freq=400
And now, after a couple of reboots to get the changes to apply, the wifi is working fine!