Alsa problems - aplay: no soundcards found...

sighmon · November 12, 2020, 12:36am

Hi there,

We’re having some issues with BalenaOS devices this morning erroring out with aplay: device_list:272: no soundcards found...

Changing the content or upgrading BalenaOS to the latest version doesn’t seem to help.

BalenaOS: balenaOS 2.58.6+rev1

Devices: Raspberry Pi 4

Number affected: 6

Application code: https://github.com/ACMILabs/media-player

Full error logs:

X.Org X Server 1.20.4
12.11.20 11:03:08 (+1030)  mediaplayer  X Protocol Version 11, Revision 0
12.11.20 11:03:08 (+1030)  mediaplayer  Build Operating System: Linux 4.15.0-99-generic armv8l Raspbian
12.11.20 11:03:08 (+1030)  mediaplayer  Current Operating System: Linux 9c73a8562368 5.4.58 #1 SMP PREEMPT Mon Oct 19 18:41:44 UTC 2020 aarch64
12.11.20 11:03:08 (+1030)  mediaplayer  Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_headphones=1 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708_fb.fbswap=1 smsc95xx.macaddr=DC:A6:32:50:D4:0A vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=null rootfstype=ext4 rootwait vt.global_cursor_default=0 root=UUID=ba1eadef-f992-439b-8a88-7c8211ac1f77 rootwait
12.11.20 11:03:08 (+1030)  mediaplayer  Build Date: 27 August 2020  08:51:48AM
12.11.20 11:03:08 (+1030)  mediaplayer  xorg-server 2:1.20.4-1+rpt2+deb10u1 (https://www.debian.org/support) 
12.11.20 11:03:08 (+1030)  mediaplayer  Current version of pixman: 0.36.0
12.11.20 11:03:08 (+1030)  mediaplayer          Before reporting problems, check http://wiki.x.org
12.11.20 11:03:08 (+1030)  mediaplayer          to make sure that you have the latest version.
12.11.20 11:03:08 (+1030)  mediaplayer  Markers: (--) probed, (**) from config file, (==) default setting,
12.11.20 11:03:08 (+1030)  mediaplayer          (++) from command line, (!!) notice, (II) informational,
12.11.20 11:03:08 (+1030)  mediaplayer          (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
12.11.20 11:03:08 (+1030)  mediaplayer  (==) Log file: "/var/log/Xorg.1.log", Time: Thu Nov 12 00:33:08 2020
12.11.20 11:03:08 (+1030)  mediaplayer  (==) Using system config directory "/usr/share/X11/xorg.conf.d"
12.11.20 11:03:13 (+1030)  mediaplayer  Displays in use after starting X
12.11.20 11:03:13 (+1030)  mediaplayer  DISPLAY=:0 DISPLAY=:1
12.11.20 11:03:13 (+1030)  mediaplayer  Setting display to: DISPLAY=:1
12.11.20 11:03:13 (+1030)  mediaplayer  xrandr: Failed to get size of gamma for output default
12.11.20 11:03:13 (+1030)  mediaplayer  xrandr: Failed to get size of gamma for output default
12.11.20 11:03:13 (+1030)  mediaplayer  Screen resolution parsed as: 1824x984
12.11.20 11:03:14 (+1030)  mediaplayer  aplay: device_list:272: no soundcards found...
12.11.20 11:03:14 (+1030)  mediaplayer  Scanning audio devices for match to hdmi|hdmi 0
12.11.20 11:03:14 (+1030)  mediaplayer  AUDIO_DEVICE_REGEX hdmi|hdmi 0 did not match any audio devices. Using default audio settings instead.
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib conf.c:4568:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib conf.c:5047:(snd_config_expand) Evaluate error: No such file or directory
12.11.20 11:03:14 (+1030)  mediaplayer  ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM default

samothx · November 12, 2020, 12:48am

Hi Simon,
to clarify your problem, could you please answer the following questions:

how many devices are affected by the problem ?
have you made any changes to the devices that could have caused the problem like updates to your software, updates to balena-OS, hardware changes ?
what device type are you using ?

Cheers Thomas

sighmon · November 12, 2020, 1:34am

@samothx Sorry I was light on details… I’ve updated the post, more answers here:

how many devices are affected by the problem ?

We have a wall of 60 Raspberry Pi 4s, and it has affected 6 so far.

have you made any changes to the devices that could have caused the problem like updates to your software, updates to balena-OS, hardware changes ?

No BalenaOS updates, hardware changes were made. We just pushed a different video onto the device to play. But when pushing an original test video that was previously playing fine back onto it the error still occurs.

what device type are you using ?

Raspberry Pi 4.

sighmon · November 12, 2020, 1:36am

@samothx Happy to provide support access to one that’s working and one that’s broken if that helps?

samothx · November 12, 2020, 1:41am

That would be helpful. You can send the devices UUID via PM.

samothx · November 12, 2020, 2:18am

The best path to debug this would probably be to open a shell in the mediaplayer container and debug alsa utils. Unfortunately the mediaplayer container is in a restart loop, which makes that impossible. So the best way to go would be to modify the container to not restart on failure, so that you can get a shell on it. I found some advice on debugging alsa here : https://tldp.org/HOWTO/Alsa-sound-6.html .

sighmon · November 12, 2020, 7:15am

I’ve got the device playing back the video without audio.

Here’s some output to aid debugging:

$ cat /proc/asound/cards 
--- no soundcards ---

$ aplay -l
aplay: device_list:272: no soundcards found...

$ aplay -L
null
    Discard all samples (playback) or generate zero samples (capture)

$ cat /proc/modules
snd_bcm2835 32768 0 - Live 0xffffff80009e6000 (C)
ip6t_REJECT 16384 1 - Live 0xffffff80012d8000
nf_reject_ipv6 16384 1 ip6t_REJECT, Live 0xffffff80012d0000
ip6table_filter 16384 1 - Live 0xffffff80012c8000
ipt_REJECT 16384 1 - Live 0xffffff80012c0000
nf_reject_ipv4 16384 1 ipt_REJECT, Live 0xffffff80012b8000
xt_state 16384 0 - Live 0xffffff80012b0000
ip6_tables 28672 1 ip6table_filter, Live 0xffffff80012a4000
bnep 20480 2 - Live 0xffffff800129a000
ipt_MASQUERADE 16384 3 - Live 0xffffff8001292000
nf_conntrack_netlink 49152 0 - Live 0xffffff8001280000
nfnetlink 16384 2 nf_conntrack_netlink, Live 0xffffff8001277000
hci_uart 45056 1 - Live 0xffffff8001266000
btbcm 16384 1 hci_uart, Live 0xffffff800125e000
serdev 20480 1 hci_uart, Live 0xffffff8001253000
bluetooth 438272 24 bnep,hci_uart,btbcm, Live 0xffffff80011d5000
ecdh_generic 28672 1 bluetooth, Live 0xffffff80011c9000
br_netfilter 24576 0 - Live 0xffffff80011be000
xt_owner 16384 0 - Live 0xffffff80011b6000
i2c_dev 20480 0 - Live 0xffffff80011ac000
brcmfmac 319488 0 - Live 0xffffff800108c000
brcmutil 20480 1 brcmfmac, Live 0xffffff8001082000
vc4 200704 5 - Live 0xffffff8000e57000
cfg80211 757760 1 brcmfmac, Live 0xffffff8000d66000
snd_soc_core 217088 1 vc4, Live 0xffffff8000d1e000
bcm2835_codec 40960 0 - Live 0xffffff8000d0e000 (C)
snd_compress 20480 1 snd_soc_core, Live 0xffffff8000d04000
bcm2835_v4l2 49152 0 - Live 0xffffff8000cf2000 (C)
rfkill 32768 5 bluetooth,cfg80211, Live 0xffffff8000cc8000
snd_pcm_dmaengine 16384 1 snd_soc_core, Live 0xffffff8000cbf000
v3d 73728 3 - Live 0xffffff8000ca4000
v4l2_mem2mem 28672 1 bcm2835_codec, Live 0xffffff8000c97000
snd_pcm 126976 4 snd_bcm2835,vc4,snd_soc_core,snd_pcm_dmaengine, Live 0xffffff8000c6c000
bcm2835_mmal_vchiq 32768 2 bcm2835_codec,bcm2835_v4l2, Live 0xffffff8000c50000 (C)
v4l2_common 16384 1 bcm2835_v4l2, Live 0xffffff8000c47000
gpu_sched 28672 1 v3d, Live 0xffffff8000c3a000
snd_timer 40960 1 snd_pcm, Live 0xffffff8000c2a000
videobuf2_dma_contig 20480 1 bcm2835_codec, Live 0xffffff8000c20000
videobuf2_vmalloc 16384 1 bcm2835_v4l2, Live 0xffffff8000c17000
snd 86016 5 snd_bcm2835,snd_soc_core,snd_compress,snd_pcm,snd_timer, Live 0xffffff8000bf8000
videobuf2_memops 16384 2 videobuf2_dma_contig,videobuf2_vmalloc, Live 0xffffff8000be8000
raspberrypi_hwmon 16384 0 - Live 0xffffff8000be0000
videobuf2_v4l2 28672 3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem, Live 0xffffff8000bc6000
hwmon 20480 1 raspberrypi_hwmon, Live 0xffffff8000bbc000
videobuf2_common 49152 4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2, Live 0xffffff8000b96000
videodev 249856 6 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,v4l2_common,videobuf2_v4l2,videobuf2_common, Live 0xffffff8000afa000
media 40960 3 bcm2835_codec,v4l2_mem2mem,videodev, Live 0xffffff8000ad8000
vc_sm_cma 36864 2 bcm2835_mmal_vchiq, Live 0xffffff8000ac9000 (C)
rpivid_mem 16384 0 - Live 0xffffff8000ac1000
uio_pdrv_genirq 16384 0 - Live 0xffffff8000a61000
uio 20480 1 uio_pdrv_genirq, Live 0xffffff80009ef000
sch_fq_codel 20480 8 - Live 0xffffff80009e0000

sighmon · November 12, 2020, 7:16am

One hint from another broken device is that it’s stuck trying to re-download an update:

Failed to download image 'registry2.balena-cloud.com/v2/4818988<SNIP>1d094b83' due to 'connect ECONNREFUSED /var/run/balena-engine.sock'

Could this be pointing to a corrupt SD card?

codewithcheese · November 12, 2020, 7:35am

Hey Simon, try running the diagnostics, navigate to https://dashboard.balena-cloud.com/devices/{{UUID}}/diagnostics and run the health checks and diagnostics. Feel free to upload the results here for us to take a look.

sighmon · November 12, 2020, 7:47am

Found the problem for a large portion of our devices! SD cards used to always have these config variables on by default:

Define DT parameters to "i2c_arm=on","spi=on","audio=on"
Define DT overlays to "vc4-fkms-v3d"

But for some reason the latest batch of 60 SD cards we burned didn’t have them set. I’m wondering if the person who burned them did it via your API and that didn’t set those variables.

Strange thing is that some of the devices still detected ALSA okay even without them set!?!

As for the failing to download image device, here’s the diagnostics output:

Some container_engine issues detected: 
test_container_engine_restarts Container engine balena has 1 restarts and may be crashlooping (most recent start time: Thu 2020-11-12 07:43:22 UTC)

Some localdisk issues detected: 
test_write_latency Slow disk writes detected: mmcblk0: 2563.35ms / write, sample size 5456
mmcblk0p1: 1647.4ms / write, sample size 5
mmcblk0p5: 1974.9ms / write, sample size 42
mmcblk0p6: 2590.75ms / write, sample size 5363

sradevski · November 12, 2020, 8:11am

Hey Simon, great that you found what the issue was on most of the devices. As for the one failing, that can point to either a bad connection (and get slow writes as a side-effect), or a bad SD card, if everything else is the same between devices and if you have access to the device, I’d try the same SD card on a different device first just to rule out any hardware/connectivity issues and if that doesn’t work try replacing the SD Card.