Random shutdowns, 'partial' off-system dvice, stutters, lags, and more

Hello!
First, many thanks for the developers, BalenaSound is a fantastic idea!
In my case, however, I experience some peculiar issues that I did not find solution for (yet). I have a 3-device ecosystem, for the sole purpose of running BalenaSound. They are all 4Gb Raspberry Pi 4Bs, and they have 32Gb Kingston SD cards and IQAudio DAC Pro hats, with original Raspberry AC adapters. Flashing was done under Ubuntu, for under Windows 10 it was impossible to do. As a sidenote, it would be useful to put that in the ‘getting started’ tips, for I was quite desperate when I bought all these just to find out that I cannot flash the cards even with Balena Etcher under Win 10. I found a workaround, so it is all right now.

I use Spotify and Airplay mainly, bluetooth is not my go-to for sound quality reasons. My devices are connected to the network by WiFi, and I would not connect them by LAN (different rooms, no intention to ‘wire about’ the entire appartment), so please no such tips.

I try to summarise the remaining issues that make me perplexed:

  1. One of my devices tends to shutdown randomly for no obvious reason, while the others are working well. It can happen anytime within a timeframe of 2 days or 10 minutes after restarting the device.
  2. The same device tends to be partially “off” the ecosystem: I mean when I send music to it either by Spotify connect or through Airplay, it works, but only that one device. The other two remain silent, as if the signal was not ‘shared’ with them. Why is this partial? Because if I send music to another device, the ‘flawed one’ gets it just all right. This issue also happens randomly - sometimes no such problem, sometimes it is there
  3. Even if the same device works all right, sometimes there are stutters, short periods of silence in the music played by the other two devices, while the flawed one plays perfectly. It seems also to be random.
  4. General problem: after a few tracks, there is a short ‘stutter’ at the beginning of each track in continuous play in Spotify Connect. It is not very good, especially when listening Pink Floyd or similar albums that are one contionuous pieace of music. But it is also annoying by ordinary tracks.
  5. Lag varies greatly. Sometimes only a few seconds, sometimes (apparently after a longer period of running, say, a day) 10-15, or even more, independent of the mode of cast (Spoti or Ap). I never got as low as 1-1,5 second that, apparently, was discussed in by others being so bad. I would be happy with such a low input lag… :slight_smile: I understand it is due to some built-in characteristics of the system, and I accept that lag cannot be eradicated completely, but I would be happier with a lower duration, and it also gives me the feeling of an unstable system. As it is, apparently, based on the above points.

I could narrow it down more or less to one SD card, but a few times, it happened with others as well. Since the last update, however, it seems to be just the one.
I have reflashed the image to the same SD card multiple times. For a while (2 days tops) it works well after a reflash, but never perfectly. I switched cards between Pis, and found that it is probably related to the SD-card or the OS. But what made me perplexed that on time I experienced the same issue with another card that worked well in another device, put in the ‘flawed’ one. So I am out of options, and don’t really know what is the problem here. My guesses:

  • SD card problem: but then why did it happen in the same device with another card that worked well in another device? I would not want to throw out money for buying a new card if it is not sure that that’s what causes the issues.
  • Hardware problem (faulty Pi): but then why does it happen with another device that works well with another card, if I put in it the ‘flawed’ card? I do not want to return the Pi until I am not completely certain it is the reason for the issues, for I should pay double shipping costs then that are not something I would want to do for no reason.
  • OS problem: but then why does it work (usually, but not always) with other card/device pairs?

I am no IT or programming expert, so all I could do to narrow down the ‘source’ of the issues is moving the cards around, once I even switched the place of two devices.

Excuse me for the long elaboration, and thank you for your kind help.

Hey @MFerenc

welcome to the forums!

and

many thanks for the developers, BalenaSound is a fantastic idea!

thanks, glad you like balenaSound :tada:

First I would like to address

Flashing was done under Ubuntu, for under Windows 10 it was impossible to do. As a sidenote, it would be useful to put that in the ‘getting started’ tips, for I was quite desperate when I bought all these just to find out that I cannot flash the cards even with Balena Etcher under Win 10. I found a workaround, so it is all right now.

Can you please elaborate on the issues you encountered? I use balenaEtcher on windows 10 and do not have any issues flashing sd cards. If this persists i would recommend starting a new forum thread. Thanks

Back to balenaSound:

  1. It does seem like some erratic behaviour. You are using a Pi with decent memory resources. I would still say we rule out the sd card issue. Can you run the diagnostic tests for the device ? You can access this feature by navigating to a device summary page, and scrolling to the bottom to select “Diagnostics (Experimental)”.

  2. The same device tends to be partially “off” the ecosystem: I mean when I send music to it either by Spotify connect or through Airplay, it works, but only that one device. The other two remain silent, as if the signal was not ‘shared’ with

At the risk of stating the obvious :sweat_smile: Can you please confirm what modes of operation you have configured for each device? You can find more details here. Ie have you configured the SOUND_MODE environment variable on any of the devices or are you using the default settings?

  1. Do you see any relevant logs when the issue happens? Perhaps you can enable persistent logs on the “flawed” device? This should be possible by going to the Device Configuration page for the particular device.

  2. Do you have any other applications running on the device besides balenaSound? Also how did you deploy the application? Are you using the latest balenaSound version? What balenaOS version are you using?

Thanks

Hi Rahul,

Thank you for your detailed reply, I try to elaborate on your questions.

  • Win 10 balena Etcher issue: to be honest, I do not really know teh specifics, only that it failed in validation phase. I’ve read some forum entries here and there, and others seem to have had the same problem. Some wrote it was probably caused by the automount function of Win 10, and maybe there was something about something that Win 10 writes on the card without explicit permission of the user that causes the issue. Under Ubuntu, however, everything was flawless.

balenaSound issues:

  1. I’ve run the diagnostics, only check_service_restarts shows ‘failed’ with the following details: ‘Some services are restarting unexpectedly: (service: /spotify_3303569_1703709 restart count: 1) (service: /multiroom-server_3303565_1703709 restart count: 2) (service: /upnp_3303570_1703709 restart count: 84)’

  2. I have no SOUND_MODE environment variable defined, so I am guessing that it should be on default mode, which is multi-room, I think. Should I configure SOUND_MODE?

  3. I have just enabled persistent logging for the entire fleet, so I will be able to follow up later to this with some logs, but unfortunately now I cannot see any logs. As I have no experience in this, I am not sure what would constitute relevant logs to the issue, so it would be great if you could tell me what I should look for.

  • I have no other applications yet, I wanted first to make this one work. The main reason for the whole fleet is to make balenaSound work with my ‘old but gold’ hifis. :slight_smile: When balenaSound is stable, I might proceed to some other useful applications, but not yet.
  • I am not sure what you mean by how I deployed the application. After I created the application, I downloaded the OS image by using the ‘Add device’ function, and flashed the same image to all of my three Pis.
  • All of my devices have the latest balenaOS (12.3.1+rev1), and I have just updated the supervisor version from 12.4.6 to 12.5.6. I don’t know how I can check if I am using the latest balenaSound version. I see a ‘Current release’ column, in which all values are ‘5e8ccce’. Is this the balenaSound version?

Just one further thing, for I made one change since last time. I’ve read somewhere that the router I got from the internet service provider might not be the most stable one for wireless connections, and I began to suspect that it might well have caused my issues. Therefore I purchased a new router (TP-LINK Archer AX-20), and installed it yesterday. Hopefully that would solve my issues, for other than that, I am out of options.
Strange thing is, that 2 of my Raspberrys connect to it through 5 GHz frequency, but one of them does through 2.4 (not sure which one, for I am not at home at the moment). I hope it is not another problem.

Thank you again for your kind help, and I am extremely sorry that I cannot express all these in a more succinct way.

Best,
Ferenc

Hi Ferenc, balenaSoud developer here! Thanks for your kind words :slight_smile:

Since you are running on Pi 4’s you shouldn’t see this kind of problems, so let’s see if I can help you out. As far as I can tell from your posts you’ve got 3 different problems:

Lag
I feel you on this! Unfortunately as you say it’s due to the technologies being used, biggest culprit being the multi-room feature (not your case, but disabling multi room greatly improves lag). It is something we’d also love to improve, I’ve thought of contributing to the third-party library we use for multi room but it’s a really hard problem to be honest and it’s a bit out of scope as we feel the tradeoff is acceptable at the least. One thing to note is that the more devices you add to multi-room the more the lag can be, this is because the system will adapt to the slowest device. So if you got one bad device it will definitely slow down the entire fleet, which leads to my next point…

Device going in and out of the "ecosystem"
The way multi-room feature works is it has devices broadcast a package to the entire network so other devices can detect it. So it will only work if all your devices are part of the same network. Devices on different networks will be completely isolated. Since you mentioned you have 2 WiFi networks running 2.4 and 5 GHz I suspect this is what is going on. Why does it work one way though? That I can not tell to be honest. My best guess is that your router might be performing some type of bridging between the two WiFi networks, but that’s probably not very reliable. My suggestion is to ensure all your devices are running on the exact same network and that they have good WiFi signal. Tying it back to the lag problem, if a device has bad connectivity where it’s loosing a bunch of packets every now and then that can lead to increased lag.

Stuttery audio
Regarding stuttery audio there are two possible causes, so we’ll need to narrow it down.
How often do you notice it stuttering? The best way to figure this out is to check the logs for the audio service on the device page on the dashboard. What we are looking for is for a line similar to this one W: [pulseaudio] module-loopback.c: Too many underruns, increasing latency to xxx ms. Do you see it being logged right about when the stutter occurs?
The other type of stuttering is mostly prevalent on Pi3 devices, not Pi4, so it’s unlikely to be the case. We have a detailed writeup about it in our docs troubleshooting section here: https://sound.balenalabs.io/docs/support#troubleshooting. Look for the item: Audio cuts or is very stuttery.

I hope this is useful. Happy to help you further debug these problems if needed, we can grab logs and see what’s going on but I wanted first to provide an overview of these items. As a starting point, please ensure you get all devices on the same WiFi network .
Cheers!

Just to throw this into the mix, I have a device that keeps dropping off the fleet during playback, with a long stutter then dead. It appears to be related to the WiFi signal not being solid. But I haven’t diagnosed it much yet. I’ll post back here if I find anything solid.

I have seen in the past that the WiFi signal can cause stutter and drop-offs, even when it appears to be a good signal (noise, maybe?) The RPi has to work a little harder to manage the data when the WiFi is rough, and that can cause data timing issues. Try relocating devices even a foot or two from where they are to see if it helps. I have one RPi ZeroW that works great on my desk, but fails completely if it’s not on my monitor stand, which is only 6 inches above where it does not work.

And as I’m typing, I’m thinking of other things I have discovered with WiFi. I found that the multi-room server device MUST be the most solid on the WiFi, compared to the multi-room clients. Having a server that has too much jitter in its network data causes the clients to lose sync frequently, which sounds terrible when multiple devices provide sound to the same air space. My server device is near my WiFi router, so it is on the wired network, while all of the client devices are on the WiFi. Make sure you are using a device as the server which is closest to your router and has the best WiFi signal possible.

Also I have noticed a device will not rejoin the fleet if playback is still going. I have to stop playback, change the input to another protocol (usually BT) and play something for a moment and stop it, then resume playback on the format I was originally using (AirPlay usually). That triggers the sound-supervisor to re-announce the IP address of the multi-room server to the fleet, and everything syncs up again.

Thanks @koyaanisqatsi for your input. Those are very good points to consider and perhaps worth adding to the docs. I want to run some tests too with bad WiFi but it’s hard for me to test it as my apartment is not so big so WiFi signal is pretty good. Maybe I need to wrap my router with aluminium foil to introduce some noise haha.

1 Like

Hi,
Thank you for your reply!
Since last time, I have installed a new wifi router, for I could not think any other source of the problem than my original router. I waited a few days, and it seems to have solved most of my problems! I do not have two wifi ecosystem only one, but this new one does seem to do the work well, even though it has both 2.4 and 5 GHz at the same time.

  • So no device has dropped off the ecosystem for 6 days now, which has never happened before. It is safe to say that in this regards, stability is reached.
  • As for the stutters, I only experienced the “beginning of track” stutter, but only a few times. It seems to be much better since the new router has been installed.
  • The lag is still a problem, but this is something I can live with. In 6 days without reboot of the devices, it increased to 9 seconds from the initial 1-2, but I guess it is ok. I guess I need to shut the system down sometimes or reboot it to return lag to normal once it is too big. I will experiment with this in the coming days.

I have not yet managed to catch do stuttering in the logs, for since the new router has been installed, they became quite rare.I could not find what you wrote, but I will look for it. I hope I will catch something and will be able to report you here soon.

You are all being very helpful, thank you, and please excuse me for writing this much about my problems when most of them were only produced by a substandard router…

And again, a huge thanks to you developers for this fantastic application, it is something I have been dreaming of for years…
My next goal with the balenaSound system is to be able to manage sending the signals from a line out source to all of my devices so I can even play a vynil in all rooms while it is being played in one. But that will be another story, hopefully as successful as this one! :slight_smile:

Best,
Ferenc

1 Like

Thank you, these are all very helpful! I have also suspected that location has to do something with my problems, but a new router seems to have solved most (it is not a big apartment, but in some places, noise may have been too much for my old router).
Thank you very much for your suggestion on using the device closest to the router as the multi-room server! Until now, when I used only one amplifier, I always chose the device plugged in the amplifier I was using. When I played music in more than one room, I just chose one randomly. From now on, I will heed your advice! I will also try connecting the server device through LAN cable, which I did not do yet, even though it is very close to the router. I was afraid it could cause even more problems if one of my devices is on the wired network, while others are on wireless. I can’t wait to see if these work for me as well as they work for you!

Thank you for all your advices!
Best,
Ferenc

1 Like

Just an update on the ‘beginning of track stuttering’.
Now it seems that it happens almost always I skip a track (the beginning of the next track ‘stutters’, i.e. within the first second of the next track), but rarely when I reach the end of a track and moving on to the next track happens automatically. I use Spotify Connect and not Airplay, so it might be related to that. I checked logs, nothing is shown there.
I will test it with Airplay as well.

I actually just noticed my fleet stutters at the beginning of songs sometimes, and it looks like it may be similar circumstances to you are describing. We usually stack up a queue of several hours of music, and don’t notice any stuttering. But when we skip songs, pause and restart, or change music sources, we hear it on some of the devices, but it differs per device.

@tmigone , I have suspected this is caused by a setting in the snapcast system for a while now, but haven’t been able to reproduce or accurately quantify it, and thus never mentioned it. IIRC (I looked this up months ago), there is a setting in snapcast that either ends and restarts the data stream during periods of silence, or leaves it active all the time. BalenaSound is currently set to stop and start the stream upon silence. I have noticed on songs that have sound, then silence, then sound again, it can cause the song to have a glitch immediately after the silent spot. (I thought it happened on Alanis Morissette’s album Jagged Little Pill, the song All I really Want, but I couldn’t reproduce it today) I wonder if changing that setting to always stream, even when silent, will help issues with playback stuttering at the beginning.

@koyaanisqatsi, thank you for noticing these, hopefully it will help the developers.

Just one update: since last time, I tried several times through Airplay as well, and the same beginning-of-track-stuttering happens just as with Spotify Connect.

@ferenc, glad to know you are liking balenaSound <3

My next goal with the balenaSound system is to be able to manage sending the signals from a line out source to all of my devices so I can even play a vynil in all rooms while it is being played in one. But that will be another story, hopefully as successful as this one! :slight_smile:

I’ve got this exact setup, you’ll need a sound card that takes the input you want (I use this one as my vynil player has RCA output) and you’ll need to add an environment variable SOUND_ENABLE_SOUNDCARD_INPUT with whatever value you want (I use SOUND_ENABLE_SOUNDCARD_INPUT=1). What this will do is turn the sound card input as a new plugin/audio source as if it was Spotify or airplay.

Regarding the stuttering, that’s a very interesting find @koyaanisqatsi. Do you remember the name of the option? If snapcast is causing this then the problem should not be present in standalone mode. I’ll test this and report back.