balenaSound - all containers exited

Product support balenaOS
1

I have done everything by the instructions online but when I switch to “audio off” and enter “hifiberry-dac” all the services (bluetooth-audio, airplay and spotify) are being killed and state exited and I have no idea why they won’t restart.

Blockquote
26.01.20 20:04:23 (+0100) bluetooth-audio Bluetooth agent registered
26.01.20 20:05:47 (+0100) Applying boot config: {“dtoverlay”:[“hifiberry-dac”],“disable_splash”:“1”,“dtparam”:[“i2c_arm=on”,“spi=on”,“audio=on”],“enable_uart”:“1”,“gpu_mem”:“16”}
26.01.20 20:05:47 (+0100) Applied boot config: {“dtoverlay”:[“hifiberry-dac”],“disable_splash”:“1”,“dtparam”:[“i2c_arm=on”,“spi=on”,“audio=on”],“enable_uart”:“1”,“gpu_mem”:“16”}
26.01.20 20:05:49 (+0100) Killing service ‘airplay sha256:5b330634160752add73ea4ea2d4aec99f51c88af3c9b12e7975c5e92f1ab3fc4’
26.01.20 20:06:12 (+0100) Killed service ‘airplay sha256:5b330634160752add73ea4ea2d4aec99f51c88af3c9b12e7975c5e92f1ab3fc4’
26.01.20 20:06:13 (+0100) Service exited ‘airplay sha256:5b330634160752add73ea4ea2d4aec99f51c88af3c9b12e7975c5e92f1ab3fc4’
26.01.20 20:06:14 (+0100) Killing service ‘bluetooth-audio sha256:bc5c5d0b8b2ed0628d947117f405816125447cbf3f5376de3fadff28178247c1’
26.01.20 20:06:29 (+0100) Killed service ‘bluetooth-audio sha256:bc5c5d0b8b2ed0628d947117f405816125447cbf3f5376de3fadff28178247c1’
26.01.20 20:06:29 (+0100) Service exited ‘bluetooth-audio sha256:bc5c5d0b8b2ed0628d947117f405816125447cbf3f5376de3fadff28178247c1’
26.01.20 20:06:29 (+0100) Killing service ‘spotify sha256:738252c4a1f3763651f5eee4fc77693a849f284c89138974282eb0ce48f3321a’
26.01.20 20:06:51 (+0100) Killed service ‘spotify sha256:738252c4a1f3763651f5eee4fc77693a849f284c89138974282eb0ce48f3321a’
26.01.20 20:06:51 (+0100) Rebooting
26.01.20 20:06:51 (+0100) Service exited ‘spotify sha256:738252c4a1f3763651f5eee4fc77693a849f284c89138974282eb0ce48f3321a’

I’d be willing to send in error logs if you let me know where I can get them from and where to send them. In addition, another issue: the logs in the little frame in the dashboard… When I clear the logs they start to reappear again and are not being refreshed. Might be related to this issue: Logs not updating in dashboard (restart needed)

When running the command for logs I get:

Blockquote
root@8dd1a74:~# journalctl -f -a -u resin-supervisor
– Logs begin at Mon 2019-12-23 09:14:37 UTC. –
Jan 26 20:49:12 8dd1a74 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 26 20:49:12 8dd1a74 resin-supervisor[8316]: deactivating
Jan 26 20:49:12 8dd1a74 systemd[1]: resin-supervisor.service: Failed with result ‘timeout’.
Jan 26 20:49:12 8dd1a74 systemd[1]: Failed to start Balena supervisor.
Jan 26 20:51:45 8dd1a74 systemd[1]: resin-supervisor.service: Start-pre operation timed out. Terminating.
Jan 26 20:51:45 8dd1a74 systemd[1]: resin-supervisor.service: Control process exited, code=killed, status=15/TERM
Jan 26 20:51:45 8dd1a74 resin-supervisor[8987]: deactivating
Jan 26 20:51:45 8dd1a74 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 26 20:51:45 8dd1a74 systemd[1]: resin-supervisor.service: Failed with result ‘timeout’.
Jan 26 20:51:45 8dd1a74 systemd[1]: Failed to start Balena supervisor.
Jan 26 20:54:20 8dd1a74 systemd[1]: resin-supervisor.service: Start-pre operation timed out. Terminating.
Jan 26 20:54:20 8dd1a74 systemd[1]: resin-supervisor.service: Control process exited, code=killed, status=15/TERM
Jan 26 20:54:20 8dd1a74 resin-supervisor[9511]: deactivating
Jan 26 20:54:20 8dd1a74 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jan 26 20:54:20 8dd1a74 systemd[1]: resin-supervisor.service: Failed with result ‘timeout’.
Jan 26 20:54:20 8dd1a74 systemd[1]: Failed to start Balena supervisor.

And when I try to shutdown, I get the following error:
Request error: tunneling socket could not be established, cause=socket hang up

3

Hi

thanks for reaching out. I have some questions to better understand the source of the problem:

  • have you already tried power-cycling the device ?
  • have you also tried by restarting the engine, running systemctl restart balena-engine on the device host os?
  • was it working ok without the external DAC ? maybe it’s a power supply issue

Looking forward for your answers

regards

federico

6

Hi Federico

When I tried systemctl restart balena-engine via the cloud terminal, it told me

Blockquote
root@8dd1a74:~# systemctl restart balena-engine
Warning: The unit file, source configuration file or drop-ins of balena-engine.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.

but then it froze.

EDIT: it didn’t freeze but returned this after a while

Blockquote
root@8dd1a74:~# systemctl restart balena-engine
Warning: The unit file, source configuration file or drop-ins of balena-engine.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.

I rebooted via dashboard and now it freezes after entering the two commands.

I have a 2GB MicroSD inserted. Isn’t that enough? Don’t get any errors beforehand. What kind of Ampere do you recommend for the Pi with the DAC?

And what about the logs that do not clear?

Power-cycling didn’t help much. I switched the SD from one device to another without the DAC but it still has the same issues it seems. Maybe the dashboard doesn’t update properly?

Cheers and thanks!

7

2GB SD card sounds quite small to me, and I am not sure if that is enough in this case. If you wish, you can enable support access and share the UUID of the device with us, and we can have a look. Alternatively, you can run diagnostics by yourself (Diagnostics menu on the device page) and see if storage is an issue.

9

Here is the UUID:
8dd1a749be494208ee0ffc856a0928c0

The diagnostics fails with this titem

Container engine balena is up, but has 20 unclean restarts and may be crashlooping (most recent start time: Tue 2020-01-28 09:30:25 UTC)

Thanks!

10

I can’t see the device, can you make sure support access is enabled? Here is how you enable it: https://global.discourse-cdn.com/balena/original/2X/7/7ec45f755b2c54ac303af7714045cb56cad06d52.png or you can do it from the “Actions” page

11

3h support granted :slight_smile:

12

I am looking at it now and it seems everything is working, am I missing something? I see there were multiple restarts, and I believe that might have been due to having a dangling whitespace in the BLUETOOTH_DEVICE_NAME envvar’s value (I have seen that behavior before, but I am not sure if that was the case). Space-wise looks fine otherwise.

13

I had the wrong Pi hooked up, the one without the pHAT DAC. Now that I do, there is an additional fail for the supervisor. Please check again.

EDIT: at the same time all three services seem to be running?!

EDIT: Trying now with another setup. Let you know when it is online again.

19

Hey, we’ve been investigating your device, and so far we can’t really say for certain what is going on. We know that the engine is struggling to communicate with the containers, and therefore is not able to complete it’s startup. If it’s ok with you we’d like to remove the containers from the filesystem, and then if that doesn’t work continue to remove the entire engine data and have it be redownloaded - would this be ok?

20

OK, I tried it with a 32GB SD but the same thing happens, it comes back online but all services stay on “exited”. I will now plug back in the 2GB SD. You can do your magic (thanks for the awesome help!) in 10 minutes when it is up and running again.

21

Back online with the UUID you know. Let me know how it goes and if I can do anything else.

And one question: is communicating my UUID like this safe or am I now open for attacks? :slight_smile:

22

Thanks, I’ll take another look. Communicating your UUID like this is completely safe, don’t worry :slight_smile:

23

After removing the containers which the engine was having trouble with, the device looks to be working well now. Please let us know if this changes!

25

OK, couple of things:

  1. thanks for your help!
  2. the Bluetooth connects well, see screenshot but then after a while it goes back to a discovered device that is not connected. However, the music keeps playing over bluetooth until I stop bluetooth on the cellphone. Should I leave it on playing and you check if it is something on the end of the device?
  3. I changed the BT name of the device so all services were stopped, it shortly says “installing” which it shouldn’t since they are not installing :wink:

Cheers
TW

28

Hi. Looks like something weird is going on with balenaEngine. Do you have another device you can test balenaSound with? Also, support access has expired so be sure to reenable that if you want us to take another look.

30

Just reenabled it for a day. Thanks!

And can I somehow add another WiFi credentials set to my existing device? Couldn’t find anything in the forums.

31

Hi, all the services seem to be running as normal right now? As for adding WiFi credentials the simplest way is to mount the SD card on your development machine. There you will see a system-connections folder and you can make a copy of the existing file and edit it. There is more on this here https://www.balena.io/docs/reference/OS/network/2.x/#wifi-setup.

32

Not really, services are ok but I get this:

check_container_engine FailedContainer engine balena is up, but has 3 unclean restarts and may be crashlooping (most recent start time: Tue 2020-01-28 21:42:46 UTC)

AND for spotify

28.01.20 23:29:50 (+0100) spotify WARN:: couldn’t parse packet from V6([fe80::25fe:f1b0:6735:c329]:5353): type 47 is invalid

35

It appears that the containerd service, which is responsible for actually tending to the lifecycle of the containers on device crashed at least once, from what I can tell. But the device recovered from it by itself. I’m not sure if the spotify error is related - I wouldn’t think so

If you experience any more issues like this please let us know here.