Chromium browser shows "Aw, Snap!" error at random times

esbenvb · March 8, 2021, 11:20am

I have a web application running in a browser, on a balena-based RPI3.

From time to time, the screen shows the “Aw, Snap!” error from Chromium, which means that the browser tab process has crashed.

I have tried to look in the balena dashboard logs, but nothing about the crash is shown. I have also extracted the Xorg.0.log file from the container itself, but it also has no clues for me.

I suspect that the browser tab is either killed because of memory throttling, or because of some issue on the web-app I run. But so far I have traces to follow, to see if I’m right.

I have not been able to reproduce the issue locally, only in the production which is on a remote site.

Details:

Hardware: RevPi 3+
OS: balenaOS 2.53.9+rev1
Browser image: Docker Hub (chromium-browser --version says Chromium 78.0.3904.108)
Web app: An online hosted React based app, communicating via MQTT with other systems.

The balena image on the machine is running 4 services: Browser, MQTT server (mosqitto), a node.js app to turn off display backlight when there’s no activity on MQTT, and a service that receives a webhook from an external device and forwards it to MQTT.

I’m also concerned whether I’m using an obsolete/outdated/broken browser image. I believe that I chose the balenaplayground/balenalabs-browser image because it worked for me and the other images didn’t, but I’m not completely sure. But it bothers me that I cannot find the related github project and I cannot see how the image is composed. Neither balena playground · GitHub or balenaLabs · GitHub seems to have the balenalabs-browser repo.

So maybe i’m better of with trying another browser image? Like the balenablocks/browser which is also used by balena-dash?

Thanks in advance for any responses

georgiats · March 8, 2021, 1:25pm

Hi @esbenvb, I believe that you’ll be interested in balenaHub, our collection of edge and IoT projects created by fleet owners and product builders. In our block collection you’ll find the browser block, which is available on GH. This is the main idea, to use it as a block of your application.
I am not sure what’s the issue with your application, but you could share the health checks and the device diagnostics in case it insists so that we can take a look and continue the troubleshooting.

rahul-thakoor · March 9, 2021, 11:37am

Hey there,

To answer your questions:

you should use the image: balenablocks/browser
you can find the source for the repo here GitHub - balenablocks/browser: A drop-in web browser block
As for the chromium instance crashing, can you please confirm how much gpu_memory you have allocated?
have you enabled gpu acceleration by setting ENABLE_GPU=1? considering your use case with the React app i believe this is not necessary

Thanks

esbenvb · April 7, 2021, 7:00am

Thanks for your responses. I just changed the package to balenablocks/browser, as you both suggested. And updated the OS to balenaOS 2.69.1+rev1 However, the problem persists.

Here’s the health

And the ENV vars for the browser:

FLAGS --window-position=0,0
KIOSK 1
LAUNCH_URL http://....
RESIN_HOST_CONFIG_gpu_mem 128

What can I do to debug this? Can I use the chrome external debugger port or something?

jakogut · April 8, 2021, 1:37am

There’s a couple things I’d check first. First of all, I’ve experienced this problem in the past with a browser process killed after an out of memory (OOM) condition. You can check dmesg for something like Out of memory: Killed process to see if this is an issue. On a device with only 1 GB of memory, this is quite likely.

If you don’t see any signs of an OOM condition, I’d suspect a GPU crash. Check about:gpu on your device to see if GPU acceleration is enabled. If it is, disabling GPU acceleration might improve stability at the cost of performance, which may not matter in your situation. You can disable GPU acceleration with the launch flag --disable-gpu.

jakogut · April 8, 2021, 1:56am

You should also be able to enable logging with --enable-logging=stderr --v=1 that might give you a better idea of the cause of the problem.

esbenvb · April 14, 2021, 9:58am

Thanks for your suggestions.

Unfortunately, I don’t see any debug info in the log, even though I added the flags that you suggested. Here’s the full list of args:

--window-position=0,0 --enable-logging=stderr --v=1 --disable-gpu

I have also tried a dmesg, and found something that might be a clue.

At 516289 I get the error
Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x0277b6b6
which seems to match the time where the users reported the issue of the “Aw, snap” error page.

At 524136 I restarted the browser package, and at 613656 I disabled the GPU using the flag you just suggested.

So, can the error that I found, be helpful to us in finding the cause?

[516289.419811] Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x0277b6b6
[516289.419965] audit: type=1701 audit(1618295219.706:114): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=5297 comm="Compositor" exe="/usr/lib/chromium-browser/chromium-browser-v7" sig=5 res=1
[524136.936577] systemd-udevd[30]: Starting version 241
[524140.046408] audit: type=1100 audit(1618303070.267:115): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[524140.048864] audit: type=1101 audit(1618303070.267:116): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_permit acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[524140.052664] audit: type=1103 audit(1618303070.277:117): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[524140.062278] audit: type=1105 audit(1618303070.287:118): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_env,pam_env,pam_mail,pam_limits,pam_permit,pam_unix,pam_systemd acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613656.919757] systemd-udevd[30]: Starting version 241
[613660.087547] audit: type=1100 audit(1618392589.577:119): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613660.089644] audit: type=1101 audit(1618392589.577:120): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_permit acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613660.091443] audit: type=1103 audit(1618392589.577:121): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613660.098568] audit: type=1105 audit(1618392589.587:122): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_env,pam_env,pam_mail,pam_limits,pam_permit,pam_unix,pam_systemd acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'

jakogut · April 15, 2021, 12:16am

This looks to be a bug in either Chromium, or the graphics driver. Has disabling the GPU helped at all? Is there a newer version of Chromium available to try?