Chromium browser shows "Aw, Snap!" error at random times

esbenvb · March 8, 2021, 11:20am

I have a web application running in a browser, on a balena-based RPI3.

From time to time, the screen shows the “Aw, Snap!” error from Chromium, which means that the browser tab process has crashed.

I have tried to look in the balena dashboard logs, but nothing about the crash is shown. I have also extracted the Xorg.0.log file from the container itself, but it also has no clues for me.

I suspect that the browser tab is either killed because of memory throttling, or because of some issue on the web-app I run. But so far I have traces to follow, to see if I’m right.

I have not been able to reproduce the issue locally, only in the production which is on a remote site.

Details:

Hardware: RevPi 3+
OS: balenaOS 2.53.9+rev1
Browser image: Docker Hub (chromium-browser --version says Chromium 78.0.3904.108)
Web app: An online hosted React based app, communicating via MQTT with other systems.

The balena image on the machine is running 4 services: Browser, MQTT server (mosqitto), a node.js app to turn off display backlight when there’s no activity on MQTT, and a service that receives a webhook from an external device and forwards it to MQTT.

I’m also concerned whether I’m using an obsolete/outdated/broken browser image. I believe that I chose the balenaplayground/balenalabs-browser image because it worked for me and the other images didn’t, but I’m not completely sure. But it bothers me that I cannot find the related github project and I cannot see how the image is composed. Neither balena playground · GitHub or balenaLabs · GitHub seems to have the balenalabs-browser repo.

So maybe i’m better of with trying another browser image? Like the balenablocks/browser which is also used by balena-dash?

Thanks in advance for any responses

georgiats · March 8, 2021, 1:19pm

Hi @esbenvb, I believe that you’ll be interested in balenaHub, our collection of edge and IoT projects created by fleet owners and product builders. In our block collection you’ll find the browser block, which is available on GH. This is the main idea, to use it as a block of your application.
I am not sure what’s the issue with your application, but you could share the health checks and the device diagnostics in case it insists so that we can take a look and continue the troubleshooting.

rahul-thakoor · March 9, 2021, 11:37am

Hey there,

To answer your questions:

you should use the image: balenablocks/browser
you can find the source for the repo here GitHub - balenablocks/browser: A drop-in web browser block
As for the chromium instance crashing, can you please confirm how much gpu_memory you have allocated?
have you enabled gpu acceleration by setting ENABLE_GPU=1? considering your use case with the React app i believe this is not necessary

Thanks

esbenvb · April 7, 2021, 7:00am

Thanks for your responses. I just changed the package to balenablocks/browser, as you both suggested. And updated the OS to balenaOS 2.69.1+rev1 However, the problem persists.

Here’s the health

And the ENV vars for the browser:

FLAGS --window-position=0,0
KIOSK 1
LAUNCH_URL http://....
RESIN_HOST_CONFIG_gpu_mem 128

What can I do to debug this? Can I use the chrome external debugger port or something?

jakogut · April 8, 2021, 1:37am

There’s a couple things I’d check first. First of all, I’ve experienced this problem in the past with a browser process killed after an out of memory (OOM) condition. You can check dmesg for something like Out of memory: Killed process to see if this is an issue. On a device with only 1 GB of memory, this is quite likely.

If you don’t see any signs of an OOM condition, I’d suspect a GPU crash. Check about:gpu on your device to see if GPU acceleration is enabled. If it is, disabling GPU acceleration might improve stability at the cost of performance, which may not matter in your situation. You can disable GPU acceleration with the launch flag --disable-gpu.

jakogut · April 8, 2021, 1:56am

You should also be able to enable logging with --enable-logging=stderr --v=1 that might give you a better idea of the cause of the problem.

esbenvb · April 14, 2021, 9:58am

Thanks for your suggestions.

Unfortunately, I don’t see any debug info in the log, even though I added the flags that you suggested. Here’s the full list of args:

--window-position=0,0 --enable-logging=stderr --v=1 --disable-gpu

I have also tried a dmesg, and found something that might be a clue.

At 516289 I get the error
Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x0277b6b6
which seems to match the time where the users reported the issue of the “Aw, snap” error page.

At 524136 I restarted the browser package, and at 613656 I disabled the GPU using the flag you just suggested.

So, can the error that I found, be helpful to us in finding the cause?

[516289.419811] Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x0277b6b6
[516289.419965] audit: type=1701 audit(1618295219.706:114): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=5297 comm="Compositor" exe="/usr/lib/chromium-browser/chromium-browser-v7" sig=5 res=1
[524136.936577] systemd-udevd[30]: Starting version 241
[524140.046408] audit: type=1100 audit(1618303070.267:115): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[524140.048864] audit: type=1101 audit(1618303070.267:116): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_permit acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[524140.052664] audit: type=1103 audit(1618303070.277:117): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[524140.062278] audit: type=1105 audit(1618303070.287:118): pid=19183 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_env,pam_env,pam_mail,pam_limits,pam_permit,pam_unix,pam_systemd acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613656.919757] systemd-udevd[30]: Starting version 241
[613660.087547] audit: type=1100 audit(1618392589.577:119): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613660.089644] audit: type=1101 audit(1618392589.577:120): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_permit acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613660.091443] audit: type=1103 audit(1618392589.577:121): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[613660.098568] audit: type=1105 audit(1618392589.587:122): pid=17103 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_env,pam_env,pam_mail,pam_limits,pam_permit,pam_unix,pam_systemd acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'

jakogut · April 15, 2021, 12:16am

This looks to be a bug in either Chromium, or the graphics driver. Has disabling the GPU helped at all? Is there a newer version of Chromium available to try?

esbenvb · May 7, 2021, 7:00am

The GPU is disabled, and I recently updated the docker image so it’s using Chromium 88. The problem still occurrs.

Here’s my dmesg log.

[1822902.987505] Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x027476b6
[1822902.987637] audit: type=1701 audit(1619601822.558:128): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=11531 comm="CompositorTileW" exe="/usr/lib/chromium-browser/chromium-browser-v7" sig=5 res=1
[1842953.698259] br-8e1500f7888c: port 1(veth51d4256) entered disabled state
[1842953.698686] veth69128b8: renamed from eth0
[1842953.895513] audit: type=1325 audit(1619621873.288:129): table=nat family=2 entries=21
[1842953.953360] br-8e1500f7888c: port 1(veth51d4256) entered disabled state
[1842953.984087] device veth51d4256 left promiscuous mode
[1842953.984133] br-8e1500f7888c: port 1(veth51d4256) entered disabled state
[1842953.984179] audit: type=1700 audit(1619621873.338:130): dev=veth51d4256 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[1842954.084495] audit: type=1325 audit(1619621873.468:131): table=filter family=2 entries=56
[1842954.248120] audit: type=1325 audit(1619621873.638:132): table=nat family=2 entries=20
[1842954.288544] br-8e1500f7888c: port 3(veth33ec060) entered disabled state
[1842954.288836] vethbc7b395: renamed from eth0
[1842954.484682] br-8e1500f7888c: port 3(veth33ec060) entered disabled state
[1842954.504122] device veth33ec060 left promiscuous mode
[1842954.504177] br-8e1500f7888c: port 3(veth33ec060) entered disabled state
[1842954.504313] audit: type=1700 audit(1619621873.868:133): dev=veth33ec060 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[1842969.732368] br-8e1500f7888c: port 1(veth14d68a8) entered blocking state
[1842969.732392] br-8e1500f7888c: port 1(veth14d68a8) entered disabled state
[1842969.738220] device veth14d68a8 entered promiscuous mode
[1842969.738559] audit: type=1700 audit(1619621889.118:134): dev=veth14d68a8 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[1842969.833746] br-8e1500f7888c: port 3(veth8ddedab) entered blocking state
[1842969.833769] br-8e1500f7888c: port 3(veth8ddedab) entered disabled state
[1842969.834149] device veth8ddedab entered promiscuous mode
[1842969.834327] audit: type=1700 audit(1619621889.218:135): dev=veth8ddedab prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[1842969.834807] br-8e1500f7888c: port 3(veth8ddedab) entered blocking state
[1842969.834828] br-8e1500f7888c: port 3(veth8ddedab) entered forwarding state
[1842970.040802] audit: type=1325 audit(1619621889.428:136): table=nat family=2 entries=19
[1842970.225787] audit: type=1325 audit(1619621889.618:137): table=filter family=2 entries=55
[1842970.370113] audit: type=1325 audit(1619621889.758:138): table=nat family=2 entries=20
[1842970.775343] br-8e1500f7888c: port 3(veth8ddedab) entered disabled state
[1842972.045066] audit: type=1325 audit(1619621891.428:139): table=nat family=2 entries=0
[1842972.084442] audit: type=1325 audit(1619621891.468:140): table=filter family=2 entries=0
[1842972.123702] audit: type=1325 audit(1619621891.508:141): table=nat family=2 entries=5
[1842972.147260] audit: type=1325 audit(1619621891.538:142): table=nat family=2 entries=7
[1842972.170875] audit: type=1325 audit(1619621891.558:143): table=nat family=2 entries=8
[1842972.326386] eth0: renamed from veth533bf08
[1842972.381912] IPv6: ADDRCONF(NETDEV_CHANGE): veth8ddedab: link becomes ready
[1842972.382102] br-8e1500f7888c: port 3(veth8ddedab) entered blocking state
[1842972.382116] br-8e1500f7888c: port 3(veth8ddedab) entered forwarding state
[1842972.508435] systemd-udevd[30]: Starting version 241
[1842973.276866] eth0: renamed from veth6d3cdd3
[1842973.337588] IPv6: ADDRCONF(NETDEV_CHANGE): veth14d68a8: link becomes ready
[1842973.337792] br-8e1500f7888c: port 1(veth14d68a8) entered blocking state
[1842973.337815] br-8e1500f7888c: port 1(veth14d68a8) entered forwarding state
[1842976.574902] kauditd_printk_skb: 15 callbacks suppressed
[1842976.574910] audit: type=1100 audit(1619621895.958:159): pid=12043 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[1842976.578914] audit: type=1101 audit(1619621895.968:160): pid=12043 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_permit acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[1842976.592963] audit: type=1103 audit(1619621895.978:161): pid=12043 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[1842976.605935] audit: type=1105 audit(1619621895.998:162): pid=12043 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_env,pam_env,pam_mail,pam_limits,pam_permit,pam_unix,pam_systemd acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[2590920.172276] systemd-udevd[29]: Starting version 241
[2590923.363596] audit: type=1100 audit(1620369836.600:163): pid=15801 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[2590923.366207] audit: type=1101 audit(1620369836.610:164): pid=15801 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_permit acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[2590923.369715] audit: type=1103 audit(1620369836.610:165): pid=15801 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'
[2590923.379305] audit: type=1105 audit(1620369836.620:166): pid=15801 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_env,pam_env,pam_mail,pam_limits,pam_permit,pam_unix,pam_systemd acct="chromium" exe="/bin/su" hostname=d896a2c addr=? terminal=pts/0 res=success'

rahul-thakoor · May 12, 2021, 12:50pm

Hi

I recently updated the docker image so it’s using Chromium 88

Can you confirm you are using browserBlock v2?

If so, you can expose port 35173 by adding

    ports:
        - '35173'

to your docker-compose file for the browser service.

This exposes the remote debugging port and you should be able to use an external chrome debugger.

Can you try this and see if you get more useful logs?

Thanks

vipulgupta2048 · May 21, 2021, 8:15am

Please let us know if this helped you in resolving your issue. If not, please do reach out. Happy to help.

esbenvb · May 26, 2021, 6:52am

Sorry for my poor understanding, but I’m not 100% sure how to make sure that I get the right version of the browser image into my container, using docker compose?

How can I translate from the versions here Tags · balenablocks/browser · GitHub

To the versions here Docker Hub

esbenvb · May 26, 2021, 11:32am

I now tried to add the port to the docker-compose file:

  browser:
    restart: always
    build:
      context: browser
    privileged: true
    network_mode: host
    ports:
      - "35173:35173"

I also added REMOTE_DEBUG_PORT 35173 to the browser service environment just to be sure

But still I can’t access http://192.168.1.200:35173/ in order to debug the browser.

I get no response on that port. What am I doing wrong?

rcooke-warwick · May 26, 2021, 12:42pm

I think that if you have the docker compose set up like this GitHub - balenablocks/browser: A drop-in web browser block then you should be using the latest version of the block, which will be v2.

Also, if you expose the port in the docker compose as rahul described - have you got your devices public url enabled? If you have got it enabled, then I think you should be able to access the port through http://<PUBLIC_DEVICE_URL>:35173

esbenvb · May 26, 2021, 1:44pm

I tried this: https://ef0e8538e69a984a122d9d1c3a04c527.balena-devices.com:35173/ after enabling public URL but I get no response.

I also tried port scanning all ports of the device on my LAN and I can easily access other services running on it (MQTT and some node.js HTTP endpoints) but not the debugging port of the browser.

My Dockerfile.template looks like this:

FROM balenablocks/browser:latest

ab77 · June 3, 2021, 8:28pm

Try mapping port 8080 to 35173 (i.e.):

     ports:
      - "8080:35173"

… then http://<PUBLIC_DEVICE_URL>:8080

Public device URLs (v.Current) only forward ports 80 and 8080 to the device, so as long as something is listening there on these ports, the request will get through.

Alternatively, if you are on the same network as the device, just change your container to host networking and whatever ports open on it will be accessible via the local LAN IP.

esbenvb · June 7, 2021, 9:21am

Thanks for the tip.

But unfortunately, it still does not give me any response on port 8080. Neither on the public URL or the local 192.168.1.200… Other services like mosquitto works fine on the RPI.

It really looks like chromium is not listening at all.

Here’s the docker compose entry for the browser

  browser:
    restart: always
    build:
      context: browser
    privileged: true
    network_mode: host
    ports:
      - "8080:35173"

Here’s the listening ports on the host instance:

root@ef0e853:~# lsof -i -P -n | grep LISTEN
systemd       1     root   53u  IPv6  18385      0t0  TCP *:22222 (LISTEN)
dnsmasq    1446   nobody    5u  IPv4  20014      0t0  TCP 10.114.102.1:53 (LISTEN)
dnsmasq    1446   nobody    7u  IPv4  20016      0t0  TCP 127.0.0.2:53 (LISTEN)
balenad    1448     root    8u  IPv6  20152      0t0  TCP *:2375 (LISTEN)
balena-en  1897     root    6u  IPv6  23316      0t0  TCP *:9001 (LISTEN)
balena-en  1913     root    6u  IPv6  23397      0t0  TCP *:1883 (LISTEN)
balena-en  1929     root    6u  IPv6  23527      0t0  TCP *:4321 (LISTEN)
node       3136     root   25u  IPv6  31773      0t0  TCP *:48484 (LISTEN)

esbenvb · June 7, 2021, 10:59am

OK now I made a little progress…

I added a FLAGS to the service --remote-debugging-port=35173 instead of the ENV variable mentioned in the documentation

Now the lsof shows

chromium-  2990     1000   97u  IPv4  33300      0t0  TCP 127.0.0.1:35173 (LISTEN)

However I cannot connect to this port from outside, neither using 35173 directly or mapping it to 8080, when accessing the local IP or the public device URL.

Just for the testing, I managed to access the chrome debugger, by doing a balena ssh ... and from the balena machine, SSH out to my server with a tunnel that forwards port 35173, but that’s not a viable solution when debugging in my production environment.

phil-d-wilson · June 10, 2021, 10:38am

Hey @esbenvb

There is a similar issue in the block repo, where I am stochastically working on this:
Chromium Remote Debugging · Issue #70 · balenablocks/browser (github.com)

I think you need to add --headless to your flags for it to work.

What we may end up doing is adding a DEBUG envar which will expose the port and add the headless flag when it’s enabled. I’ll try to have another look/test of this when I’m on support later.

Phil

esbenvb · June 10, 2021, 10:51am

In terms of chromium itself, I think that the --remote-debugging-port=35173 flag works as it should. When doing tricks with SSH tunnels, I can access the remote debugger.

My main problem now is that the host OS and the Docker does not expose the 35173 port to the LAN side or to the public URL, even though I try mapping it on port 80 or 8080…