Answering my own question…
I included such a xorg.conf and it became normal: https://raw.githubusercontent.com/beagleboard/repos-arm64/e5e9f2b55abc394bd07a8ec7ce60f90d17e058e6/bbb.io-xfce4-desktop/suite/bullseye/debian/fbdev.xorg.conf
Just save this file as xorg.conf and include this line in your Dockerfile:
COPY xorg.conf /etc/X11/
Then it works!