Update, for anyone else interested in doing the same I managed to figure it out.
Ask NetworkManager to scan for all the available APs and include the BSSID, Frequency and signal quality info in the list.
nmcli -f in-use,ssid,bssid,freq,signal,bars dev wifi
Your router broadcasts 2.4ghz and 5ghz under the same SSID, so what you want is to connect to the 2.4ghz specific BSSID.
nmcli d wifi connect XX:XX:XX:XX:XX:XX
All set.
I haven’t figured out how to make that preference persist across reboots though