We currently use a container interacting with mmcli and nmcli of the host to manage two modems and here it is straight forward just to add another configuration and switch based on the carrier. I would prefer moving as much functionality as possible to the OS to avoid issues during updates that I eventually expect to happen although we haven’t seen any yet.
What is the recommended way to support multiple carriers in terms of apn, number, etc.? If the answer is multiple configurations in the system-connections file how do I then add a new carrier on the fly (in the containerized solution it would just be a matter of pushing and update).
You can create a connection profile using the NetworkManager D-Bus API or even better the libnm client library. The library will communicate with the NetworkManager daemon running on the host OS, and it will save a configuration file in /etc/NetworkManager/system-connections.
A connection could be modified on the fly through the API as well, but I am not sure whether changing the APN will take immediate effect as I have never done that. If that does not work you may always create a new profile, activate it and then delete the old one.