Serialize build in local mode with docker-compose.yml

I have a project with three services defined in docker-compose.yml. Each runs a different command, but all three have a build section using the same Dockerfile:

  build:
    context: .
    dockerfile: Dockerfile.template

When pushing in local mode, this starts three docker builds on the device, one for each service, despite the fact that they are all the same Dockerfile. This usually results in a OOM kill and I need to delete all but one service, push, add the other two back, and then push again.

Is there some way to force these builds to be serialized so that each uses the cached build from the first? Or even better, tell the system that there really is only one Dockerfile and it only needs to build once?