Where is the latency actually coming from when using Airplay?

So I’m trying to see if I can reduce the latency as much as possible for streaming audio, and I’m wondering where the latency is actually coming from.

I set my input and output latency to 100ms, and my multiroom latency is set to the default 300ms, so there should only be 500ms coming from the actual configuration, but I have about 3-5 seconds of actual latency from my airplay source to the speaker.

Does anyone know where the rest of the latency would be coming from? I.e. is it from the multiroom server and client containers, or the airplay container, or from the airport source/protocol directly?