It sounds like you are using the core A/V function in foundry which is peer to peer. If anyone has a poor connection in the group it will drag the others down as well. Most people do not use it and instead will use discord or if you want something within foundry there is support for Jitsi and LiveKit via modules. See this for more info
If you are on world builder subscription, you can access the LiveKit integration on the forge which would be mush more stable than the core A/V in foundry.