Hi there 
Are you using Foundry’s native audio/video integration?
We’ve noticed that this can be unreliable. I’d encourage you to look at the LiveKit AV Client module and read through the article here: Livekit Voice and Video Chat
Hope that helps!