Hi there 
There’s a slight difference between the built-in AV client and the world builder Livekit Voice and Video Chat module with its own Forge hosted server available on World Builder. I’ll assume that you guys are using the Livekit module and have correctly configured it to use the Forge server.
You’re right in that connection issues shouldn’t arise when there are module issues, and also in that connection issues shouldn’t prevent a character from being controlled or the tab interacted with to the point of needing to restart the browser.
A module error might send the whole tab or parts of the canvas or Foundry interactions with tokens into a non-reacting state, which could break the functionality of many modules, including livekit. It sounds like that might be what’s happening here, rather than the connection dropping. The fact that there is no error message might be explained by the error not being caught by proper error handling and printed as a notification inside of Foundry. A handled error usually wouldn’t cause the kind of behaviour you’re describing.
There are a couple of ways to approach troubleshooting it.
One simple way is to keep the dev tools (F12) console open when playing and seeing what gets logged there. Very often errors will get logged but won’t print out to a notification. If there is something that gets printed right before the behaviour you described happens, it’s possible that that is related in some way. Often the stack trace associated with the error message will mention the modules/data/files associated with the error, so if you could post a screenshot of that it might help us figure out what’s going wrong.
Another way to approach it would be to disable all modules except the Livekit module, and to check if it still happens. If it doesn’t, enable modules in small groups (from most necessary to least necessary), until it starts happening again, and then one of the modules in that group might be responsible.
I’d recommend starting with checking the dev tools console (F12) for errors if you’re running very few modules which you’re confident are compatible with each other and with the current Foundry version. If instead you’re running a lot of modules and aren’t sure about their compatibility, I’d recommend first checking if the error still happens with all of them turned off except Livekit.
Hope that helps! If you can grab a screenshot of one of those errors as they happen, it would help give context to the issue you’re running into!