Hi,
The audio/video would require a certain amount of bandwidth and processing power, so if you have a bad internet connection, switching to audio only was probably the best choice.
I don’t know exactly about the audio issue with jitsi, I’ve generally heard that it’s stable, so it may have been a fluke, or it might have been related to the bad internet connections you had, which would be disruptive for this kind of thing, though it’s supposed to be able to recover from those kinds of connectivity issues.
There’s unfortunately not much I can help with. I’ve well versed in that (I wrote the native audio/video feature in Foundry) and I also have no idea where to look for the source of the issues. All I can say is that audio/video is a peer to peer system that works as a mesh, so it wouldn’t go through the servers, so if an issue happened, it’s more like due to a connection issue between you and the other people in your group. Note that it’s a mesh network, so you would be connected individually to each of your 5 players (assuming it was 5) and uploading your audio to each of the 5 players separately, and downloading from audio from them individually as well.
Jitsi will generally work better for people with a lower internet speed, while sacrificing latency, as it would all go through the Jitsi server, which would mean you have a single connection and would send once and receive once, thus decreasing the bandwidth and CPU resources needed.
For more information about all that, maybe you will find this post informative :
I hope that helps