Do the world size check as @subourbonite suggested, and also do the Module Check: Trouble with Modules
Also list out your Modules, it sounds like you have one that is leaking memory based on what you describe. I play for 5.5 hours each Saturday, and have never had Forge/FoundryVTT slow down on me at all (or for my players).
In addition, do you have large audio files? If so, check “Large File Streaming” on them individually, that vastly improves the audio handling in 0.7.9.