Hello!
There could be a lot of reasons for this load time.
Locally, your own launched game will usually load quicker, if you have a better PC, while it will still take as long for external players to load in, in my experience.
Load times like these are usually based on Server placement, or world size.
Please make sure your Forge Server is running in your region ^^
Here is a link for the World Size Check, if you could respond with your result I would greatly appreciate it 