Forge Feature Roadmap

Hi everyone,

I thought you may all be interested in knowing about my current roadmap of features that I’m working on or planning on adding within the next few weeks or months. I’ll try to keep this up to date as much as possible.

There are of course many things that I want to do, but let’s concentrate on the next few major features :
Multiple Worlds Integration => Infrastructure revamp => Bazaar Update

Multiple Worlds Integration

This is the feature I’m most excited about and which I wanted to implement since the first day I envisioned The Forge. I am talking of course about the Story Teller features of having full user integration. What I want is to have the ability to launch into a world directly by going to its URL. You can already do that when you have a world that goes idle, but I want to be able to have one unique URL per world and have it auto launch the correct world based on the URL, but also auto-login into the game. This means that when you send an invitation link to a player and they join the game, The Forge would automatically create a new user for them within the world and log them in automatically as their player when they go to the world.

What I basically want is for the Forge to completely abstract away the /setup, /join and /players pages from Foundry to make it easier to manage those elements of the game and have full integration.
This requires a little bit more work and is a massive feature, and that’s what the Story Teller Tier is all about. At least, the systems and modules management parts of the setup page will already have been done with the inclusion of the Bazaar.

Infrastructure Revamp

To my great pleasure, The Forge has been growing quite well since the release, and while I’ve set things up to be able to scale easily as demand grows, I want to work a little bit more on that aspect of the service to make things even better. For most people, this will mean nothing, no particular change, or advantage, other than the potential for running more smoothly and less work for me to maintain the servers and keep an eye on them, especially during peak hours.
For the more technically inclined, I want to move the entire design of the service to using Kubernetes. This has quite a few challenges, and it seems I’ve already implemented a few of the things that Kubernetes does, so I’d need to adapt The Forge to use a different method of orchestration and process launching and scaling and routing and region inter-communication, etc… I’m a little excited to be doing this, cause it looks like a really interesting challenge and there will be a lot of knowledge that I would be gaining (and it’s always fun to learn new things!).
One additional challenge is that I will of course want to switch the infrastructure from what I currently have to the new Kubernetes system with zero downtime and a completely transparent move without any effect on the users. So I’ll work hard on making that happen. Unfortunately, it’s unrealistic to think that there won’t ever be a small period of adaptation as I iron out the bugs that I won’t be able to discover until I make the change happen in production.

Back to the Bazaar

When I’m done with these, I’ll go back to working on the Bazaar. This was one task that I did before but never fully completed. I want to mostly finish creating the new rich user interface for all the modules and add the system that needs to be in place for creators to sell their content on The Forge. Hopefully the UI for the Bazaar will be as great as the one I just did for the assets library (hey, I hope you liked that UI!).

Future

There’s more of course for the future, The World Builder tier features of course, but many other things that I want to do to improve the overall experience for everybody. For now, I don’t want to speculate on which features I will do in which order, and I think I’ll be busy enough with these 3 major features already. I don’t have an ETA but I expect within a couple of months, I’ll be revisiting this thread and updating it with the new roadmap.

Let me know if you have any questions or feedback on this.
Thanks!

9 Likes

4 posts were split to a new topic: Discussion about paid content and copyright

Roadmap update:

The Bazaar is (partially) done and has been removed from the roadmap. I’ve added two tasks to the roadmap now : Improving the payment management interface, and infrastructure change.

Here is a copy of what I had originally planned for the Bazaar, it might help keep track of the progress.

Bazaar

My current task is to work on the Bazaar. The Bazaar is the “Systems & Modules Installer” of The Forge, or you can also think of it as the “marketplace”, but I don’t like that term because it implies buying, but the Bazaar will be a mix of many things, and most of those will be free.

The reason why I think this is one of the most urgent features is that other than providing a way to install systems and modules (and worlds and assets) and having those packages auto-update when you launch your game, one important feature of the Bazaar is that if you install a package from it, it does not count towards your data quota.

I know that a lot of users think the 500 MB Data quota limit is low, but it’s actually quite high if your Data is only used for your worlds, because on average, a world takes only 6.1MB (according to the average of 2233 worlds from all user data directories), so 500MB is enough for a hundred different worlds, much more than you would ever need.
The real problem is when you install many modules or big systems (D&D 5e takes 61MB, PF2 takes 81MB and WFRP take 147MB for example) that take a lot of space, but I want everyone to be able to install modules without using up any disk space at all.

The urgency is made greater with the recent release of music and map packs modules which can take a couple of hundred MBs. I’ve made a temporary solution for that, but not everyone is aware of it and also not everyone knows that the modules are not supposed to use up your quota.

Post-release thoughts

The Bazaar is up and running for a few days now, and it seems to be going pretty well! It’s doing exactly what I wanted it to do, which is to free up everyone’s Data from system and module use.

What I originally had in mind was a bit more elaborate, with a complete and slick user interface. Think of your favorite Play Store, where you can search, see screenshot, reviews, comment on a system or module, etc… That’s what I wanted, but I didn’t get time to do it, and realized it would take a lot more effort actually, while the integration I did with Foundry’s installer worked out great already, so it became less of a priority.
This means that the Bazaar isn’t done yet, there’s still the UI portion of it that will need to come back into the roadmap eventually. For now however, I’m happy with what I have.

Another update with one major milestone finally achieved!

I have done a lot of improvements over the last couple of weeks with regards to the payment processing system but the major change is the ability now to pay with PayPal!

I will spare everyone the headache of explaining how ridiculously bad Paypal is! Sure, it’s great for users, I myself like to use Paypal when paying online, but from a developer’s perspective, it’s really really bad, and I guess people put up with it just because of its popularity. I had tried to do this back in May already, and I had to give up on it eventually, now I’ve redesigned the way I’m doing things and it’s much better, as it simplified the whole thing and it allows me to offer the free trial to Paypal users as well.

One big change as well is the new payment details screen. It may not look very different to you because the UI is practically the same, but it was entirely rewritten to use the Vue.js library, a reactive components framework that makes writing dynamic UI incredibly easy (at least, much easier than what I attempted to do before). This is important because I used that as a test bed for myself to learn how to use Vue and get myself ready to create better UIs. This will be extensively used for the new Assets Management interface, so it’s good that I got that done.
I wanted to also redo the subscription selection screen with Vue so it’s much smoother when you make changes to the UI, but I think I’m going to leave that to some other time as I’m eager to get started on assets, and I really want to finally do the story teller features! Yes, that interface could be improved greatly, but it’s not as urgent as the other stuff, so it can wait.

Regardless, I think I’m done now with the payment processing improvements. There’s still a few smaller things that need doing and I will keep slowly improving that aspect of the site, but I’m moving on now to concentrate my full attention on the assets library revamp.
That’s going to be a big task because it will involve a major rewrite of the backend that handles the assets, before I can tackle the actual user interface, and as usual, I do everything I can to make sure it has the least amount of impact on running games, but you should expect some maintenance period at some point when it’s all ready (I expect the maintenance would just be “you can’t upload new assets, but you can still access your existing ones”).

Here’s a list of the things related to the payment processing that I still want to do and expect will slowly be rolled out during the coming weeks as I do small updates while I work on the next big thing.

  • Show a direct link to the latest invoice and have a “Pay Now” button if payment had failed on it, so the user can trigger retries manually
  • Automatic retry a failed payment if the user updates their payment details
  • Rewrite the subscription manager UI to use Vue.js and be a better/more reactive UI
  • Allow removal of extras from the plan when upgrading to a higher tier
  • Add option to gift a sub to someone

Here is the original plan for this that was in the roadmap, keeping it here for archival purposes. I think I managed to do most of what I had set out to do, so I’m happy about that!

Payment processing improvements

I know that after the Bazaar, the next task was going to be Assets Management, but I feel like this is a slightly more urgent thing to take care of, so I’ve bumped it to the top of the list. It’s also because it’s a rather simple and quick thing that I will do, whereas the Assets revamp task will be quite extensive and take a lot of my time before it’s ready.

If you’re wondering what this task is about… just checkout the current process needed in order to change your billing information and you’ll understand how badly needed it is.

What I basically want is these things :

  • An area to change your billing information and payment methods
  • A quick link to view your payment history and invoices
  • Add Paypal support
  • Add warning notifications on the site when your payment has failed so you can be aware of it (for those who don’t check their emails)
  • Add a new Extra option to allow to increase the asset file size limit (so you can use UHD maps without having to upgrade your tier if you don’t want to)
  • Allow plan upgrade with disabling extras (for those who don’t need the extra quotas anymore on a higher tier)
  • Automate the locking/unlocking of accounts based on payment status

That last one is because at the moment, I need to check daily on who has failed to pay their subscription (due to credit card issue, insufficient funds, or blocked by bank while they verify the purchase is legitimate, etc…) and when a subscription has failed to be paid after multiple attempts, I need to lock the account (which usually results in the person finally noticing the emails I send and fix the problem), then once the invoice is paid, I need to be notified about it then manually unlock the account again. This takes time and is annoying, so it’s better if I make the website take care of that automatically, so it can free me up to do other things than handle these issues every day.

After about a month of hard work, and a huge amount of testing, I’m happy to say that we’re getting close to the new assets revamp being done!

For those of you who missed it, last Friday, I launched Phase 1 of the assets revamp. Today, I’ve just launched Phase 2. This was very stressful, but everything went well so far, so I’m happy.
There are two more phases to the whole process, and let me detail them here and explain what changes you’ll notice in the coming days.

  • Phase 1: Asset download proxy
    In this first stage of the migration, I’ve setup a new proxy that sits between the S3 storage which holds all of the asset files and the geo distributed CDN network that serves the files from 42 locations around the world. Now instead of downloading the files directly from the S3 storage, the CDN goes through my download proxy which will allow me to use the new assets redesign in a much more efficient way, and also eliminate one issue I’ve seen happening due to the S3 storage from the DigitalOcean provider having ridiculous low rate limiting (750 requests per second) which the CDN servers were sometimes hitting.
    This whole system was set up on a Kubernetes cluster, and a lot of my time in the recent month was to teach myself Kubernetes and how to set things up properly. This is really good news, as it means it can now scale automatically based on demand, and will allow us to always have good performance serving the files. It also serves as “A foot in the door” so to speak for my 3rd big feature on the roadmap, which is the infrastructure change that I spoke about before.
  • Phase 2: Multi-upload optimizations
    Today’s release adds multi upload optimizations. Well, it does much more than that, it actually replaces the entire assets library backend with the new revamped system, but because I can’t jump into the new design right away, I also have a compatibility layer in between, so that people who are currently already playing a game, or haven’t refreshed their browser’s page, would not have any issues uploading files. I also didn’t want to put everything available to use right away until I’ve tested things in the live production server and made sure it works, before I flip the switch on Phase 3.
    In the meantime, the biggest change you will notice is that when you upload files through the assets library page on the Forge (not yet from the import wizard), it will do batch uploads of multiple files so that the uploads are faster and more efficient. Have a look at a little demo showing the optimized uploads vs the previous method (and notice how most of the time is spent extracting the zip file contents!).
  • Phase 3
    Once this is in use for a while (12 hours should be enough), I will enable phase 3 of the update, which will be the removal of the legacy compatibility layer. It will give us… Phase 3 is now live, and it gives us the ability to actually use the new assets library backend to its full power, and allow things like renaming the assets and an even higher level of optimizations to make file uploads efficient and fast (i.e: if the file you want to upload was already uploaded and available on the server, you can save your bandwidth and it will be copied directly from the server without you having to actually upload it). This should be coming tomorrow, though the rename functionality is not yet implemented, it should be done soon (I might even be able to finish it before tomorrow’s release). If there were no issues so far with the mass uploads from the assets library page, I will also enable it in the import wizard. Multi-file uploads are now supported in the import wizard, and file and folder renames are also available from the assets library. :partying_face:
  • Phase 4
    This would be the final phase of the assets revamp, and will be the rewrite of the file browser to have a more rich interface, allowing things such as drag & dropping files and folders for example and selecting multiple files for delete. This is not yet done, and will be what I’ll work on in the coming weeks.

One big item that will be missing (and which I’ll consider as a Phase 5) will be the auto-rename of assets within all existing worlds. I realized that doing that would be extremely time consuming (both to implement but also in terms of CPU usage per operation), and I’ve decided to drop it for now, so I can move on to other tasks. I’ll come back to it later, but in the meantime, when renaming files, a warning will be shown to remind the users that if they rename a file that is in use in a world, then that asset will become broken and they’d need to manually change it to the new URL within Foundry. Hopefully that’s enough to take care of most potential issues.

Once the new UI is built, I can move on to the next step in my roadmap: Adding Story Teller features, more specifically, the ability to have one separate URL per world and auto switching of worlds based on which URL is being accessed. I’m very excited to finally be able to do this.

3 Likes

Aaaaaaaaaaand it’s done!

I’ve finally finished with the task of the assets revamp. The new UI is now available on the site and you can now select multiple files to delete or drag/drop them to move them to other directories, rename your files, you can also do batch uploads that go blazing fast and saves bandwidth for everyone! I’m really quite happy with the overall results of this Assets revamp project and I hope everyone is too.

I’ve managed to achieve most of everything that I wished to do, and more, but there is one thing that I didn’t get to, because it turned out to be extremely complicated to do properly: automatically renaming files in worlds to match file renames/moves.
Although I now have a design on how to achieve that optimally, it would still take a lot of time to implement it in a safe and risk free manner, so I’ve decided to drop that feature until the next time I revisit the assets library functionality, so I can move on to the more important/urgent issues to tackle. For now, there’s a big warning any time you want to edit a file or folder’s name that warns you that it will break content in games that reference those files, and I think that’s an acceptable solution for now.

I’ve had a lot of fun designing the new system, then implementing it, I’m also quite happy with my newfound skills at making reactive and usable UIs.

The next task now is going to be adding support for multiple game URLs per account and automatically switching the world based on the URL being accessed. That’s the Story Teller features. I may or may not decrease the scope again and leave the “auto sign in” features until later, depending on how urgent it becomes that I upgrade the site’s infrastructure as demand is growing and it slowly becomes both difficult and costly to maintaining it manually. I’m really excited about these, since that’s the original feature I wanted to do when I started this over a year ago now (we just celebrated the one year anniversary of the first commit! And launched a Contest to celebrate it too), so it basically took a year to get to a point where I could implement this feature. Can’t wait!

And for archival purposes, as always, here’s the original roadmap feature text which I’ve now removed from the first post :

Assets Management

One of the most requested feature is the ability to rename assets, as well select multiple files to delete, or drag & drop an asset in a different directory to make it easier to organize files after upload.

This is unfortunately not just a change in the UI, but it will require a revamp of the Assets Library system to even allow it because of how the S3 storage that backs the Assets Library does not allow for renames, only copies, and it’s rate limited to 2 copies per 5 minutes (absolutely ridiculous), so I want to create a new abstraction layer to allow for instant renames and moves and deletes, and it will allow me to bring back one of my original optimizations from before I moved to this S3 storage, which allowed me to save user’s bandwidth and time by doing a local copy if the same file had already been uploaded by another user, so you wouldn’t need to upload all your assets, only the new unique assets. I’ll also use this opportunity to improve the upload process to allow batch uploads in parallel to speed up big imports for example.

This feature will also allow me to do smart in-world renames, which is something I feel is needed before renames are even made possible. What this means is that if you rename an asset and that asset is in use within one of your worlds, I need to update that world to use the new asset URL automatically so it doesn’t break it.