After about a month of hard work, and a huge amount of testing, I’m happy to say that we’re getting close to the new assets revamp being done!
For those of you who missed it, last Friday, I launched Phase 1 of the assets revamp. Today, I’ve just launched Phase 2. This was very stressful, but everything went well so far, so I’m happy.
There are two more phases to the whole process, and let me detail them here and explain what changes you’ll notice in the coming days.
- Phase 1: Asset download proxy
In this first stage of the migration, I’ve setup a new proxy that sits between the S3 storage which holds all of the asset files and the geo distributed CDN network that serves the files from 42 locations around the world. Now instead of downloading the files directly from the S3 storage, the CDN goes through my download proxy which will allow me to use the new assets redesign in a much more efficient way, and also eliminate one issue I’ve seen happening due to the S3 storage from the DigitalOcean provider having ridiculous low rate limiting (750 requests per second) which the CDN servers were sometimes hitting.
This whole system was set up on a Kubernetes cluster, and a lot of my time in the recent month was to teach myself Kubernetes and how to set things up properly. This is really good news, as it means it can now scale automatically based on demand, and will allow us to always have good performance serving the files. It also serves as “A foot in the door” so to speak for my 3rd big feature on the roadmap, which is the infrastructure change that I spoke about before. - Phase 2: Multi-upload optimizations
Today’s release adds multi upload optimizations. Well, it does much more than that, it actually replaces the entire assets library backend with the new revamped system, but because I can’t jump into the new design right away, I also have a compatibility layer in between, so that people who are currently already playing a game, or haven’t refreshed their browser’s page, would not have any issues uploading files. I also didn’t want to put everything available to use right away until I’ve tested things in the live production server and made sure it works, before I flip the switch on Phase 3.
In the meantime, the biggest change you will notice is that when you upload files through the assets library page on the Forge (not yet from the import wizard), it will do batch uploads of multiple files so that the uploads are faster and more efficient. Have a look at a little demo showing the optimized uploads vs the previous method (and notice how most of the time is spent extracting the zip file contents!).
- Phase 3
Once this is in use for a while (12 hours should be enough), I will enable phase 3 of the update, which will be the removal of the legacy compatibility layer. It will give us…Phase 3 is now live, and it gives us the ability to actually use the new assets library backend to its full power, and allow things like renaming the assets and an even higher level of optimizations to make file uploads efficient and fast (i.e: if the file you want to upload was already uploaded and available on the server, you can save your bandwidth and it will be copied directly from the server without you having to actually upload it).This should be coming tomorrow, though the rename functionality is not yet implemented, it should be done soon (I might even be able to finish it before tomorrow’s release). If there were no issues so far with the mass uploads from the assets library page, I will also enable it in the import wizard.Multi-file uploads are now supported in the import wizard, and file and folder renames are also available from the assets library.
- Phase 4
This would be the final phase of the assets revamp, and will be the rewrite of the file browser to have a more rich interface, allowing things such as drag & dropping files and folders for example and selecting multiple files for delete. This is not yet done, and will be what I’ll work on in the coming weeks.
One big item that will be missing (and which I’ll consider as a Phase 5) will be the auto-rename of assets within all existing worlds. I realized that doing that would be extremely time consuming (both to implement but also in terms of CPU usage per operation), and I’ve decided to drop it for now, so I can move on to other tasks. I’ll come back to it later, but in the meantime, when renaming files, a warning will be shown to remind the users that if they rename a file that is in use in a world, then that asset will become broken and they’d need to manually change it to the new URL within Foundry. Hopefully that’s enough to take care of most potential issues.
Once the new UI is built, I can move on to the next step in my roadmap: Adding Story Teller features, more specifically, the ability to have one separate URL per world and auto switching of worlds based on which URL is being accessed. I’m very excited to finally be able to do this.