Created
Jul 17, 2022 03:23 AM
Introduction
It’s been one and a half years since our last technical blog post about the initial new ride system and a lot has happened since then. So let’s sit down and talk about the progress, innovation, and engineering that has taken place since then.
With that said, this article will go super in-depth about the system we build and how it works, our last article had a main focus on the math behind these systems, and this blog post will go into the design and software side of things.
It’s certainly not for everyone, but you might have some cool Aha! moments after reading.
Foreword
I want to preface this blog post with a notice about traincarts. It’s served us well ever since the first server opened in June of 2018 and might as well be the reason why this server was able to evolve from a dream into reality. The release of the companion plugin, TCCoasters was the spark that lit what now turned out to be the second wave of Minecraft theme park servers and the team behind it deserves all the credit and respect for their work.
But like in any relationship, personal growth, ambitions, and different philosophies can lead two to grow apart. We’ve hit a point where we need to continue growing and build new amazing rides and park features, without being held back by restrictions from a time long gone.
The problem
Mouske had a great explanation in the latest newswire about tick-time and performance implications that the old ride system had. But for those who didn’t tune in;
For those who don’t know, the tick speed is how Minecraft executes time. Movement, interaction, plant growth, and more all rely on this. If the game is running perfectly, it runs at 20 ticks per second. If it’s not, however, this can plummet. If you notice rides freezing on the server, that’s usually because the tick speed has gone down. The old ride system we used was incredibly powerful but had drawbacks. It ran calculations every tick, 20 times a second, for every ride vehicle on the server to keep things running. These thousands of calculations per minute would cause the server to begin slowing down as it caught up on the countless things being asked of it. For reference, everything on the server outside of rides such as commands, player movement, toys, animatronics, and more all took up 20% of the server workload. That ride system alone took the other 80% at any given time. The old system made up approximately 80% of the server’s workload on average. ~Mouskegamer, https://youtu.be/Zx8yQOcUnlk
This means that we have calls to update all in-game elements (toys, rides, commands, and tons more) which all have to be done within 50 milliseconds, going over that threshold by only 1ms will result in lag and show timings desyncing, completely ruining the player experience.
Profilers (which are applications that show which lines of code are the slowest by proportion) show that this 80% overhead comes from a few core elements, these are;
- Entity updates
- Track/orientation calculation
- Entity tracking
- Entity interactions
- Ride-related triggers and Redstone (effects etc)
So we set up a plan to completely redesign our stack, and eliminate all of these components.
Blender animations rev. 1
As discussed in the previous blog post, we set up our first revision as a proof of concept of a blender-based animation system. It served us well, but also brought up some technical challenges that had to be overcome. Two months after the release of the Luigi's ride, we decided to drop that project and start again from scratch with everything that we learned. The main changes in the second revision should include:
- Raw usage of netty (hooking into the network)
- Trying to completely circumvent any native Minecraft server code
- Moving to Java instead of Kotlin so we could all work on it
- More careful design choices in abstraction to re-use entity logic for non-ride elements
This made the project more ambitious and challenging but would allow us to not only have a system that could handle flat rides, but a system that could completely replace all entity handling, including animatronics, static display entities, and toys.
Cobalt - Project setup
To understand the engineering magic behind our system (called “Cobalt”), we first need to understand how plugins typically work. Typically when you execute any command, your Minecraft Game Client will send a packet (which you can think of as a message) to the server telling it what the player did. The server then “schedules” that command to run in the next tick, notifying the corresponding plugin to go ahead and do its thing.
Cobalt works entirely on the network layer, meaning that it intercepts all traffic coming from and to the Minecraft server, and immediately handles it in a worker. This means that when a player does anything related to Cobalt, the Minecraft server won’t even know that something happened, not wasting time doing any processing at all. This also means that we can see whenever the server tells something to the client, like sending a new piece of the world, in which case we can register that this specific player now has a specific portion of the world loaded for them.
This is massively oversimplified to not give too many details and keep it understandable for non-developers, but all of this allows us to handle player interaction and do low-level computation with the Minecraft protocol, not adding any server overhead (as cobalt handles everything in its thread pool and is entirely horizontally scalable)
Within this framework, we implemented as many normal spigot API’s as we could, to make it incredibly easy for our other developers to hook into and interact with entities.
In practice, this means that we can handle 100k tracked and functional entities, without the Minecraft server even knowing about their existence, interfering with existing plugins and making the Client believe that everything is perfectly normal. On top of this rock-solid foundation, we build our entire new ride system and some other projects that you’ll hear about shortly.
Quick code examples;
Spawning an entity with the Bukkit API
Spawning a Cobalt entity and registering it
ArmorStand armorStand = (ArmorStand) loc.getWorld().spawn(loc, ArmorStand.class);ArmorStand armorStand = new VirtualArmorStand(location, entityHandle);Both have the same client-side behaviour, but one running in Minecraft, and the other running in Cobalt.
We also have a handful of tools at our disposal, like this in-game visualization of the cobalt entity tracker, showing the chunks around me, and the amount of tracked Cobalt entities within them.
The ride system
Entity handling and tracking are unfortunately only a small part of designing a ride system. Arguably, the most important part is making smooth animations and being able to play them back without problems. Animations (like ride vehicles, animatronics, and shows) consist of “frames” just like regular 3d animation as you see in movies, this means that there are thousands or millions of small pre-determined movements, all being played in quick succession to visualise movement and ease.
Our old system used to calculate every single frame on the fly during an animation, meaning that if you were on a ride, the server would have to constantly calculate the next position of every single seat and model from that vehicle. This is extremely taxing and introduced most of the overhead when used on a big map like ours, which resulted in lag.
Cobalt animations are completely pre-baked, meaning that there’s a large file for every animation specifying exactly what needs to be spawned in, where they need to be at what point in time, in what frame, a how they should transition between frames, and what effects should be triggered when.
This leaves nothing up to be decided during runtime and means that very little (if anything) has to be calculated on the fly and the few things that have to still be done, will be handled by Cobalt instead of the Minecraft server.
So even if there were millions of calculations to be made, then that overhead will be limited to that single ride instance, instead of lagging the entire server.
Future-proofing
This entire endeavour had the goal of dropping the limitations that we’re being held by design decisions prior, so the last thing we wanted to do is to create a new rule-set that will have to be enforced for our future projects. To give us the technical freedom that we desired so much, every Entity is built on top of layers of abstraction, which allows us to create new custom entities with logic (like seats that the player can ride), Animatronics that don’t have any user interaction logic, show elements with special rendering rules (like element-specific render distance and optimization), and much more.
Each
VirtualEntity carries its own EntityNetworkManager which is a translation layer between our APIs and tracker, and the Minecraft netty protocol, handling direct communication with clients.
In practice, this means that a “Vehicle”, which makes up most of the ride logic, consists of multiple parents that can have overarching and default behaviour, like implementing the “
onSteer” method for interactive rides like Teacups or the cabs from Roger Rabbit’s Car Toon Spin. Each vehicle holds multiple “VirtualEntity” instances, which make up the seats, models, or particle effects like the chimney from Big Thunder Mountain Railroad
Creating and baking animations
As we discussed in our previous article, we make our new ride releases in Blender, giving our Imagineers the most accurate and industry-standard tools to create anything they want and making their vision of a ride come to life.
Problem is, however, that we already have dozens of rides in our park that would need to be converted to this new system to completely discontinue our old ride system. Though, it would probably stress Mouske out if I asked him to animate every ride we have in blender in like a week.
So instead of doing all that, we developed an in-house pipeline capable of;
- Loading tc path nodes
- Using sign discovery to add markers for events
- Load TC animations and bake them in the parent animation
- Apply post-processing
Converting a ride with this toolchain takes only 10~ minutes, and leaves us with a complete 3d baked animation with all the required metadata for Cobalt.
This video shows a converted render of the dataset from Big Thunder Mountain (note that it isn’t running in real-time for debugging purposes)
The old article goes into further detail about our blender pipeline, so you might want to check that one if you’re interested in the other pipeline.
Metrics and results
This article mainly focused on Cobalt as it's used in our rides, but it’s only the beginning of where we want to apply this new system and we’re still discovering new possibilities it opens for us. We’ve already done some tests converting all “static” (non-moving) armor stands and animatronics to cobalt for testing purposes which had some incredible results.
The following screenshot is from production measurements on the 8th of July, and it shows us that cobalt is actively tracking 500 entities per server as well as calculating movement updates and animation playback, with the tick-time being measured in microseconds, making our new system almost 200 times faster with real-world load!
I hope you find this as exciting as we do because all of this nerd talk means that given these current developments and roadmap for releases, we’ll have tons more headroom to fit a lot more players in a server, as well as have plenty of technical freedom to build even bigger and bolder things
Extra’s
Here’s an early demo of the new entity tracker with player-specific rendering options
Here are just some random behind-the-scenes screenshots from datasets, utilities and early tests that are neat to look at
Written by