By Kathryn Nave, Contributor
On the morning of January 27, 2014, Manfred Sidious, a fleet commander in space role-playing PC game “Eve Online,” missed a monthly in-game payment required to maintain the security of his coalition’s home star system. Over the next few hours, thousands of enemy players—called up from all over the globe by a rival coalition that had spotted this temporary weakness—converged upon the B-R5RB system’s borders. At 13:00 they attacked.
When the battle of B-R5RB came to a close 22 hours later, the damage amounted to 11 trillion ISK, around $300,000 when converted to real-world currency. To this day, B-B5RB remains the costliest battle in “Eve”‘s 17-year history—a history defined by relatively rare, but hugely destructive, massive-scale battles between the game’s many rival organizations.
“Eve” has its own in-game currency, ISK, which can be traded for paid subscription units—meaning players often invest their own money into the game. Earning enough ISK to construct a starship, by activities such as mining and trading, takes many weeks of real time, not including the additional time to actually build it. So when it comes to “Eve,” players’ investment is more than just emotional.
As the battle raged, Eve’s developers, CCP Games, were less concerned with this virtual infrastructure than its physical underpinnings—the single server, tasked with processing the actions of all 2,670 concurrent combatants and their consequences.
“It is just marvel of organizational ability to see the way that thousands of our players are able to coordinate on a scale like this,” explains CCP Games CEO Hilmar Veigar Pétursson. “Of course, it’s also extremely painful on the technical side.”
“When you have a situation where the server architecture can’t cope with the amount of commands that the players are sending, you have two options, neither of which are great,” Pétursson explains. “You’re either going to limit the number of commands that players can send or you stretch out in-game time to give the server more time to execute them. We chose consistency, at the cost of real-time gameplay.”
Fortunately, by the time of the battle of B-B5RB, CCP Games had a solution, albeit a far from perfect one. As numbers of online players in the system increased, they activated a mechanic called “time dilation.” This slows the in-game clock to as little as a tenth of normal speed, leaving players fighting in slow-mo, but granting the server time to process their actions in an orderly fashion.
Today, “Eve” is still one of the world’s most advanced massive-multiplayer online games (MMOs). Most MMOs, deal with large numbers of players by distributing them between separate and non-interacting duplicates of the game world, each of which is hosted on a separate server. In “Eve,” however, all 300,000-plus users play together. Actions taken in one sector ripple across the entire universe, and players can jump seamlessly between its 7,805 star-systems.
That means that the interactions of all these 300,000 players must, largely, be dealt with by one single server. Most of the time this works smoothly.
Yet when too many people gather, as they are doing right as Pétursson speaks, to wage massive-scale war in a single sector, on a single over-stressed server, then time dilation kicks in and the game slows to a crawl.
Pétursson is not entirely satisfied with this solution, and neither are “Eve”‘s players. But, as he points out, “Eve” was created a long time ago. Its foundations are fundamentally non-distributed with the majority of the simulation code organized around a single CPU core. To enable the distribution of a single complex virtual world across multiple servers, while allowing hundreds of thousands of players to seamlessly interact within it, would require rebuilding “Eve” from the ground up.
Foundations in the Cloud
Over the years, Pétursson has spoken with many companies working to solve the problem of distributed simulation. Every time, their efforts proved futile. Then, in 2018, he met Craig Beddis, the CEO of a small, relatively unknown, London-based start-up called Hadean.
Beddis believed the problem facing CCP Games had a deeper history that stretched back to the design decisions that laid the foundations of modern computing, long before the creation of “Eve.” The problem, as he sees it, lies with outdated operating systems that were based around a single CPU, created by people who had no expectation as to the vast networks of interconnected hardware that have since become available.
“The world has become increasingly cloud-centric, but the technology stack that we’re working with was designed for a different time and different era,” he explains. “We’re having to make everything we design backward-compatible, and, as a result, you end up with a lot of additional complexity that impacts the performance of any cloud-native applications you’re trying to build.”
“The world has become increasingly cloud-centric, but the technology stack that we’re working with was designed for a different time and different era..”—Craig Beddis, CEO of London-based startup Hadean
Hadean’s has spent the past five years stripping things right back to the metal, to design a cloud-native operating system from the ground up. They’ve taken the millions of lines of code, built up over the years to support cloud-based applications, and they’ve stripped it down to just tens of thousands of lines. What Hadean needed next was a way to show what this new platform was capable of enabling: The world’s largest virtual space battle seemed the perfect demonstration.
Virtual Warfare in Real Time
On March 20, 2019, Beddis and Pétursson stood on the stage at the Game Developers Conference in San Francisco to launch the first demo of a massive, multiplayer space warfare game: “Eve: Aether Wars.”
“They say, ‘never do a demo, and never do it live,'” Beddis recalls. “Well, we did it live at the biggest gaming event on the planet.”
Between, 5:30 pm and 6:30 pm, the number of concurrent players blasting each other’s spaceships to bits hit 2,379, supplemented by 10,000 AI-driven accounts that Hadean had added to maximally stress test their servers. “Aether Wars” stood up to the pressure.
The first demo was a crude proof of concept, built by a small team of Hadean’s developers using assets provided by CCP Games. Next, Hadean handed development over to CCP, to show what a group of professional game designers could build with the platform.
“This was to show how quickly Hadean allows you to build cloud-native game,” Beddis explains. “So a seven-person team out of CCP in Iceland put that together in just eight weeks.”
After demonstrating the second iteration in August 2019, a third demo followed in November of that year, hosted at London’s O2 Arena.
Game design is as much an art as a science, Pétursson points out, and by the time of the third demo the involvement of CCP’s designers had produced something that felt more like a game you’d actually want to play, rather than just a demonstration of the technology.
None beat “Eve”‘s previous Guinness World Record for the largest number of users engaged in a simultaneous player-versus-player game, held by the 6,142 pilots of January 2018’s Battle of 9-4RP2. But for the first time, all those thousands of players were able to engage in massively multiplayer dogfights, without having to wait for minutes to see the effect of their actions.
The name “Aether Wars” comes from a particular component of Hadean’s technology stack, called the Aether Engine, which was created specifically to enable cloud-based distributed computing for spatial simulations.
“The Aether Engine decomposes 3D space into boxes of different sizes depending on the complexity of a particular region,” explains Hadean’s VP of Operations Miriam Keshani. “So if this room was a simulation right now, there’d be one large box representing the empty space behind me, whereas I would be decomposed into lots and lots of smaller boxes.”
Each of these boxes can be assigned to different machines, and dynamically rescaled as regions increase or decrease in complexity.
“If I start pouring my water—well, now we’ve got a fluid dynamics problem,” Keshani says, holding her bottle up to the camera. “That’s really complex so we’d recruit thousands of little boxes, thousands of computers, to be responsible for just that part of the simulation. Then, the minute I stop pouring, it can cut off those resources and send them back to the cloud. So the developer is only paying for the resources that are being used at any given time.”
CCP Games are continuing to explore the potential of Hadean’s platform, while also working with Microsoft’s “Minecraft,” one of the world’s most popular online games.The desire to find more efficient ways to run higher-fidelity complex simulations is relevant not only to game designers building imaginary worlds, but also to scientific researchers seeking to create detailed models of complex physical interactions, allowing them to better predict what may happen in this one.
As the emergence of COVID-19 has brought increased attention to epidemiological modeling, Hadean, has begun working with the Francis Crick Institute and Imperial College London on using the spatial simulation capabilities of the Aether Engine to create agent-based models of transmission of the virus, both within the individual lung and throughout a population.
Where statistical models take a set of generalizations—such as average age or number of people per household—and extrapolate how they will affect the spread of a disease over time, agent-based approaches simulate interactions at the individual level to discover emergent global properties.
“Our model with Imperial is about looking at how people travel around known transportation networks,” explains Keshani, who has been leading the work on Hadean’s side. “Viruses spread by physical proximity, so you need to understand how people are literally moving through space. We’ve been helping add this spatial aspect to their simulation, so we can see, say, if we lockdown Leicester, what impact would that have on how the virus spreads.”
To Beddis, the COVID-19 pandemic looks like the moment for Hadean’s technology to come of age, both as people increasingly seek better ways to recreate their disrupted social networks in virtual space, and as it becomes ever more important to understand the complex physical networks enabling the transmission of disease.
“Technological innovation tends to come out of crisis,” he says, “For Hadean, you can say it’s partly fortuitous and partly good strategy that we are able to apply our technology to things that are very relevant in what is a very troubling and trying time.”