Hello! I'm Andrey Makhorin, Server Developer at Pixonic (MY.GAMES). In this article, I'll share how my team and I created a universal solution for backend development. You'll learn about the concept, its outcome, and how our system, called Singularity, performed in real-world projects. I'll also go deep into the challenges we faced.
Background
When a game studio is starting out, it's crucial to quickly formulate and implement a compelling idea: dozens of hypotheses are tested, and the game undergoes constant changes; new features are added and unsuccessful solutions are revised or discarded. However, this process of rapid iteration, coupled with tight deadlines and a short planning horizon, can lead to the accumulation of technical debt.
With existing technical debt, reusing old solutions can be complicated since various issues need to be resolved with them. This is obviously not optimal. But there is another way: a “universal framework”. By designing generic, reusable components (such as layout elements, windows, and libraries that implement network interactions), studios can significantly reduce the time and effort required to develop new features. This approach not only reduces the amount of code developers need to write, it also ensures the code has already been tested, resulting in less time spent on maintenance.
We’ve discussed feature development within the context of one game, but now let’s look at the situation from another angle: for any game studio, reusing small pieces of code within a project can be an effective strategy for streamlining production, but eventually, they'll need to create a new hit game. Reusing solutions from an existing project could, in theory, accelerate this process, but two significant hurdles arise. First of all, the same technical debt issues apply here, and second, any old solutions were likely tailored to the specific requirements of the previous game, making them ill-suited for the new project.
These issues are compounded by further issues: the database design may not meet the new project's requirements, the technologies may be outdated, and the new team may lack the necessary expertise.
Furthermore, the core system is often designed with a specific genre or game in mind, making it difficult to adapt to a new project.
Again, this is where a universal framework comes into play, and while creating games that are vastly different from one another may seem like an insurmountable challenge, there are examples of platforms that have successfully tackled this problem: PlayFab, Photon Engine, and similar platforms have demonstrated their ability to significantly reduce development time, allowing developers to focus on building games rather than infrastructure.
Now, let’s jump into our story.
The need for Singularity
For multiplayer games, a robust backend is essential. Case in point: our flagship game, War Robots. It’s a mobile PvP shooter, it has been around for over 10 years and it’s accumulated numerous features requiring backend support. And while our server code was tailored to the project's specifics, it was using technologies that had become outdated.
When it came time to develop a new game, we realized that trying to reuse War Robots’ server components would be problematic. The code was too project-specific and required expertise in technologies that the new team lacked.
We also recognized that the new project's success was not guaranteed, and, even if it did succeed, we’d eventually need to create yet another new game, and we’d be facing the same "blank slate" problem. To avoid this and do some future-proofing, we decided to identify the essential components required for game development and then create a universal framework that could be used across all future projects.
Our goal was to provide developers with a tool that would spare them the need to repeatedly design backend architectures, database schemas, interaction protocols, and specific technologies. We wanted to free folks from the burden of implementing authorization, payment processing, and user information storage, allowing them to focus on the game's core aspects: gameplay, design, business logic, and more.
Additionally, we wanted not only to accelerate development with our new framework, but also to enable client programmers to write server-side logic without deep knowledge of networking, DBMS, or infrastructure.
Beyond that, by standardizing a set of services, our DevOps team would be able to treat all game projects similarly, with only the IP addresses changing. This would enable us to create reusable deployment script templates and monitoring dashboards.
Throughout the process, we made architectural decisions that took into account the possibility of reusing the backend in future games. This approach ensured that our framework would be flexible, scalable, and adaptable to diverse project requirements.
(It’s also worth noting that the development of the framework was not an island – it was created in parallel with another project.)
Creating the platform
We decided to give Singularity a set of functions agnostic to the genre, setting, or core gameplay of a game, including:
- Authentication
- User Data Storage
- Game Settings and Balance Parsing
- Payment Processing
- AB Testing Distribution
- Analytics Service Integration
- Server Admin Panel
These functions are fundamental to any multi-user mobile project (at the very least, they’re relevant to projects developed in Pixonic).
In addition to these core functions, Singularity was designed to accommodate more project-specific features closer to the business logic. These capabilities are built using abstractions, making them reusable and extensible across different projects.
Some examples include:
- Quests
- Offers
- Friends list
- Matchmaking
- Rating tables
- Online status of players
- In-game notifications
Technically, the Singularity platform consists of four components:
- Server SDK: This is a set of libraries based on which game programmers can develop their servers.
- Client SDK: Also a set of libraries, but for developing a mobile application.
- A set of ready-made microservices: These are ready-made servers that do not require modification. Among them are the authentication server, balance server and others.
- Extension libraries: These libraries already implement various features, such as offers, quests, etc. Game programmers can enable these extensions if their game requires it.
Up next, let’s examine each of these components.
Server SDK
Some services, like the profile service and matchmaking, require game-specific business logic. To accommodate this, we've designed these services to be distributed as libraries. By then building on top of these libraries, developers can create applications that incorporate command handlers, matchmaking logic, and other project-specific components.
This approach is analogous to building an ASP.NET application, where the framework provides low-level HTTP protocol functionality, meanwhile, the developer can focus on creating controllers and models that contain the business logic.
For example, let's say we want to add the ability to change usernames within the game. To do this, the programmers would need to write a command class that includes the new username and a handler for this command.
Here's an example of a ChangeNameCommand:
public class ChangeNameCommand : ICommand
{
public string Name { get; set; }
}
An example of this command handler:
class ChangeNameCommandHandler : ICommandHandler<ChangeNameCommand>
{
private IProfile Profile { get; }
public ChangeNameCommandHandler(IProfile profile)
{
Profile = profile;
}
public void Handle(ICommandContext context, ChangeNameCommand command)
{
Profile.Name = command.Name;
}
}
In this example, the handler must be initialized with an IProfile implementation, which is handled automatically through dependency injection. Some models, such as IProfile, IWallet, and IInventory, are available for implementation without additional steps. However, these models may not be very convenient to work with due to their abstract nature, providing data and accepting arguments that are not tailored to specific project needs.
To make things easier, projects can define their own domain models, register them similarly to handlers, and inject them into constructors as needed. This approach allows for a more tailored and convenient experience when working with data.
Here's an example of a domain model:
public class WRProfile
{
public readonly IProfile Raw;
public WRProfile(IProfile profile)
{
Raw = profile;
}
public int Level
{
get => Raw.Attributes["level"].AsInt();
set => Raw.Attributes["level"] = value;
}
}
By default, the player profile does not contain the Level property. However, by creating a project-specific model, this kind of property can be added and one can easily read or change player-level information in command handlers.
An example of a command handler using the domain model:
class LevelUpCommandHandler : ICommandHandler<LevelUpCommand>
{
private WRProfile Profile { get; }
public LevelUpCommandHandler(WRProfile profile)
{
Profile = profile;
}
public void Handle(ICommandContext context, LevelUpCommand command)
{
Profile.Level += 1;
}
}
That code clearly demonstrates that the business logic for a specific game is insulated from the underlying transport or data storage layers. This abstraction allows programmers to focus on the core game mechanics without worrying about transactionality, race conditions, or other common backend issues.
Further still, Singularity offers extensive flexibility for enhancing game logic. The player's profile is a collection of "key-typed value" pairs, enabling game designers to easily add any properties, just as they envision.
Beyond the profile, the player entity in Singularity is made up of several essential components designed to maintain flexibility. Notably, this includes a wallet that tracks the amount of each currency within it as well as an inventory that lists the player's items.
Interestingly, items in Singularity are abstract entities similar to profiles; each item has a unique identifier and a set of key-typed value pairs. So, an item doesn't necessarily need to be a tangible object like a weapon, clothing, or resource in the game world. Instead, it can represent any general description issued uniquely to players, like a quest or offer. In the following section, I’ll detail how these concepts are implemented within a specific game project.
One key difference in Singularity is that items store a reference to a general description in the balance sheet. While this description remains static, the properties of the individual item issued can change. For example, players can be given the ability to change weapon skins.
Additionally, we have robust options for migrating player data. In traditional backend development, the database schema is often tightly coupled with the business logic, and changes to an entity’s properties typically require direct schema modifications.
However, the traditional approach is unsuitable for Singularity because the framework lacks awareness of the business properties associated with a player entity, and the game development team lacks direct access to the database. Instead, migrations are designed and registered as command handlers that operate without direct repository interaction. When a player connects to the server, their data is fetched from the database. If any migrations registered on the server have not yet been applied to this player, they are executed, and the updated state is saved back to the database.
The list of applied migrations is also stored as a player property, and this approach has another significant advantage: it allows migrations to be staggered over time. This allows us to avoid downtimes and performance issues that massive data changes might otherwise cause, such as when adding a new property to all player records and setting it to a default value.
Client SDK
Singularity offers a straightforward interface for backend interaction, allowing project teams to focus on game development without worrying about issues of protocol or network communication technologies. (That said, the SDK does provide the flexibility to override default serialization methods for project-specific commands if necessary.)
The SDK enables direct interaction with the API, but it also includes a wrapper that automates routine tasks. For instance, executing a command on the profile service generates a set of events that indicate changes in the player’s profile. The wrapper applies these events to the local state, ensuring the client maintains the current version of the profile.
Here’s an example of a command call:
var result = _sandbox.ExecSync(new LevelUpCommand())
Ready-made Microservices
Most services within Singularity are designed to be versatile and do not require customization for specific projects. These services are distributed as pre-built applications and can be utilized across various games.
The suite of ready-made services includes:
- A gateway for client requests
- An authentication service
- A service for parsing and storing settings and balance tables
- An online status service
- A friends service
- A leaderboard service
Some services are fundamental to the platform and must be deployed, such as the authentication service and gateway. Others are optional, like the friends service and leaderboard, and can be excluded from the environment of games that do not require them.
I'll touch on the issues related to managing a large number of services later, but for now, it's essential to emphasize that optional services should remain optional. As the number of services grows, the complexity and onboarding threshold for new projects also increase.
Extension Libraries
While Singularity’s core framework is quite capable, significant features can be implemented independently by project teams without modifying the core. When functionality is identified as potentially beneficial for multiple projects, it can be developed by the framework team and released as separate extension libraries. These libraries can then be integrated and utilized in-game command handlers.
Some example features that might apply here are quests and offers. From the core framework’s perspective, these entities are simply items assigned to players. However, extension libraries can imbue these items with specific properties and behavior, transforming them into quests or offers. This capability allows for dynamic modification of item properties, enabling the tracking of quest progress or recording the last date an offer was presented to the player.
Results so far
Singularity has been successfully implemented in one of our latest globally available games, Little Big Robots, and this has given the client developers the power to handle the server logic themselves. Additionally, we've been able to create prototypes that utilize existing functionality without the need for direct support from the platform team.
However, this universal solution is not without its challenges. As the number of features has expanded, so has the complexity of interacting with the platform. Singularity has evolved from a simple tool into a sophisticated, intricate system—similar in some ways to the transition from a basic push-button phone to a fully-featured smartphone.
While Singularity has alleviated the need for developers to dive into the complexities of databases and network communication, it has also introduced its own learning curve. Developers now need to understand the nuances of Singularity itself, which can be a significant shift.
The challenges are faced by folks ranging from developers to infrastructure administrators. These professionals often have deep expertise in deploying and maintaining well-known solutions like Postgres and Kafka. However, Singularity is an internal product, necessitating that they acquire new skills: they need to learn the intricacies of Singularity's clusters, differentiate between required and optional services, and understand which metrics are critical for monitoring.
While it's true that within a company, the developers can always reach out to the platform's creators for some advice, but this process inevitably demands time. Our goal is to minimize the barrier to entry as much as possible. Achieving this necessitates comprehensive documentation for each new feature, which can slow down development, but is nonetheless considered an investment in the platform's long-term success. Moreover, robust unit and integration test coverage is essential to ensure system reliability.
Singularity heavily relies on automated testing because manual testing would require developing separate game instances, which is impractical. Automated tests can catch the vast majority—that is, 99%—of errors. However, there's always a small percentage of issues that only become evident during specific game tests. This can impact release timelines because the Singularity team and project teams often work asynchronously. A blocking error might be found in code written long ago, and the platform development team may be occupied with another critical task. (This challenge is not unique to Singularity and can occur in custom backend development as well.)
Another significant challenge is managing updates across all projects that use Singularity. Typically, there is one flagship project that drives the framework's development with a constant stream of feature requests and enhancements. Interaction with this project's team is close-knit; we understand their needs and how they can leverage our platform to solve their problems.
While some flagship projects are closely involved with the framework team, other games in early development stages often operate independently, relying solely on existing functionality and documentation. This can sometimes lead to redundant or suboptimal solutions, as developers might misunderstand the documentation or misuse the available features. To mitigate this, it's crucial to facilitate knowledge sharing through presentations, meetups, and team interchanges, although such initiatives do require a considerable investment of time.
The Future
Singularity has already demonstrated its value across our games and is poised to further evolve. While we do plan to introduce new features, our primary focus right now is on ensuring that these enhancements do not complicate the platform’s usability for project teams.
Besides this, it is necessary to lower the barrier to entry, simplify deployment, add flexibility in terms of analytics, allowing projects to connect their solutions. This is a challenge for the team, but we believe and see in practice that the efforts invested in our solution will definitely pay off in full!