Simple Rules for Building Simple Systems

Once a design of a system meets all the business constraints necessary, my next priority is to make the system as simple as possible. By simple, I mean a system that is both easy to understand and easy to change. It’s important for a system to be simple because the cost of maintaining a system is often going to be larger than the cost of building it. On top of that, if a system is easy to change, you can adapt the system to meet new business requirements as necessary.

This post is going to have a focus on backend software since that’s what I’ve been building my entire career.

When building a system, I try to keep all the data for the system in a single centralized database. The second you start storing state across multiple different servers you introduce a whole bunch of problems because you are now dealing with a real distributed system.

First, it’s now likely you will have to perform “cross database joins”. Perform a query against one of the databases, take the result of that, and use that data to query the second database. This makes it more difficult to consume data from your databases and moves something that should be happening in the database into your application logic.

A second problem with storing state across multiple servers is keeping the data in sync. Cache invalidation is known as one of the hardest problems in software engineering and is a form of keeping state across multiple servers. If there are bugs anywhere in your code, you can easily wind up with the data across the different databases not lining up. Once the data is out of sync, your data is in a sense “corrupted”. On top of debugging the problem, you now have to deal with resolving the conflicts between all the data.

Keeping data in a single centralized place means you don’t have to deal with any of these problems. Your application can get all the data it needs in a single query, and there’s less of a need to walk on eggshells when updating state.

There’s a lot of systems out there that are great for specialized use cases:

The thing is while these systems are all fantastic at the tasks they were designed for, you usually do not need to use them. Often services you are already using will be good enough. For example, if you’re dealing with a moderate amount of data, you could use Postgres for all of the above use cases. Working with a single Postgres DB is way easier than working with all three of ElasticSearch, Kafka, and MongoDB.

Of course, there is a point where you’re working with enough data that you will need to use a system like ElasticSearch or Kafka in order to scale to your needs. Fortunately you can usually get by with a more general piece of software like Postgres or Redis.

If you do need to introduce a new system, you should look at buying a solution instead of setting up and managing your own. These days, there’s tons of purchasable services out there focused on solving common software problems. Some of my personal favorites are:

You can think about purchasing any of these services as kind of like hiring an entire team of engineers to focus specifically on that problem. They take care of all the hard work of getting everything running and provide you a simple and easy to use interface. In some cases, buying software can cost you more than if you were to build it in house, but you’ll usually know when that will happen before you make the purchasing decision.

I hope you take these lessons to heart. I’ve found them useful many times throughout my career. If you have any rules or guidelines that you personally use to design simple systems, please let me know on Twitter at @mmalisper.