You Can Build a Database, and how I'm doing it

5

Share

For years, I built web applications. While the web provided a comfortable layer of abstraction, my curiosity, particularly my long-standing interest in how databases work, pulled me deeper. I decided to dive into the depths of systems programming, and the shift has been great. On the web, we are shielded from actual hard problems; here, the challenges of managing memory, concurrency, and durable, stateful storage are completely different. Below is a collection of resources I’ve found valuable on this journey:

The right community makes all the difference when you’re learning something challenging, here's a list:

Database Internals Discord: Active discussions about database design, storage engines, and implementation details.

Software Internals Discord: Broader systems programming topics.

r/databasedevelopment: A subreddit for people interested in building databases.

I skip most podcasts and YouTube videos; too easy to fall down rabbit holes. I focus on lectures and talks:

CMU Database Group: Andy Pavlo’s database courses are legendary. These lectures cover everything from storage engines to query optimization. He is the GOAT.

The Cherno’s C++ Series: While Zedis is written in Zig, understanding C++ helps you grasp systems programming concepts.

Systems Distributed: This conference ages like wine, it only gets better.

Software Should Work: Each year it gets better.

Read a Postgres extension, like pgvector, PostGIS, timescaledb, etc.

Understand how Postgres connects a C function to a SQL statement, how it handles memory, and try to grasp everything.

If you’re interested in Zig, I recommend pgzx, the Zig binder I forked from xata.io for Postgres extensions

TidesDB: great embedded LSM engine, the creator Alex Gaetano is such a nice person.

SlateDB: great embedded LSM engine, too. It's written in Rust, but language should not be a barrier at this point, in my my mind the language should not be in the way of the metal, it’s all the same.

SQLite: It’s not quite straightforward to understand if you don’t know dbs already, but it’s worth studying.

You must know LSM, B-Trees and it's variations, skip list, hash maps (open address, linear probe, rehashing), disk based algorithm tradeoffs, e.g. a binary search is not the best algorithm for a disk based system,because of random I/O, when it comes to disk, the most you can get from each disk page, the best it is for your overall performance.

Reading complex codebase from the day 1, like ClickHouse, Postgres, MySQL.

Learn on your own, join communities. It's incredibly valuable.

Not coding. Do something, start small then increase the complexity.

These are the books I keep coming back to:

Database Internals by Alex Petrov: The book that demystified storage engines and distributed systems for me.

Designing Data-Intensive Applications by Martin Kleppmann: Essential reading for understanding how databases work in production.

Database System Concepts: The classic textbook. Comprehensive and detailed.

Zig is what made this journey possible for me. I tried C/C++ several times and bounced off hard, call it a skill issue if you want, but the C++ APIs feel like they’re fighting you. Zig was built for low-level control with good ergonomics, and it’s become my comfort language. I can’t recommend it enough, also the community is really great.

Zig Guide: Practical examples and explanations

Zig Discord: Incredibly helpful community

Build Your Own X: A curated list of tutorials for building your own databases, operating systems, compilers, and more.

awesome-dbdev: More focused collection of database development materials.

Tigerbeetle: The best database written in Zig.

Lastly, this Substack, of course. Thanks for reading!