Stalwart v1.0: The Fastest Mail Server on the Planet

Five years ago, on 3 October 2021, the first commit landed in a small repository called mail-parser. It was an e-mail parsing library written in Rust, and nothing more. Over the following five years it grew into a complete mail and collaboration server with JMAP, IMAP, POP3, SMTP, CalDAV, CardDAV and WebDAV, a spam filter, clustering, and support for half a dozen storage backends. Today, after five years of continuous development, we are proud to announce Stalwart v1.0, the first stable release.

Version 1.0 was never meant to be another feature release. It had two goals. The first was performance: making every part of the server as fast as it can be, and proving it with numbers. The second was stability of the data itself: finalising the database schema so that, from now on, upgrading Stalwart means installing a new version and nothing more, without the disruptive migrations some earlier releases required. Along the way a few new features found their way in too, including full-text search inside PDF and office documents.

Section titled “Months spent chasing every microsecond”

Most of the work in this release happened under a profiler. For months we measured where Stalwart spends its CPU cycles, its memory and its database round trips, rewrote whatever stood out, and measured again. The release contains close to 200 commits touching more than 1,300 files, and the vast majority of them exist to make something faster or lighter.

Section titled “Parsers rewritten with SIMD and SWAR”

A mail server spends a surprising share of its time looking for specific bytes: the end of a header line, a MIME boundary, an equals sign in a quoted-printable body, the space that ends an IMAP tag. Done one byte at a time, that is a lot of loop iterations for a 10 MB message. Two techniques let a CPU do this work many bytes at a time:

Stalwart now uses both, everywhere it matters. At the bottom of the stack sit two libraries that reach 1.0 together with the server:

Above them, the protocol layer got the same treatment. The IMAP command parser is now table-driven and scans whole runs of bytes instead of stepping a state machine per character. The SMTP command parser, the JSON parser behind JMAP, the spam filter tokenizers, the HTML-to-text converter, the variable-length integer codec and the log serializers all use SWAR scanning. Even the expression language, which administrators use for routing and filtering decisions, now compiles to a compact bytecode with specialised instructions for common comparisons and evaluates them in a recycled memory arena.

Section titled “Fewer, faster trips to the database”

Fast parsing is wasted if every request then waits on the database, so we spent just as long on how Stalwart talks to its storage. Identifiers are now assigned inside the same write transaction that uses them, removing whole round trips from every message delivery. Read-modify-write updates, such as adding a keyword or appending to a search index, run as merge operations inside the backend. Stored objects are compressed with zstd using dictionaries trained specifically on e-mail, calendar and contact data.

Each backend was also tuned individually:

On top of the storage layer sits a redesigned cache subsystem. Each account’s message index is held in memory in a compact packed form, and IMAP sorting, threading, flag fetches, searches on message properties and resynchronisation are now answered from that cache with few or no database reads. Very large accounts get their own cache tier with a memory budget, and cache snapshots can be swapped out to local disk, Redis or the blob store instead of being rebuilt.

Section titled “A new search engine”

The built-in full-text search engine was rewritten from the ground up. The previous design stored one bitmap per word and had no notion of word positions. The new engine stores postings in blocks of 65,536 documents and keeps word positions for every message. That makes real phrase searches possible, and lets a search inside one mailbox read only the blocks that mailbox touches. It indexes word stems and prefixes, compresses its data with zstd, and indexes new mail through a background queue with concurrency control, retries and rebuilds. The PostgreSQL and MySQL search backends were reworked as well. On a mailbox of 250,000 messages, a full-text search of every message body now returns in under 4 milliseconds.

Section titled “Stress tested on hundreds of vCPUs”

Optimising code without measuring it under realistic load is guesswork, so we built a dedicated benchmarking and stress-testing lab to go with this release. It runs on Microsoft Azure, across more than 30 regions, with up to 18 complete test labs running in parallel and several hundred vCPUs busy at once.

The test catalogue covers 35 kinds of test, spread over more than 550 individual test cells, and they cover:

Runs were profiled throughout. CPU time was sampled with perf and turned into flame graphs, off-CPU analysis showed where threads waited on locks and I/O, the heap was profiled with jemalloc and heaptrack, and a special instrumented build counted every storage round trip, per command, so we could see exactly which requests hit the database more than they should. Each fix was then measured against the previous build on matched machines before it was accepted.

The first full round of this cycle turned up 64 problems, which were fixed and retested. The second round re-ran everything and fixed 33 more. Some examples of what the lab caught:

This lab is not a one-off effort for the release. It is now part of how Stalwart is tested internally: every future release goes through the same benchmark and stress-test rounds before it ships, so performance gains stay gained.

Section titled “Faster than Postfix and Dovecot”

Internal numbers only tell you whether Stalwart got faster than it was. We also wanted to know how it compares to what most of the world runs. So we put Stalwart v1.0 head to head with the most widely deployed open-source mail stack: Postfix 3.11 for SMTP and Dovecot 2.4 for IMAP.

Both ran on identical Azure servers (8 AMD EPYC Zen 5 vCPUs, 32 GiB of RAM, local NVMe storage), loaded with the same 1,000 mailboxes plus three very large ones of up to 250,000 messages, and received exactly the same work: receiving mail, delivering it and serving it to IMAP clients. Encryption, password hashing and full-text search were configured the same way on both, and the two systems were measured alternately on the same hardware.

Stalwart was faster in 35 of the 38 headline comparisons.

The gap grows with the size of the mailbox. On a 250,000-message mailbox, a full-text search of every message body takes 3.75 ms in Stalwart and 136 ms in Dovecot, and moving or deleting 25,000 messages finishes in about a tenth of a second instead of 24 to 34 seconds: more than 200 times faster.

The three tests Dovecot won are all the same operation: marking every message in a mailbox as read at once.

Only Postfix and Dovecot were tested, but they are not just any two products. They are the engine under nearly every popular self-hosted mail server: every open-source dockerized mail server with more than 10,000 stars on GitHub, including docker-mailserver and mailcow, runs Postfix and Dovecot under the hood, as does Mail-in-a-Box. Beating that combination on almost every benchmark means beating the stack behind most of the self-hosted mail on the internet. We are now comfortable saying it plainly: Stalwart is the fastest mail server on the planet.

Don’t take our word for it. Every script, configuration, test scenario and raw result is in the benchmarks repository, together with a methodology and a frank list of caveats, such as the differences in disk-sync behaviour between the two systems. Anyone with an Azure account can provision the same servers and run the whole suite. Benchmarks of Stalwart’s different storage backends and clustered setups are being prepared for the same repository.

Section titled “What else is new”

Performance was the headline, but v1.0 also brings a set of new features:

Section titled “Upgrading to v1.0”

Finalising the database schema has a price that is paid once: Stalwart v1.0 uses a new database layout, and existing data has to be migrated to it. A v1.0 server will not open a store created by an earlier version.

We know that some past upgrades were bumpy, and we took that seriously. This migration was designed and tested to be as smooth as we could make it, and it is the last of its kind: the schema is now stable, and future releases will upgrade in place. There are two ways to get to v1.0.

Section titled “The migration tool”

For deployments running Stalwart v0.16, stalwart-migrate is the easiest route. It requires v0.16.27, so earlier 0.16 releases should be updated to it first. The tool converts your settings to the v1.0 schema and copies all data into a new, empty v1.0 store. It only reads from the old store, so the original deployment stays intact and can be restarted at any point before the switch.

It also migrates incrementally. The first copy runs while v0.16 keeps serving users, however long it takes. Each later run copies only what changed since the previous one, and the only downtime is the short final run, after v0.16 is stopped. Account and message identifiers, IMAP UIDs and change history are preserved, so IMAP and JMAP clients carry on without downloading anything again. Users who sign in with OAuth will need to sign in again, and the full-text index is rebuilt in the background after the switch.

Section titled “The migration proxy and Vandelay”

For anyone running Stalwart v0.15 or earlier, or anyone who prefers to move users one at a time, the migration guide describes the second route. The migration proxy sits in front of both the old and the new server and sends each user to whichever one holds their account, and Vandelay copies accounts across one by one, incrementally. Each account is switched by flipping one entry in a mapping table, with no downtime for that user, and the old server keeps a full copy until you decide to retire it. The same procedure also works for moving from Dovecot and Postfix to Stalwart. The migration overview compares the two approaches.

Section titled “What changed”

The structure of your settings is largely the same. Around a hundred keys were renamed to follow consistent naming conventions, a few objects (OIDC directories, AI models, Azure storage, forwarded headers) were restructured, and stalwart-migrate converts all of it automatically. Two changes need attention before you upgrade:

If you use an external identity provider that is not conformant, Stalwart’s token checks can be relaxed through the compatibility settings on the OIDC directory, at your own risk, since each one weakens a security check. These settings mainly keep web clients such as the WebUI working. What they cannot fix is a provider that lacks the profile itself, and that is going to matter soon: popular e-mail clients, including Apple Mail, are adopting the profile, and once they do, their users will not be able to authenticate against a non-conformant provider. If you depend on an external identity provider, now is a good time to ask them to support it.

A related change affects the WebUI. Instead of a fixed client id, it now identifies itself to the identity provider with a Client ID Metadata Document, a URL that describes the client. Some identity providers do not support this yet. With those, administrators can register a client for the WebUI with the provider and set its id in oauthClientId, and the WebUI signs in as before. The details are in client id of the WebUI.

Section titled “We are here to help”

If you run into a problem or have a question during the migration, post it on support.stalw.art. We will be answering questions there daily throughout the migration period.

Section titled “Thank you”

Five years ago this was one Rust