
Building Durable Streams on Postgres
How and why we designed a solution for durable streaming and built it on our favorite database.
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
How and why we designed a solution for durable streaming and built it on our favorite database.
Julius ParkFounding EngineerHatchet
There is an ancient proverb that goes something along the lines of:
No man ever steps in the same stream twice, for it is not the same stream and he is not the same man.
This is obviously a pretty ludicrous statement. The year is 2026, and we have the technology to step in the same stream as many dang times as we would like to. Put that in your pipe and smoke it, Heraclitus.
At Hatchet, our goal is creating useful and ergonomic building blocks for developers building distributed, async-heavy applications. We originally had a streaming feature, but it was not durable, putting the onus on the developer to cache responses and avoid races between workflow starts and subscriptions.
This is not ergonomic, thus we have built durable streams as a more ergonomic solution. Let’s walk through a common use-case for this feature, and the technical decisions that flow from it.
Chat interfaces have become ubiquitous in the age of large language models, to applause and chagrin. Most chat interfaces revolve around the user inputting text, and getting back a textual response.
But LLMs can be slow, and most users (like me) are too dopamine-fried to watch a spinning loading indicator while the matrix multiplies happen. So apps use streaming to send the results back to the user as soon as they’re available, from which we get a few requirements:
Of course, streams are useful for things beyond chat, especially in multi-agent coordination. But most of these systems really boil down to the same set of requirements: ordering, durability, and low latency.
Ordering is a hard nut to crack and inevitably involves tradeoffs. With Kafka, only a single partition can guarantee ordering, even within the same topic, so chats either share a partition keyed by chat ID, leaving readers to filter out the others, or each get their own, which doesn’t scale to many chats. With s2.dev, message order is guaranteed only when using their Sessions abstraction, which uses a shared HTTP/2 connection to guarantee the ordering matches the client’s side.
Even with sessions, let’s say a network blip drops the connection after a message is stored, but before the client hears back. Should the client retry, and risk writing the message twice? Or should it give up, and risk losing it? S2’s default retry policy allows duplicates, and avoiding them takes conditional appends or deduplicating them later.
In our implementation, the Hatchet engine is responsible for enforcing strict ordering guarantees per topic, with tunable batch timeouts on the client side to tune throughput vs latency. The ordering is enforced by checking that each incoming message is exactly one past the producer’s last stored sequence number.
With those guarantees in mind, we chose to use Postgres as the persistence layer for these messages. Our self-hosted users already have Postgres, and we have loads of experience with it. Hatchet Cloud is a little more bespoke and artisanal, and probably worthy of its own blog post in the future, but the core of the feature is still Postgres.
IDs are assigned at insert, but rows only become visible at commit. Tx 2 commits first, the reader's cursor jumps to 42, and row 41 shows up behind it, never to be read.
There are some tricks that can get around this issue by storing the transaction ID of every committed message, then filtering returned messages by the minimum active transaction ID. This is a nice trick, and I wanted it to work, but it has some scalability downsides. Long-running transactions in the database can indefinitely block reading these messages by keeping the minimum active transaction ID stuck, increasing latency. In a system like Hatchet’s, where Postgres is the core DB, that is a real risk.
Instead, we designed it by using a pessimistic locking setup that 1. reserves message IDs in batches and 2. commits those messages in the same transaction. The reservation happens on the v1_stream_topic table—incrementing the last_offset field and holding a row-level lock on that table. The messages are then inserted into v1_stream_message, safe in the knowledge that no other process can steal the lock on the topic.
This does two things: allows batching the messages, which increases throughput, and provides topic-level ordering consistency without having to wait for transactions from other topics before messages are visible.
Hatchet engines and the SDKs primarily communicate using gRPC. gRPC is great, with some frustrating limitations. One of those is the configurable message size limit. It defaults to 4MB, and can be set higher, but must be set to compatible values for both sides of the channel. This is possible for self-hosted Hatchet, but difficult for our cloud deployments. Setting the limit too high also incurs performance costs, like increased latency.
However, chat sessions may need to send messages larger than 4MB (for example, a generated image, or very large responses). To work around this, we created a side-channel for sending messages where the SDK will publish the large payload to a REST endpoint prior to sending a message over gRPC (with a reference to the large payload).
This made performance for the most common case quite good, while still supporting edge cases. This is a similar approach to that taken by s2.dev, but using Postgres rather than object storage.
Anything over gRPC's 4 MB limit is uploaded over REST first. The ordered gRPC stream only carries a small ref, which the client resolves when it arrives.
As has been mentioned throughout this article, one of the things we were aiming for was low latency. The primary requirement is that users don’t want to wait.
But because publishes are serialized per topic, the latency of each individual publish impacts the overall throughput of the system. We primarily optimized for this through the techniques elucidated above—using pessimistic locking rather than transaction ID tricks, as well as by thoughtfully partitioning and indexing the message tables. Messages need the lowest latency when they are being read real-time, closing the gap between when the workflow publishes a message and when the user sees it. However, messages that are older can be read in batches, reducing the per-message time cost. We partitioned the messages table by hour, with the primary key including the topic name, namespace, and tenant ID.
We’ve written quite a bit more about optimizing throughput versus latency in our blog post the fastest Postgres inserts.
This post covered some of the more interesting design decisions that went into the backend of this feature, but the goal is to make something that is incredibly simple and delightful to use. To show how simple it is, we created a demo chat application using durable streams and Hatchet workflows. All of the same benefits you get from using Hatchet apply here—simple scalability, concurrency management, and observability.
We’ll let you be the judge of how delightful it is.
This exact quote is apocryphal, but we conjecture that he probably did say this during his final moments experiencing dung-induced delirium trying to cure his fluid retention. ↩
Julius is a founding engineer at Hatchet, where he works on optimizing engine internals. He previously worked at Obsidian Security and MongoDB.
Distributed systems, workflow engines and developer tools, when we publish.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。