Under the hood, off the hot path: How D1 scales to millions of databases

Traditional database architecture assumes you have one or two big database clusters per application, usually accompanied by a connection pooling solution. AI-assisted development and vibe coding platforms turn this model inside out and are driving applications toward many small databases: one per user, application, preview environment, or tenant. That model fits D1 particularly well. D1 is Cloudflare's managed, serverless database based on SQLite. It allows you to create millions of databases and only pay for what you use. The question becomes: how do you scale the control plane for millions of databases without making every query slower by sending it through centralized bottlenecks? We made a lot of architectural changes to scale out D1 to support platforms with millions of created databases, hundreds of thousands of which are active at any point. This post is a behind-the-scenes deep dive into how the D1 control plane works. Many of the architectural patterns we will describe can be used by anyone building on top of Cloudflare’s Developer Platform. The post also gives a glimpse into how we build our own control planes. Growth in numbers There was already a steady growth in D1 usage when we started planning these changes, but we knew we would need to support a much bigger scale once large platforms started building with D1. Here is how things have played out so far. D1 has passed 5.5 million databases and has seen a 15x growth in the number of live non-deleted databases over the past one year, and a 65x growth over the past 2 years (as of October 8, 2026). We have also seen 18x growth in the number of daily active databases over the past year (as of October 8, 2026). A database is considered active if it executed at least one query during that day. Established and successful platforms are already building with D1, using the same tools available to all Cloudflare users. "ChatGPT Sites lets anyone turn ideas into shareable experiences. D1 gives our users a simple, scalable way to add features like custom metrics, leaderboards, and comments to their dashboards, games, and tools."

• Jon Abrams, ChatGPT Sites @ OpenAI While we were making the changes described in this blog post, we periodically load-tested against a target well above our expected growth. This guided where to focus our optimization efforts. Guiding engineering principles Applications built on the Cloudflare Developer Platform range from full-stack applications to service APIs, from internal single-user portals to massive multi-tenant vibe coding platforms, and everything in between. While building out the D1 control plane to support millions of databases, the following tenets and principles were guiding our architecture decisions and are useful for anyone building their control and data planes on our platform.
• Design all processes with failure and idempotency in mind. Failures will happen, so the system needs to self-correct where possible and continue making progress.
• Avoid centralized single-region components in any hot path for data plane operations (in our case all query requests).
• Use Workers KV, a globally distributed KV store, for critical routing metadata so that cache misses only rarely need to query central databases.
• Use Hyperdrive to access central Postgres databases for efficient connection pooling and optimized routing and caching, using read replicas wherever possible.
• Periodically precompute aggregations in the database to avoid scan queries during user request processing.
• Use eventual consistency and queuing between components that don’t need strong consistency to increase reliability and availability.
• Load-test the application with realistic workloads to discover bottlenecks and inefficiencies, and focus on optimizing those first. There are more engineering principles we follow internally to build our systems, but the above are generic enough and apply to anyone building on Cloudflare. The following sections will dive into the main system components and processes of the D1 control plane regarding the lifecycle of a D1 database, and showcase how to put these principles into practice with a real-world system. High-level architecture Let’s start off by introducing the main components of D1. There are a few more arrows and boxes missing from the diagram for secondary processes supporting D1, but these are the ones providing D1’s public feature set. We will explore the rest in dedicated sections in the rest of the blog post. First of all, each D1 database is backed by one SQLite Durable Object, which makes it very easy to create millions of them. The “HTTP API” shape represents all our API surfaces including the Cloudflare dashboard, Wrangler, and Cloudflare CLI. All API requests go through Cloudflare’s edge gateway and are served by the D1 Edge API Worker. The diagram shows the HTTP endpoints that allow users to execute SQL queries against their databases as the “Query endpoints”. The remaining endpoints include the usual database operations like creating, deleting, and listing databases, along with import/export and time travel features. These are shown as the “CRUD endpoints”. The CRUD endpoints use Hyperdrive to query a Postgres cluster in our central data centers (referred to as “Core”). There are two configurations. One configuration has caching enabled and targets the Postgres read replicas. The other configuration has caching disabled and targets the Postgres primary. The Edge API Worker might invoke the D1 Eyeball Worker (depending on the type of request). The Eyeball Worker is our proxy for all requests that need to be handled by the underlying D1 Durable Objects (DO). This allows us to put all query handling inside the Eyeball Worker, including observability, automatic retries, error handling, and other query-specific behaviors. Workers KV acts as our caching layer at the edge for read-heavy metadata and can be read by all the D1 Workers. The “Worker Binding” shape represents all users’ Workers with a D1 Worker binding. Requests from the D1 Binding go directly to the D1 Eyeball Worker, which in turn forwards the request to the Durable Object backing the database and proxies the response back to the User Worker. We have two methods for users to execute queries in their databases. One is over the HTTP API query endpoints, and the other is through a Worker they deploy in their accounts with a D1 Binding. In terms of operations, we fold each operation into one of the following categories:
• Data plane: any database query related request; should be as fast as possible and always at the edge.
• Control plane: any HTTP API operation or internal flows that run in response to a non-query user request; they can depend on central single-region components.
• Background asynchronous plane: any analytical and billing-related workflow; can be eventually consistent and runs anywhere. There is a lot more under the hood, so let’s dive into the key flows. Fast on-demand database creation close to the user Multi-tenant platforms and AI agent-driven development require creating a lot of database instances in a fast, cost-efficient, and reliable way. The following is an example Wrangler command to create a D1 database: % time wrangler d1 create "demo-01" ✅ Successfully created DB 'demo-01' in region WEUR ... 0.56s user 0.13s system 42% cpu 1.614 total When a user creates a D1 database, several things need to happen before the database is ready to receive queries.
• Check that the account can create more databases against the free tier limit or its account limits.
• Generate a unique D1 database UUID, a unique Durable Object ID (newUniqueId()), and initialize the underlying Durable Object (DO) with the account and database information.
• Persist the newly created database’s information in Postgres (one row per database, including metadata and the mapping to the initialized Durable Object ID).
• Write the mapping from a database ID to the Durable Object ID in Workers KV (later used by the data plane and the query endpoints). Let’s examine the actual flow and how we recover from failures at each step. Once a database creation request comes in, our Edge API Worker handler is invoked in the nearest Cloudflare location at the edge. Edge API queries our Postgres read replicas over Hyperdrive with caching enabled (with cache TTL of a few seconds) to get the number of non-deleted databases for the given account. The Edge API Worker forwards control plane requests to data centers near our Postgres cluster for endpoints that do multiple round-trips to Postgres (step 2), and the “create database” endpoint is one of them. The reason is that a round-trip from afar can be hundreds of milliseconds, whereas being near Postgres can be a few milliseconds. For endpoints that do at most one Postgres query we don’t need to do this optimization since Hyperdrive already handles efficient routing through the Cloudflare network. Creating a D1 database without any options places the database near the request’s location by default. You can influence the location of the database by providing an explicit location hint or a jurisdiction. The location hints and jurisdiction options are identical to what Durable Objects support, so when these are provided they are just passed straight through to the underlying Durable Object initialization. With the default creation we want the D1 databases to be created near the original location of the request, close to the user. To achieve this, during step 4 of the diagram, we internally forward the request execution to a data center close to the original request’s location and invoke the underlying Durable Object from there. This ensures that the underlying Durable Object is placed near the user. You can implement similar routing techniques by using explicit Worker placement hints and at runtime invoking the right Worker via Service Bindings. The creation endpoint is a bit of a weird case in terms of placement because even though we query Postgres at least two times, we also do a long-distance call to the underlying Durable Objects, which differ per request, so running the request to either side of the world could be inefficient. That's why we leaned towards forwarding the initial request near Postgres. The next step is to generate the database UUID along with a new unique Durable Object ID using the newUniqueId() API and invoke the init(options) RPC function on the new Durable Object’s stub. The Durable Object stores the information in its durable storage so that every incoming request from that point on has all the metadata for the linked database. Once the Edge API Worker gets the initialization response, it persists the database metadata into our central Postgres primary database over Hyperdrive, one metadata row per database. The steps above are wrapped in a retry loop (steps 5-7 in the diagram). When the DO initialization or the write to Postgres fails we retry with different IDs (this is important, as we will see in the next section). The write to Postgres is the “creation checkpoint” where a database is considered created even if the final response fails to reach the user. The database from that point on is included in all the control plane operations and can be bound to a Worker to serve queries. At this point the D1 database is ready to receive queries. The final creation confirmation RPC to the Durable Object happens in a waitUntil() and is OK to fail. This call signals the DO that the database is now active, and the DO updates Workers KV with the database metadata so that any subsequent query can immediately read the information from Workers KV cache without needing to query Postgres. A common question is why can’t the creation flow run entirely inside a Postgres transaction to simplify the process? The answer is simply that it cannot scale, since it leads to connections being held while the Durable Object is being initialized, which could be on the other side of the world.