Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

How we optimized Azure Cosmos DB in our application storage platform, Habitat, to manage unprecedented growth.

By Jon Lee, Hrachya Hakobyan, and Chaomin Yu, Members of Technical Staff

Share

Making multi-tenancy reliable at scaleMaking multi-tenancy reliable at scale

A simplified primer on Azure Cosmos DB containers

Phase I: Relying on Azure Cosmos DB primitivesLogical partition key limits

Noisy neighbors

Weak partitions

Phase II: Trying workload isolation and single-tenancyLack of stable aggregate growth

Capacity planning

Phase III: Using multi-tenancy by default, with data movement capabilityHot key throttling

Quality of Service (QoS)

Serverless mode

Better reliability as co-managed services

Improving performance at a global scaleStore data near users when capacity allows

Broadly replicate only the smallest and most latency-sensitive datasets

Rely on integrated caching for everything else

Reduce proxy overhead with Rust

Scaling Change Data Capture

Looking ahead

Making multi-tenancy reliable at scale

A simplified primer on Azure Cosmos DB containers

Phase I: Relying on Azure Cosmos DB primitivesLogical partition key limits

Noisy neighbors

Weak partitions

Phase II: Trying workload isolation and single-tenancyLack of stable aggregate growth

Capacity planning

Phase III: Using multi-tenancy by default, with data movement capabilityHot key throttling

Quality of Service (QoS)

Serverless mode

Better reliability as co-managed services

Improving performance at a global scaleStore data near users when capacity allows

Broadly replicate only the smallest and most latency-sensitive datasets

Rely on integrated caching for everything else

Reduce proxy overhead with Rust

Scaling Change Data Capture

Looking ahead

All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.

This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.

In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

Habitat is the online storage platform we built so OpenAI products can quickly and reliably access needed information.

Play

Request

Response

Changes (CDC)

Clients

ChatGPT

API

Codex

Internal services

And more

Habitat

Azure Cosmos DBOnline storage

NanobaseOnline storage

ValkeyCaches

Blob storageStorage resources

CDC ServicesChange Data Capture

Databricks

Rockset

Kafka

And more

Making multi-tenancy reliable at scale

As companies scale, infra teams often evolve from running single-tenant systems to multi-tenant ones. Single-tenancy, where every product surface team maintains their own system, reduces blast radius and provides easy ownership boundaries and workload isolation. As teams and product surfaces grow, this becomes expensive to run. Single-tenancy makes every product team responsible for capacity planning and often results in significant over-provisioning.

Conversely, multi-tenancy assigns control to a centralized database team, with better knowledge and experience for provisioning resources. But multi-tenancy creates its own problems, such as noisy-neighbor issues, where one tenant consumes an unusually large share of the resources to the detriment of the other tenants.

In Habitat, we had to weigh all these tradeoffs to design a reliable, efficient, storage system at scale. Today, Habitat is a multi-tenant database layer for many OpenAI products, but we went through a circular journey from using one multi-tenant Azure Cosmos DB container to provisioning single-tenant backing containers to reverting to a multi-tenant container by default. We’re still very early in our multi-tenancy story on Habitat: we support a large set of use cases, but we’ve been able to defer investment in this area by leveraging Azure Cosmos DB’s built-in primitives wherever possible.

Before walking through our Azure Cosmos DB journey, some basic context:

An Azure Cosmos DB account defines a set of containers and a replication policy

Each container is provisioned for a certain amount of throughput in request units per second (RU/s)

Data within each container is stored in a typical shard-by-specified-logical-partition-key manner with documented limits on storage and throughput per logical partition key

Azure Cosmos DB handles sharding logical partition keys across fewer backing physical partitions, splitting them into multiple physical partitions when storage exceeds a threshold or when the container requires more physical partitions to sustain the provisioned throughput

We originally allowed any client load through, relying on the stated 10K RU/s limit per logical partition key to provide back pressure. When we first set out, we also used only one default Azure Cosmos DB container in each account. This worked well enough, but proved problematic for a few reasons:

Due to early API choices, we exposed logical partition key selection directly to clients. As a result, we have frequent collisions on commonly used partition keys, such as user_id or workspace_id, across different product use cases. Because these use cases shared Azure Cosmos DB limits on the same logical partition keys, the design constrained overall throughput for a given user.

One common problem we faced in scaling Postgres before we moved to Habitat and Azure Cosmos DB was the ability for many requests from a very hot organization or hot key—one tenant generating a disproportionately high load—to take out the whole database. Significant collective effort was spent to cache those keys and build leases into cache misses so that the resulting cache miss load does not stomp on the underlying database.

Figure 02a · Logical partition key limits

Play

Cosmos DB containerPhysical partitionPK: user-id-28104CapacityMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedMemory / dreaming : capacity usedConversationsSuccess rateResponses returned in this animation loop—AuthSuccess rateResponses returned in this animation loop—Memory / dreamingSuccess rateResponses returned in this animation loop—

Noisy neighbors

While documented data size and throughput limits apply to logical partition keys, the physical partition is actually where limits are enforced. In practice, this means logical partitions are unable to reach 10,000 RU/s. Instead, they share the physical partition’s limits with every other logical partition key colocated on that physical partition.

This allowed noisy neighbors to impact many unrelated logical partition keys, with workloads operating on data for niche product features disrupting core login and signup flows.

Play

Cosmos DB containerPhysical partitionPKs: user-id-28104, user-id-10384Conversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversations user-id-28104: capacity usedConversationsuser-id-28104Success rateResponses returned in this animation loop—Conversationsuser-id-10384Success rateResponses returned in this animation loop—

Weak partitions

The throughput limit for a given physical partition is also variable, dependent on the total throughput assigned to the Azure Cosmos DB container. The container’s provisioned throughput is distributed roughly evenly across the physical partitions for that container.

As data volume grows, physical partitions split, increasing the total partition count while total provisioned throughput remains unchanged. The result is that the effective throughput limit per partition shrinks as data size grows.

For our most storage-heavy, blob-like use cases, this creates a failure mode where storage growth alone reduces throughput capacity for active workloads even while request rates and usage remain stable.

A partition splits when it reaches its storage limit. Each child receives half the throughput capacity.

After split: 5K RU/s capacity × 2 partitions

Separate containers help partially solve these partition key limits and noisy neighbors. By using separate containers, each can be assigned its own independent physical partitions and only the varying workloads on that container itself could be noisy neighbors or overlap in partition key space. On the downside, there is non-zero overhead in provisioning new containers—each container requires individualized capacity planning—so we want to create containers only at a service or use case boundary that makes sense.

We began to use separate Azure Cosmos DB containers for the most critical new use cases requiring strong isolation. Combined with a script for managing weak partitions by synchronizing the total throughput provisioned for each container with its number of physical partitions, we mostly avoided all the problems we were facing.

However, since we started with a default container, many legacy critical use cases were unfortunately stuck there without a data migration. To protect critical services that still relied on data in the shared default container, we often advised teams to create separate containers for new services or major product features. As we scaled, pressure on the default container grew and production incidents also grew. We began to provision containers for any new service or large product feature just to protect the default container from more load.

This saved us from many site-wide outages, limiting the blast radius to feature or service degradation outages. However, the overhead of provisioning new containers eventually caught up to us:

In the old shared default container, any particular new use case going from 0 QPS to 1 million QPS was unlikely to significantly shift the total aggregate usage. On an isolated container however, a new use case going from 0 to 1 million QPS might represent 1,000x more load than the baseline attributed to only internal dogfooding.

With dedicated containers, teams now needed to remember to pre-provision containers to the expected level of throughput for their launches. Product teams would often ship their features to production and forget to pre-scale and we’d sit in SEV rooms on launch day attempting to scale up containers last minute.

This is a core workload mixing benefit of multi-tenant systems that we lose in favor of better isolation with this container strategy.

Figure 03 · Lack of stable aggregate growth

1K RU/s during load testing → 1M RU/s at launch

Over time, monitoring usage of dedicated containers also requi