Netflix Conductor: The Next Chapter

Netflix Conductor : The Next Chapter by Aravindan Ramkumar on behalf of the Conductor team Conductor is the workflow orchestration engine Netflix uses to stitch micro-services into reliable, observable business processes. If you’ve followed Conductor from the outside, the last thing you heard was probably the note on its GitHub repository: Netflix discontinued maintenance of Conductor OSS to refocus its resources on the internal fork. That announcement marked the end of the open-source chapter, and from the outside it may have looked like the end of the story. It wasn’t. Internally, Conductor never slowed down. If anything, it grew faster, and the demands of that growth pushed us to rebuild it from the ground up. This post is the answer to the natural question that announcement leaves open: what happened to Conductor after that? A lot, it turns out. The clearest way to put it is in terms of growth. Not long ago, Conductor crossing a billion workflow executions in a single year felt like a milestone. Today it clears that bar roughly every quarter, and the monthly volume is still climbing. Sustaining that kind of growth meant rethinking nearly every layer of the system. The data plane it runs on was modernized piece by piece. The evaluation engine was rewritten. The way it allocates work, controls concurrency, and runs compute all changes. The through-line is simple: scale forced a rethink of the evaluation engine and the layers beneath it, and everything else followed. The rest of this post walks through that evolution: how the architecture changed, the performance and scale work at the heart of it, the leap forward in developer experience, and the newer capabilities for controlling how workflows and tasks execute. How Netflix uses Conductor When we last wrote about Conductor, it ran a few hundred workflow definitions for a few dozen teams. That footprint has grown by orders of magnitude. Conductor now backs roughly 200,000 workflow definitions owned by around 150 applications, spanning Content and Studio Engineering, Ads, Games, and more. The runtime side has grown to match. In a typical month it executes on the order of 420 million workflows, and on a given day it runs anywhere from 5 to 20 million, with teams adding hundreds of new workflow types every month. Conductor and the Media Production Suite One of the clearest ways to see what Conductor does is Netflix’s Media Production Suite (MPS). It’s a set of applications that move production data into the cloud. An average Netflix title generates around 200 terabytes of original camera files, and MPS is what gets that media ingested, validated, transcoded, and delivered to the right people around the world. The multi-step pipelines behind MPS’s workflows (Footage Ingest, Dailies, VFX Pulls, and Conform Pulls) run as Conductor workflows. A dedicated orchestration layer called Studio Orchestrator stitches together the platform services these pipelines depend on, such as asset management, media encoding, and storage, using Conductor under the hood to ensure reliable execution and the durability to recover from the inevitable failures of long-running, distributed work. Beyond the studio: Ads and Games Content engineering is where Conductor started, but it’s no longer the whole story. Two of Netflix’s newer businesses lean on it just as heavily. Netflix Ads has made Conductor a backbone of its programmatic pipelines. Creative ingestion, advertiser-certificate management, and privacy-preserving Data Clean Rooms all run as Conductor workflows. Netflix Games uses Conductor to orchestrate the path a game takes from a partner’s submission to the app stores. When a studio uploads a build, a Conductor pipeline ingests the binary, runs the required validations (for example, iOS privacy-manifest checks), and drives App Store publishing, coordinating steps that span multiple platform services and external stores with the same reliability and visibility the studio pipelines rely on. How the architecture evolved Our earlier post described a Conductor built on a particular set of building blocks: DynoQueues for queueing, Dynomite for metadata, Cassandra (partially) for execution data, and Elasticsearch for indexing. That stack carried Conductor a long way, but as adoption climbed, almost every layer of it was eventually rethought. The data plane Conductor runs on today looks very little like the earlier one. 1.0: the original stack, and where it strained In its first generation, Conductor leaned heavily on Dynomite (with DynoQueues as the queueing recipe layered on top) backed by Elasticsearch for indexing. It worked, but two classes of problems showed up as scale grew. The first was the datastore. Dynomite is an in-memory store: a clustering layer that adds sharding and replication on top of Redis. You scale it by adding memory and nodes, with each replica rack holding a full copy of the data, which suits a cache but not the durable record for millions of workflows a day. Dyno Queues, built on the same store, inherited those limits, and its zone-sharded design made task scheduling uneven. The second was the engine’s design: evaluation ran synchronously and mutated workflow state in several places, so competing updates could leave a workflow inconsistent with no automated way to repair it. The 2.x line was a sustained campaign to address these, and two replacements did most of the heavy lifting. 2.0: Cassandra as the backbone The most consequential change was making Cassandra the primary datastore for execution data, displacing Dynomite. Cassandra gave Conductor the elastic, horizontal scaling it needed, and removed the scaling ceiling and zone-affinity constraints that the old datastore imposed. Around the same time, large task inputs and outputs were offloaded to S3, keeping bulky data off the primary datastore and further easing the pressure on it. Timestone: a purpose-built queueing layer The second replacement was the queue itself. Timestone, Netflix’s high-throughput, low-latency priority queueing system, took over from DynoQueues. Beyond shedding the Dynomite dependency, it brought two capabilities that mattered for reliability: a delayed-queuing mechanism, and a background repair process that could detect and recover workflows. Timestone also turned out to be foundational for work that came later. The async evaluation queue and the queue-depth-driven dynamic thread allocation described later in this post both build on it. Alongside these, the 2.x series also decoupled indexing from the critical path, moving to a Kafka-buffered, independently scalable indexer writing to Elasticsearch. 3.0: consolidating the modern data plane By Conductor 3.0, these threads came together into a coherent platform. Dynomite was fully sunset. Timestone was the queue and Cassandra the datastore. A Kafka-fed indexer pushed to Elasticsearch, with Iceberg added for long-term storage. The release also shipped automated workflow repair and multi-cluster deployment, the last of which lets workflows run in fully isolated contexts (for example, a dedicated context for the Media Production Suite, kept separate from the shared default and scaled on its own). The net effect is that by 3.0 the foundational reliability and scaling problems of the early era were largely solved, and the data plane reached roughly the shape it has today. 1.0 to 3.0 architecture evolution Rebuilding the engine: Conductor 4.0 By 3.0, everything around the evaluation engine had been modernized, but the engine and its data model were largely the original, and at this scale that became the bottleneck. The engine evaluated a workflow by loading its entire state into memory to decide what to do next, which meant heavy memory pressure and slow evaluations as workflows grew toward tens of thousands of tasks. It also stored a workflow’s metadata and its task payloads in the same Cassandra partition, producing oversized “wide rows” and leaving room for race conditions when several updates to the same task competed. Evaluation overhead, vertical-scaling pressure, and occasional inconsistent state needed to be addressed. A partitioned task model The old data model co-located everything about a workflow in one place. The rewrite separates workflow metadata from task and user data and gives each task its own record. The engine no longer loads the whole workflow graph to make a decision. It evaluates against a lightweight blueprint of the workflow and loads only the specific task data a decision needs. The practical result is a dramatically smaller memory footprint per evaluation and far less work to advance a workflow by one step. The 3.0 engine topped out around 2,500 tasks per workflow. The rewritten engine scales to 30,000 tasks per workflow, more than a 10x increase. Evaluation got faster too: in production, the rewrite delivered roughly a 40% reduction in p99 workflow-evaluation latency. Metric graph showing a reduction in workflow-evaluation latency Making competing updates safe using partitions Separating the data is only half the story. The engine also has to make competing updates to the same task safe, because a task can be written from more than one place at once. Picture a long-running task whose worker sends periodic updates to keep it alive, each one marking the task in-progress, before finally reporting it COMPLETED. If a stray in-progress update arrives just after that completion, reordered by the network, it could quietly revert a COMPLETED task back to IN_PROGRESS. The first version leaned on Cassandra lightweight transactions (LWT) to serialize these writes, but mixing LWT with ordinary writes is a known Cassandra anti-pattern. The two use different timestamping mechanisms, and combining them breaks the strict consistency Conductor needs. So the data model evolved away from locking entirely. Task state is split across two partitions: a pending partition for tasks still in flight, and a terminal partition that a task is written to exactly once, when it reaches a terminal state (COMPLETED, FAILED, etc). When the engine evaluates a workflow, it reads both and reconciles them in the application layer with one simple rule: a terminal state always wins over a non-terminal one, and between two writes of the same kind (two in-progress, or two terminal) ordinary last-write-wins applies. So in the race above, when the task is marked COMPLETED it lands in the terminal partition, and the late in-progress update loses on read. Race condition illustration The net of the rewrite is reliable task lifecycle management, a lower memory footprint, faster evaluation, and far higher concurrent throughput. Conductor 4.0 — Architecture Removing the last synchronous bottleneck: async workflow evaluation A faster engine still left one structural problem. Workflow evaluation ran synchronously. When an update arrived, a new workflow or a task completing, the request itself evaluated the workflow, and to do that safely it had to take a lock on that workflow. Under load, concurrent updates to the same workflow raced for that lock. Only one won, and the rest were deferred to a background repair process that, in extreme cases, could take several minutes to catch up. The result was exactly the kind of unpredictable tail latency that hurts most under peak load. The fix was to decouple the trigger from the evaluation. Updates no longer evaluate inline. Instead they enqueue the workflow onto a Timestone exclusive queue, and a central asynchronous evaluator processes each workflow’s pending work in order. Because the queue is exclusive, only one worker ever evaluates a given workflow at a time, so the lock contention simply goes away. The results are most dramatic at the tail (p99 and p99.9), which is the point, since the tail is what users feel under load: Unsuccessful lock-acquire attempts, which had spiked to around 2,700 per interval under contention, drop to essentially zero with async evaluation. Unsuccessful l