Infinity: A runtime for extreme agent concurrency

Agents spend much of their time waiting on tools. A coding agent may start a build that runs for twenty minutes, a triage agent may wait hours for the next pull request on a repository, and an assistant may wait until the next morning for an approval. Most agent harnesses run the agent loop as a sequence of blocking steps: the harness calls the model, invokes a tool, waits for the tool to return, and repeats. While the agent waits, it holds a process and its memory, and it cannot respond to events that arrive in the meantime.

Today, we’re releasing Infinity under the Apache 2.0 open source license as part of the Hydro project, a collaborative effort between AWS and UC Berkeley. Infinity is a research project that re-imagines agent runtimes, based on the techniques that programming languages use to suspend and resume asynchronous functions. When an agent calls a tool, Infinity dispatches the call, saves the conversation, and suspends the agent. When the tool result arrives, the runtime resumes the agent from its saved state.

With Infinity, concurrent agents are extremely lightweight. For example, we can fit 75,000 concurrent agents in the 8 GB of a Raspberry Pi 5. Infinity also makes it possible to deploy concurrent agents on standard serverless platforms. When Infinity suspends an agent during a tool call, the serverless function can end. So even if a tool call takes days, nothing needs to run while the agent waits.

Infinity is an early research release, and we are sharing it to get early feedback. The rest of this post describes the ideas behind Infinity and its architecture. We start an overview of how languages suspend asynchronous functions with stackless coroutines, and why an agent can be suspended in the same way. We’ll explore how Infinity uses this to run many agents concurrently, even on serverless platforms. We then describe synthetic tool calls, which Infinity uses to deliver asynchronous events to existing models, and we use synthetic events for agent-to-agent communication. Finally, we discuss some open questions that we hope to explore further.

Many languages, including Rust, JavaScript, and Python, let developers write asynchronous logic with async and await. When a function awaits an async operation, such as a network request, the function is suspended, and the thread that was running it moves on to other work. When the operation completes, the runtime resumes the function from the point where it was suspended. To make this possible, the runtime has to save the function’s continuation, which is the state it needs in order to resume.

Languages save this state in one of two ways. Operating system threads and stackful coroutines keep the task’s entire call stack in memory while it waits. Stackless coroutines, which Rust uses to implement async fn, instead have the compiler turn each function into a state machine that holds only the relevant values at each await. This approach results in a smaller memory footprint, so an executor can efficiently run a large number of stackless coroutines.

An LLM agent can be suspended in the same way as a stackless coroutine. At the moment an agent calls a tool, its conversation history contains everything needed to continue the agent once the result is available. If the runtime persists this history, it can discard all in-memory state, and any process with access to the store can resume it later.

The Infinity Runtime uses this idea to suspend and resume long-running agents. It executes each agent as a sequence of bounded slices. In a slice, the runtime loads the agent’s history from a pluggable store, runs one model completion, persists the output, and dispatches at most one tool call. The slice then ends. Tool calls play the role of await points: a slice never blocks for a tool, a timer, or any other external operation.

Everything that can resume an agent arrives as a message on a shared input queue, including user text and tool results. Because the queue absorbs pending events, the runtime does not need to keep an agent alive while it waits. When the Infinity Runtime dequeues a message, it launches a slice that reloads the agent state and processes the message.

Since a suspended agent occupies no memory, the runtime can manage a large number of agents and schedule them as messages arrive. Agents run inside an agent system, which is a pool of agent threads that share stores, tools, and a model. The runtime organizes the agent threads in a way similar to an actor system. Each conversation thread has its own input queue, threads do not share mutable state, and threads communicate only by sending messages to each other’s queues. Each message carries a thread_id that identifies its destination thread. The runtime guarantees that messages for the same thread are processed in order, one slice at a time, while slices for different threads run in parallel.

Ordering within a thread is required for correctness, because two slices that read and write the same history at the same time would overwrite each other’s turns. Different threads have nothing to coordinate. In a long-running process, the agent system gives each active thread a channel and a lightweight task, and schedules the pool in the same way that an async executor schedules futures. When a thread has no queued messages and no pending tool results, its task exits, and the runtime starts a new one when the next message for that thread arrives.

Because a suspended agent consists only of its history, the memory needed to keep agents resident is close to the size of their histories. To measure this, we wrote a benchmark that launches waves of agents in one agent system, backed by a simulated model. Each agent runs twenty turns, and each turn makes a full round trip through the runtime: the model streams a completion of about 1.2 KB that calls a tool, the tool result returns through the input queue, and a second completion ends the turn.

We ran the benchmark with 80,000 agents, for a total of 3.2 million completions, on a single thread of an AMD EPYC 9R14 with in-memory stores. Resident memory grew linearly, at about 100 KB per idle agent. At this rate, about 75,000 agents fit in 8 GB, which is the memory of a Raspberry Pi 5. Most of this memory is taken up by the conversation histories, since an idle agent has no task, stack, or connection associated with it.

Let’s look at how this appears to a developer who embeds the runtime in local (in-memory) mode. The following program builds an agent system with one custom tool, launches one agent, and prints the events that the agent produces:

let system = AgentSystemBuilder::new_local( InMemoryConversationStore::new(), InMemoryStateStore::new(), StaticModel::new(provider, "global.anthropic.claude-sonnet-4-6").await?, ) .tool(Box::new(GetBuildStatus { builds })) .start();

let mut thread = system.thread_builder().launch().await; thread.send_user_text("Is build 42 green yet?").await?;

while let Some(event) = thread.recv().await { println!("{event:?}"); }

The difference from most agent frameworks is in how tools are written. Rather than returning its result, a tool’s execute method starts the work and returns immediately. When the work completes, the tool sends the result to the agent’s input queue:

#[async_trait] impl Tool<ChannelSender> for GetBuildStatus { // name(), description(), and parameters() describe the tool to the model.

async fn execute( &self, args: serde_json::Value, id: ToolCallId, call_id: Option<ProviderCallId>, context: &ToolContext<ChannelSender>, ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> { let build_id = parse_build_id(&args)?; let (builds, sender, thread_id) = (self.builds.clone(), context.message_sender.clone(), context.thread_id.clone());

tokio::spawn(async move { let text = match builds.status(&build_id).await { Ok(status) => format!("Build {build_id}: {status}"), Err(error) => format!("Error reading build {build_id}: {error}"), }; // The result returns to the agent through its input queue. let message = tool_result_message(thread_id, &id, call_id, text); if let Err(error) = sender.send_to_input_queue(message, id.as_str()).await { tracing::error!(%error, "failed to deliver build status"); } });

Ok(()) // The slice ends without waiting for the build. } }

When the model calls get_build_status, the runtime persists the turn containing the call, invokes execute, and ends the slice. If the status lookup takes a hundred milliseconds, the next slice begins almost immediately. If it takes three days, the agent occupies only its stored history in the meantime. The tool and agent code are the same in both cases.

The call to send_to_input_queue is the only point where the tool interacts with the runtime. InputSender is a trait, so the queue can be an in-memory channel in a long-running process, as in this example, or a durable message queue in the cloud, which we describe next.

Because a slice does not wait on external operations and loads all of its state from storage, the runtime can run on serverless platforms, where a function is invoked for each unit of work and may exit as soon as the work is done. The library supports this through step mode. A step-mode agent system keeps no tasks running. Instead, the platform calls step once for each batch of messages it delivers, and nothing is retained in memory between calls:

let mut system = AgentSystemBuilder::new(conversation_store, state_store, model, sqs_sender) .tools(tool_impls) .build();

// One call per queue delivery; the process may exit as soon as it returns. let collector = EventCollector::new(); let outcomes = system.step(inputs, &collector, &mut NoDeferral).await?;

Both modes run the same pipeline, and the local mode adds per-thread drivers that handle batching, interruption, and automatic compaction around it. Tools, model providers, and tests written against the core library therefore run in either mode without changes. The main difference between the two is the InputSender, which the runtime uses for anything that should happen later, such as a timer wake-up or a message for another thread. In step mode, the sender writes to a durable queue, and the message comes back as a later delivery.

Our reference deployment on AWS uses an SQS FIFO queue as the input queue, with one MessageGroupId per thread. Each delivery invokes an AWS Lambda function that runs one step and exits. Conversation histories are stored in Aurora DSQL and deduplication state in DynamoDB, and the sleep tools use SQS delivery delays and EventBridge Scheduler for their timers. The repository includes CDK constructs that provision these resources.

Running on such a platform places several requirements on the runtime:

Traditional runtimes that block on tool calls can also be hosted on a serverless platform, but each invocation must then stay open while its tools run. In Infinity, a waiting agent has no running invocation, so an agent waiting on a three-day CI pipeline uses no compute until the pipeline reports its result.

Agents that wait on tools also need a way to learn about events that happen while they wait. A long-running command may produce output over time, and a subscription to a repository should produce an event each time a pull request is opened. In both cases, a single tool call produces a sequence of results instead of one, and the runtime has to present each of them to the model in a form that the model can reason about.

Model APIs do not provide a way to represent an asynchronous event. A conversation consists of user turns, assistant turns, and pairs of tool calls and tool results, and each tool call must be answered by exactly one result. We considered several ways to fit events into this structure. Appending a second result to the original tool call violates the one-result rule, and models typically reject such a conversation. Adding the event as a user message is accepted by the API, but it attribute