Running Agent Factory for Months

Artificial Intelligence

Software Engineering

Automation

Herdr

Who Watches the Agents? What Four Months of Building an Agent Factory Taught Us

Suguna Paulraj

5 min read·

9 hours ago

--

Making work faster with AI is the easy level. The hard one is building the agents that do the work, and then knowing they did.

Your clients are asking where the AI gain is. Your developers have the licences, and say they’re faster. But delivery looks about the same, the pull requests got bigger, and your senior people are reviewing late into the evening.

Here is a question worth asking on Monday: which of your AI agents did real, finished work last week, and which only said they did? Most leaders can’t answer it without someone opening every agent and reading. With twenty agents, nobody does.

We have been building an agent factory (many AI agents doing real work in parallel, on their own schedules) for about four months. Most of Deemwar’s work now goes through it: code, leads, social media, short videos. We’ve tried several setups along the way. Some seem to work. We are still finding out. Others are trying their own versions too: Gas Town, OpenAI’s Symphony, OpenClaw, the Ralph loop, or simply many terminals.

Most teams are at step 3: many agents, each in its own terminal, waiting to be asked.

Most teams we see are on step two or three: an agent here, a few terminals there, with a senior person checking each one. That is where the late evenings come from. Real factories, with schedules and an orchestrator handing out the work, are still mostly on paper. And the ones that get there hit the crack in step five: the work goes out, and nobody reports back.

We hit it too. At 22:13 on October 2, one of our coding agents stopped on a pop-up asking whether to enable a setting. Nothing crashed. Eleven and a half hours later it was still waiting, looking exactly like an agent that was busy.

It wasn’t the only one. The same week, our social media agent’s records showed three posts with no links. By its own report it was working. By the evidence, nothing had gone out.

Swap “agent” for “team member” and every engineering leader has seen this: work that looks in progress, a status that says done, and a client who finds out first. Agents just do it faster and in larger numbers. We had a factory and an orchestrator. Nobody was looking.

The reason, as we see it: AI multiplied the work one person can get done. It didn’t multiply the decisions one person can make well. The work keeps coming, the person at the top can’t look at everything, and the agents don’t ask.

Everything runs on herdr, on our laptops and on servers over SSH. This is the sidebar on one of our machines: every project and every agent has its own row and a status light.

Terminal window running herdr. The left sidebar lists projects and agents, one per row

The orchestrator, our CEO agent, splits each goal and sends the pieces straight to the agents. Agents message each other directly, and every one of them checks with a judge, jevx, before calling something done. Agents from different vendors can share it. In one test, a Claude agent split a small job between Codex, OpenCode and Devin agents and reported back:

One herdr workspace, five panes: a Claude agent leading, Codex writing the code, OpenCode writing and running the tests, Devin reviewing, and jevx checking the result.

Press enter or click to view image in full size

The lead agent’s report: Codex wrote the code, OpenCode wrote and ran six passing tests, Devin reviewed it.

For most of those four months, that was the whole factory. What it lacked was someone walking the floor. We added that only two and a half weeks ago: foreman, a micro-manager agent. It takes no orders and assigns no work. It does one round at a time:

Its very first round found the agent stuck on the pop-up, the posts with no links, and, within the hour, a research agent whose score fell from 45 to 15 in two rounds. None of those would have set off an alarm.

Foreman’s first round, October 3: every verdict comes with its evidence.

It’s early. So far we’d keep a micro-manager separate from the orchestrator, proof of work defined for each agent, and a written rule after every mistake. This is what works for us with Claude Code and Codex agents in herdr; your stack may need something different.

Step back and there are two levels to any AI transformation. The first is making the work faster: the same people, the same tasks, with AI helping. Everyone can do that, and most teams already are. The second is harder: building the scripts, tools and agents that do the work you were doing a month or a year ago, so you build each one once and never do that work by hand again. That second level is how an agent factory gets built. And the moment you have one, someone has to watch it.

So, which level is your team on? Where do you stand on that ladder: an agent here and a few terminals there, or something that keeps working while you sleep? And when the next twenty agents arrive, who will notice the one that quietly stopped?

Planning an AI transformation? Connect with us for an assessment: deemwar.com/consulting