We spent $15k and benchmarked 5 DOCX-editing approaches on Harvey LAB

Oct 24, 2026Research

Benchmarking DOCX MCPs on Harvey LAB

We ran 4 DOCX MCPs and a python-docx baseline on 180 Harvey LAB legal tasks. The baseline scored highest, was cheapest, and was the only arm to finish every task. We didn't include Vespper here, but we'll share its results in a follow-up post.

Listen to article11:30

Share

On this pageIntroduction

Background

Experiment Setup

A few more technical details

Data

Results

Overview

Completion rate

Editing-phase metrics

Cheating ratio

Baseline internal turns

Caveats

Conclusions

On this page

Introduction

Background

Experiment Setup

A few more technical details

Data

Results

Overview

Completion rate

Editing-phase metrics

Cheating ratio

Baseline internal turns

Caveats

Conclusions

Introduction

Word documents are everywhere. In domains such as legal, finance, and healthcare, the Word document is the deliverable. Contracts, regulatory submissions, and audit reports get drafted, redlined, and signed in .docx, and companies usually have libraries of Word templates they work with on a regular basis.

That work is increasingly shifting to agents. Microsoft Copilot and Claude in Word have made in-document AI mainstream, while a growing number of vertical agents; particularly in legal tech—need to interact with .docx files.

In this post, we wanted to see where things stand today when it comes to editing .docx files in complex tasks. To find out, we used Harvey LAB, Harvey's open-source Legal Agent Benchmark. It's built from real client matters, and each task asks an agent to work through the case materials and produce a finished legal deliverable, which then gets graded against expert-written rubrics.

As of this writing, it contains 2,010 tasks across 27 practice areas. We measured pass rate and all-pass rate, along with efficiency metrics like latency, cost, and token count.

In general, there exist several ways for agents to edit Word docs today, which mainly fall into three categories:

In this experiment, we chose to evaluate the first two options (libraries and MCPs). We skipped the third one because a DOCX → Markdown → DOCX round trip loses the styles and formatting of existing Word documents.

We picked 5 tools, which serve as our experiment arms.

Each arm uses Harvey's agent harness and sandbox, with grok-4.7 as the agent model (high reasoning). The agent runs inside the sandbox (via Podman) and has access to bash, read, write, edit, glob, and grep. In the DOCX MCP arms, it also gets the tools the MCP exposes, and we unload the DOCX skill and remove some dependencies (more on that below).

As we mentioned above, Harvey LAB contains 2,010 tasks across 27 practice areas. Not all of them require a .docx deliverable, since some only ask for an .xlsx or a .pptx. Our focus was also on editing rather than writing from scratch, so we used Jev to classify tasks that involve editing an existing .docx file. That left us with 360 tasks. To keep costs down, we ran the experiments on a stratified sample of 180 of them.

Each task (like this one) comes with a rubric, a list of criteria that an LLM judge checks one by one against the task's deliverables (e.g. "nda.docx"). Each criterion is binary, PASS or FAIL. The main metric is the all-pass rate, where a task only counts as a success if every criterion passes. We also report the mean rubric score, which shows how close an agent got when it didn't fully pass.

Our judge was grok-4.7 with high reasoning effort. Across all 2,010 tasks, the p50 number of criteria per task is 54, the p90 is 83, and the max is 1,114.

The following graphs show the main metrics: mean rubric score, all-pass rate, cost per task, latency per task, and total cost:

Mean rubric score and all-pass rate, total cost, and per-task cost and latency across the five arms (n=180 each).

A few highlights from these graphs.

The following plots also show input and output token consumption:

Input tokens (including cache) and output tokens per task. The baseline uses roughly half the input tokens of the MCP arms.

What we found interesting is that the tools that are supposed to reduce the agent's "cognitive" load actually consume more tokens overall, and by a wide margin compared to just letting the agent use python-docx and LibreOffice (even when the agent is outputting more tokens).

Like we said earlier, some arms didn't manage to finish all their tasks. SuperDoc, for example, hit a lot of context window errors caused by superdoc_inspect tool calls that returned big responses. Harvey's harness doesn't have compaction or tool result trimming, and we didn't add any, since we wanted each MCP to handle its own responses without us introducing bias. The following graph shows the completion rate across the experiment arms.

Share of tasks where the agent called finish and left a .docx in output/.

As you can see, the baseline was the only arm to finish all 180 tasks, while the MCP arms ranged from 61% (SuperDoc) to 94% (Adeu).

Now, we want to take a deeper look into the "editing" phase. Namely, during Harvey's tasks, the agent usually spends the first part of the task just exploring the materials (each Harvey task comes like this one with an "input/" folder that includes files the agent needs to read in order to complete the task). After this first part, the agent usually starts creating the deliverables. We wanted to see how hard it is for the agent to create it. Like we said before, in this experiment, we focused only on .docx deliverables (we left .md, .xlsx, .pptx and other file formats).

Editing-phase metrics over the 77 tasks every arm completed.

Some highlights here:

We want to highlight that last point because it's counterintuitive. You'd expect the legal work itself to be the heavy part, but in the Harvey tasks that we chose, editing dominates across every dimension (latency, cost, and tokens). In other words, dealing with the .docx itself (paragraphs, tables, lists, formatting, etc.) takes a real toll on the agent. Even the baseline's 6M input tokens is not a trivial amount, especially for users on smaller Anthropic/OpenAI subscriptions.

In the DOCX MCP arms, a call counts as cheating when the agent touches a .docx file with its built-in tools instead of the MCP. A few examples of what counts:

File operations (ls, cp, mv, etc.) and using the MCP's own CLI (like in OfficeCLI's case) don't count.

We first tried to detect cheating with deterministic logic, but it was too noisy, since cheating can take a lot of shapes. So we used Jev again to go over each tool call along with its surrounding context and classify whether it's a cheat.

The cheating ratio is the number of cheating calls divided by all .docx calls (cheating + MCP calls).

We see cheating as a bad signal for the tool. It means the agent found the tool too rigid and looked for other ways to get the job done.

Here are the results across the four DOCX MCP arms.

Cheating ratio per task (p50 / p90) and over all tasks, for the four DOCX MCP arms.

We see that cheating was not a common thing, which is reflected in a very low/zero median across the 4 arms. Our prompt instructions, plus the fact we removed python-docx + docxtpl, helped keep the model in check and make it mostly use the DOCX MCP.

We wanted to dig a bit deeper into what's going on in the baseline experiment. We've spoken to a lot of legaltech companies, and this seems like the most popular approach out there, i.e. building a custom harness with a sandbox, python-docx, and LibreOffice. Some even take a very opinionated harness e.g Claude Code and just tune its prompt and tools.

Specifically, we wanted to know a few things:

Here are the numbers from the baseline experiment.

Python programs per task and median lines of code per program in the baseline arm.

A few things stand out here.

We also wanted to dig into the errors these scripts hit and what kinds of errors they are:

Python failures by error type, failures per task, and output tokens per Python tool call in the baseline arm.

We see here that most tasks didn't experience errors (0 errors as the median), and also their output is quite short. Meaning, the scripts' output is dense and informative.

In this post, we compared 5 approaches to editing Word documents on Harvey LAB. Overall, the baseline (python-docx and LibreOffice) had the best mean rubric score and came second on all-pass rate, right behind OfficeCLI. It was also the cheapest arm, the only one that finished all 180 tasks, and had the cheapest and fastest editing phase. Looking inside it, the agent writes around 25 Python scripts per task, mostly short exploratory ones, and rarely hits errors.

That's a bit counterintuitive, since MCPs are supposed to take load off the agent. In practice, they consumed more tokens, and some of them ran out of context before finishing. Cheating was rare, so these numbers reflect the MCPs themselves rather than the agent going around them.

We also showed that editing .docx is still a token-hungry task for agents on an absolute level. Editing accounted for 85% to 95% of the input tokens in every arm, and even the baseline's editing phase had a median of 6M input tokens and $5.85 per task.

We'd love to hear what others are seeing. If you're building agents that edit .docx files, which approach are you using, and does this match your experience?

Today we're launching Vespper DOCX MCP: the first model fine-tuned specifically for editing Word documents, shipped as an MCP. On our internal benchmark, it allows agents to be 3× faster, 2× cheaper, and more accurate than the closest alternative.