Knowledge graphs could help orgs respond to new laws

Our first benchmarkfirst benchmark showed that a knowledge graph of a firm lets an agent answer firm-wide questions faster at the same level of accuracy. To forecast new regulations, a firm has to find every agreement, client and document that will be impacted. That means joining long-horizon connections through a large system, in our case, four examples from 120-120k matters.

We generated one fictional firm at four sizes and wrote a hypothetical regulation that changes the payment terms allowed with certain counterparties. The task is to find every live agreement that must be amended. No single document answers that: whether an agreement is a target depends on who its parties are, what terms it sets (sometimes only by pointing to a firm precedent) and whether its matter is still open. At 12,000 matters there are 562 targets among 24,315 documents.

Without the graph our agent could list, search and read the firm’s files; with it, it could also query a graph built from the same files. Each setup answered three times, and every answer was scored against the list of targets the firm was generated from.

Reading one file at a time is the weakest way to search without a graph. The point of a production-grade information retrieval exercise is to compare against the best alternatives, so at 12,000 matters we also split the firm across 12 copies of our agent and ran two coding agents, OpenAI’s Codex and Anthropic’s Claude Code, over the same files, once limited to search and once free to write their own scripts to customise their strategy for the given task.

Splitting found 96%, but read 35 times as many tokens as the graph, and three of its 36 agents missed a whole class of agreements without saying so. Both coding agents found every agreement, using search patterns or a short script to pull the parties and terms out of every file at once. The graph was still the fastest, reading a third of the tokens or fewer, because the facts the other approaches work out when the question is asked were worked out once, when the graph was built.

At 120,000 matters there are 5,274 targets among 243,547 documents. The graph would not fit in our machine’s memory, so we built it in ten partitions, as a graph database does, and merged each query’s answers. Every setup ran with no limit on model calls, time or spending.

Limited to search, both coding agents gave up on their own after five to ten minutes, having listed fewer than a dozen agreements. Writing scripts, both found every one: Codex in four and a half minutes, Claude Code in 28, because it chose to open all 243,547 files one at a time.

Our agent with the graph had the right total from its first query. But as the product stands, a graph query hands its rows back to the model as text, 70 to a page, and every run stopped after the first page. It gave the total and said its list was incomplete rather than copy 76 pages into a reply. No limit stopped it; it chose to.

The graph’s times in this post leave out loading the graph from disk. A graph database loads once and stays ready, but our test setup could not hold the largest graph in memory and reloaded it for every query, which took up most of each run’s time.

Codex kept pace at both sizes because its approach barely grows with the firm either. Each time, it read the firm’s document index, which already records every document’s type and matter, kept the agreements on open matters and ran a short script over just those files. Its tokens went on learning the firm’s layout and writing the script, not on the list itself, so it read about the same at 12,000 and 120,000 matters. That index is a small, flat version of the graph. Without it a coding agent is left opening every file, as Claude Code did.

A graph’s cost does not grow in step with the firm: the firm grew a thousandfold, from 120 to 120,000 matters, but the tokens our agent read with the graph grew at most fourfold, because the joining is done once, when the graph is built. The hard part is getting the knowledge in. Much of it lives in people’s heads, so a graph has to start from what the firm already records, such as each matter’s counterparty and their documents, and be extended by rules and language models. That takes work, but once a firm has a graph it can keep extending, every later firm-wide question, from the next new law on, costs a fraction of what it would without.