Claude versus "GPU'less AI architecture" reasoning competition

We put 1,186 questions about a water-treatment plant and a set of logic puzzles to two “thinking machines”, SapienTronic AI (previously called Minotaur AI) and Claude.

SapienTronic AI got a perfect score in 28 seconds on a laptop, using no tokens. Claude got one answer wrong, was certain of it, and took 28 minutes and about half a million tokens. This post explains the contest, what each machine had to work out, and why one confident mistake in a thousand matters more than you might think.

In earlier posts we introduced SapienTronic AI (Introducing Minotaur), showed it reasoning about systems without a language model or a GPU (the deep dive), and showed it doing Aristotle’s logic by letting a system settle, with no rule of inference anywhere inside it (Logic Without Rules).

So what were the questions, how did the two engines tackle them, and what was it about Claude’s one mistake that made the difference?

We set both engines three kinds of problem (described below) and marked every answer against the correct one.

Getting answers right is only half the story, though. We also asked each engine to say how sure it was of every answer.

An honest “I’m not sure” is useful. It tells you to double-check that answer. The worst kind of mistake is a wrong answer given with confidence, because nothing warns you and you act on it. So alongside the scores, we counted those confident mistakes separately.

Let’s say we have a small drinking-water plant fed from a river. It has pumps, valves, filters, a chlorine feed, a standby generator, sensors and alarms (69 parts in all). The parts are connected, so what happens to one can ripple through to many more. The plant also has to cope with things it can’t control, like a power cut, a storm upstream or a river that has turned cloudy after rain.

We then ask the engines questions about it (the kind an engineer on duty might ask):

The questions didn’t come one at a time. They came in chains, each building on the answers before it (much as a real control system decides one thing and then acts on that decision). In the later rounds each engine had to carry its own chain for up to 200 steps, using its own earlier answers as facts. That makes the test unforgiving: a slip at step 3 doesn’t stay at step 3, because every later step that relies on it inherits the mistake.

The second scenario is a set of logic puzzles about small towns. Each town has three people. Every person may belong to any of three groups, which we’ll call a, b and c (think of them as, say, bakers, builders and clerks, where one person can be both a baker and a clerk). And one person can answer to another, the way an employee answers to a manager.

We’re given a few rules that are true of the town. For example:

Then we’re asked a question about the town, and the answer is yes only if it holds in every town that obeys the rules. Two questions show how this works.

Is every a also a c? Yes. Take any a. By rule 1 they answer to some b, and by rule 2, anyone who answers to a b is a c. So every a is a c, in any town that obeys the rules.

Is every b also a c? No. Take a town where Ann is an a and a c, Ben is a b, and Ann answers to Ben (the third person, Cal, belongs to no group and answers to nobody). Ann answers to a b, so rule 1 holds, and Ann is a c, so rule 2 holds too. But Ben is a b who isn’t a c, so the claim fails.

A town like the one with Ann and Ben is called a counterexample: a complete town, with every fact filled in, where all the rules hold and the claim fails. That is what an engine had to produce to answer no. Anyone can check a counterexample by hand, so a no can’t be bluffed. To answer yes, an engine had to be sure that no counterexample exists at all, which is much harder.

To make the puzzles harder still, we put several towns in a row and joined them with bridges. A bridge is a rule that ties one town to the next. Each town’s people sit in numbered seats (1, 2 and 3), and a bridge links the people in the same seat in neighbouring towns. For example:

Now a fact in town 1 can force a fact in town 2, which in turn forces one in town 3. To answer a question about one town, you may have to work through all of them. The largest world had 14 towns, each with three people, which comes to 252 separate yes-or-no facts (who belongs to which group, and who answers to whom), and the rules ran to 79.

The third scenario is a trick question, to see whether an engine knows the limits of its own answer. Here are three rules about who answers to whom among a town’s lamplighters:

The question is: can a town of three lamplighters obey all three rules?

Try it with Ann, Ben and Cal. Ann answers to Ben, Ben answers to Cal, and Cal has to answer to someone too. Whoever Cal picks, the line of command loops back. If Cal answers to Ann, for instance, the second rule passes Ann’s line of command all the way round, so Ann ends up answering to herself, which the third rule forbids. More people only postpone the problem. With a thousand lamplighters, the chain still has to loop back to someone eventually, and the loop always breaks the third rule.

With endlessly many people, though, the rules work perfectly well. Person 1 answers to person 2, person 2 to person 3, and so on forever. Each person also answers to everyone further along the line, as the second rule requires. Nobody ever has to answer to themselves, because the line never loops back.

So the answer is no, for a town of three people. But take that answer out of its question, and a bare no could be mistaken for a claim about every possible town. The qualification matters.

Across the three scenarios, each engine had to answer more than a thousand questions without a careless slip: follow every plant question through the wiring, weigh every unknown both ways, notice a question that describes a plant that couldn’t exist, fill in every fact of a counterexample, and say how far each answer reaches.

SapienTronic AI answered every question correctly, and it did so about sixty times faster than Claude, using no tokens and no GPU. Claude used about half a million tokens, while SapienTronic AI’s running cost was a laptop’s electricity for half a minute.

SapienTronic AI did this on an ordinary laptop, while Claude ran on specialised data-centre hardware. So the result isn’t only that SapienTronic AI finished sixty times faster, but that it did so with far less computing equipment.

The speed matters for more than cost. SapienTronic AI took about 24 thousandths of a second per question on a laptop, which is already close to real time, and faster hardware would shorten it further. A language model can’t get there the same way, because of how it reasons. It thinks by writing its reasoning out, one token at a time, and each token takes time to produce. Here Claude used about 450 tokens per question. Faster chips make each token quicker, but careful reasoning still means a lot of tokens, so a language model’s careful answers still take far longer than an engine that doesn’t have to write anything out.

None of this takes away from Claude, which came within one answer of a perfect score. Its counterexamples were valid, its 200-step chains held up, and it handled the trick question well. It marked 1,168 of its answers certain and the other 18 uncertain, and all 18 of those were right. On raw accuracy, the two engines were almost indistinguishable.

Claude’s one mistake was treating an impossible situation as real. It came in the first round. The question set out the plant’s conditions: nobody on site, the reservoir nearly full, the river not too cloudy, a modest chlorine dose, and the chlorine in the water not confirmed by the analyser. Then it forced the chlorine feed on by hand and asked about pump B.

Without the override, those conditions fit together. With the reservoir nearly full, the plant shuts down, the filters go out of service and the chlorine feed stays off, so the analyser has no chlorine to confirm. Forcing the feed on changes that. The plant’s rules say that whenever chlorine is being fed at a dose like this one and the river is this clear, the chlorine level is in range, and a working analyser confirms it. Under the plant’s rules, the analyser is working unless a failure is specified, and the question specified none. So with the feed forced on, the analyser must confirm the chlorine. Yet the question had stated that it didn’t. No state of the modelled plant can satisfy all those conditions, and the right answer was impossible.

Overriding a part is a fair thing to ask about. This question crossed the line because the override contradicted something else the question had stated as fact.

Claude didn’t notice the contradiction. It answered the question as asked, said pump B would stay off (“pump B needs run request”), and marked its answer certain.

A language model can make this mistake because it reasons by writing out an argument, one step after another, and each step follows naturally from the last. Asked about pump B, the natural argument runs from pump B to what pump B needs, and it was correct as far as it went. Checking that the question itself describes a possible plant is a separate job. The model has to think of doing it, and nothing in the way it works makes it do so every time. Its certain then reflects how sound the argument felt, which says nothing about the check it skipped.

SapienTronic AI can’t skip that check, because it isn’t a separate step. It builds the whole plant, with the feed forced on and every stated condition held in place, and lets it settle. When no state of the plant satisfies everything at once, there is nothing for it to settle into, and the answer comes back as impossible without anyone having to think of asking.

We ran three rounds, with a fresh Claude agent each time. The first two used Fable 5.1, and the third used Opus 5.5 (after Fable hit its usage limit). Claude got each problem described in plain English and was asked to reason it through without writing code, and SapienTronic AI got the same problems set up as systems. We wrote every round down before running it (so we couldn’t move the goalposts afterwards), and sealed the answers from both engines. Both were marked by the same independent programs (including an industrial constraint solver for the puzzles). We logged tokens and time alongside the scores.

A solver could find these answers, but only with a person setting up each problem and writing a program around it.

A constraint solver is a program that you give a puzzle (like the towns), and it searches for the answer. Solvers are good at this. We used one to check the answers in this contest.

A solver does the individual calculations well, and good ones do more (prove that a problem has no solution, or point to the rules in conflict). The harder job is everything around the solver, and it has three parts:

The catch is that the program is written for one problem. It must anticipate how to interpret and act on each kind of result the solver may give, and a new problem means a new program.

SapienTronic AI needs no problem-specific reasoning program. The plant and the towns were both described to it as systems, and the same engine answered every kind of question about both. Its chains run on one general rule, driven by its own labels: an exact answer is carried forward, an undecided one is held back and named, and an impossible situation is reported as impossible. Because that rule is the same for every problem, a new problem needs a new description, and no new reasoning code.

Back to Claude’s one mistake. One slip in 1,186 is a fine record for a chatbot. But a machine entrusted with a chemical plant or a moving car needs something more than a good average score, for three reasons.

You can’t tell which answer is the bad one. Claude said certain 1,168 times and was wrong once, and the wrong answer looked like every other. An operator acting on that advice acts on the b