SpacebarBench: Benchmarking Agents on Whiteboards

Fourteen requests, a changing financial model, and one workspace that has to keep making sense.

October 2026 benchmark: Six model configurations, fourteen user requests, and a 100-point audit of the final work. Runtime is cumulative execution time. Scores describe delivered artifacts, not a statistical model ranking.

Open three pricing pages. Put them next to each other so we can compare them. Keep them open; we will need them again.

That is how our benchmark begins. It is an ordinary request, and deliberately so. A person evaluating customer-support software does not usually start by specifying every formula, chart, risk note, and procurement rule that the project will eventually require. They begin with a question, inspect what they find, and develop the work from there.

The evaluation target is the evolving project. Later requests inherit earlier artifacts; a correct reply is insufficient if the retained work becomes inconsistent.

The benchmark follows one project: compare three customer-support platforms and prepare a budget and recommendation. Fourteen consecutive user requests build on the same work:

Workflow at a glance. These are successive requests in one workspace—not separate, fresh-start tasks.

By this point, answering the latest message is only part of the job. The agent must find the right existing artifacts, update them in place, preserve the assumptions that still apply, and make sure the spreadsheet, dashboard, and memo tell the same story. A correct paragraph in chat does not repair a stale chart. A polished memo is not enough if its recommendation rests on a broken formula.

This is the shift we set out to study: from an agent that returns work to a person, to an agent that participates in the workspace where people are already working.

In Spacebar, the agent works within a persistent collaborative canvas containing browser viewers, documents, spreadsheets, and interactive apps. These are shared working objects, not just attachments at the end of a conversation. A viewer opened early in the discussion can be revisited later. A spreadsheet can remain on the board as its assumptions change. An app can hold a shared scenario selection that the agent is asked to operate and verify.

The interactive application in this workflow is a Mario app: a custom collaborative application built dynamically by the agent, hosted by Spacebar itself, and using the same synchronization layer as Spacebar. For this benchmark, the agent builds the CX Vendor Decision Lab, a dashboard with vendor and scenario selectors, editable assumptions, cost charts, and a recommendation area. Its selections and editable assumptions live in shared application state, so the app becomes a working tool for collaborators rather than a static dashboard delivered at the end of a chat.

That changes what success means. The question is not only whether an agent can create a useful artifact. It is whether it can keep the work useful as instructions, evidence, and decisions evolve.

The important property of this environment is that people and agents can work on the same persistent objects. They can refer to the same browser viewer, inspect the same workbook, and use the same collaborative app. The work does not have to be copied out of an agent-only environment before it becomes available to the team.

Sharing an object is not the same as sharing every low-level interaction. In our implementation, agents can use browser controls, board tools, and Google Workspace helpers. People interact through the visible application interfaces. Both operate on shared working artifacts, but the agent is not restricted to reproducing every human mouse click. The benchmark evaluates the deployed agent and its tools, not a vision-only computer-use system.

Shared objects, distinct interaction paths. This describes the workspace's interaction model and the pilot's scope, not a vision-only system or a concurrent multi-human experiment.

This distinction also keeps the novelty claim precise. Human–agent collaboration has prior art: Microsoft's Magentic-UI supports collaborative browser execution, and research on collaborative document editing has placed AI agents within multi-user writing environments. Our focus is the combination of persistent viewers, native documents and spreadsheets, an interactive shared-state app, and revisions that must propagate across them. Magentic-UI, Collaborative Document Editing with Multiple Users and AI Agents.

The broader workspace supports collaboration. This particular pilot tests a narrower slice: one scripted sequence of user messages, delivered over an evolving session. It does not measure simultaneous human edits, conflicting instructions from several participants, or how well a team coordinates with the agent. Those are important next experiments, not results established here.

[Table 1 caption: The environment enables a broader form of collaboration than this first pilot measures.]

We chose a customer-support platform evaluation because it connects research, calculation, presentation, and decision-making without making any one of them sufficient on its own. The user compares Intercom, Zendesk, and Freshdesk for a hypothetical 2027 support operation. The commercial information comes from live vendor pages; missing prices must remain explicit rather than being filled in from memory.

The workflow starts with 42 full agents in January, growing by two each month, 18 light users, 55,000 monthly support conversations, and 6,000 AI-resolved outcomes. Conversations initially grow by 3% per month and AI outcomes by 4%. The budget starts with a 10% contingency. Subsequent messages change these assumptions and add constraints.

These are not fourteen independent attempts. Each later request inherits the work left by earlier ones. Recreating a dashboard instead of changing the existing one can violate the request even if the replacement looks good. Navigating the wrong viewer can remove useful context. A late edit can undo calculations that were previously correct.

Figure 2. Requested touches to the persistent artifacts. C marks creation or addition; R marks reading, operation or revision. This is the workflow design, not a claim that every run completed each operation.

The final procurement instruction makes the dependency especially clear: use annual billing only when a documented discount is at least 10%; otherwise model monthly billing, and do not infer an undisclosed discount. The agent must establish the pricing evidence, update the selected rates, let the workbook recalculate, revise the app, and reconcile the memo. If a required price remains unknown, a numeric ranking may no longer be justified.

Figure 3. The final procurement request creates a chain of required checks across existing artifacts. The sequence describes work the agent must perform, not automatic synchronization between the workbook, Mario app, and memo. An undisclosed price must remain unknown rather than silently becoming zero.

The agent's conversation is evidence of what it attempted and what it told the user. It is not sufficient evidence of what exists.

Our artifact-first audit inspects the delivered workbook, document, board inventory, app source or state where available, and execution trace. For spreadsheets, that means examining formulas and resolved values as well as visible formatting. For documents, it means checking whether the recommendation is supported and whether the document is readable. For the app, implementing a selector and successfully persisting a selection are different checks; neither alone proves that the final interface renders correctly for collaborators.

The retained rubric allocates 100 points across six categories:

These are judgmental audit scores, not probabilities of success. A score of 84 does not mean an agent will succeed on 84% of workplace tasks. Small differences between single runs should not be treated as established capability gaps.

We also need to distinguish a defect from missing verification. A failed rendering tool can prevent us from checking an app's appearance without proving that its interface is broken. An unavailable commercial price can be handled correctly by leaving the total unresolved. In contrast, displaying that unresolved total as zero can mislead a reader even if the underlying data model records it as unknown.

We evaluated the final artifacts produced by six model configurations on the fourteen-message CX workflow in Spacebar. The results use the October 7–8, 2026 records and a frozen 100-point rubric. Reasoning settings are shown explicitly; the same effort label does not imply an equivalent compute budget across providers.

Table 3. Runtime is the sum of measured execution time contributing to the evaluated result, including unsuccessful work within that execution history. It excludes inactive gaps, provisioning and artifact export. Scores assess the final retained artifacts; this is not a first-attempt reliability ranking.

Figure 4. Scores come from a descriptive, non-blind artifact review. We use the lower endpoint of each audited category range, awarding only clearly supported credit. Totals sum those category scores.

The category breakdown is more useful than assigning a different winner to every task. GPT 6.1 Sol has the highest awarded scores for research, the CFO memo and cross-artifact consistency. Opus 5.5 is close on spreadsheet quality (22/25 versus Sol’s 23/25) and delivers its result in 39.8 minutes versus Sol’s 155.0 minutes. These describe the audited results, not statistically established capability differences.

Category profile. Cells show the published category scores; color shows each score divided by its category maximum.

These are observed strengths of the deployed configurations on this workflow—not general recommendations for unrelated tasks. A good app score, for example, does not establish that the same model will reliably maintain a financial workbook.

Each model built its own version of the CX Vendor Decision Lab. These are screenshots of the actual Mario apps, captured through Spacebar’s live app renderer—not mockups or recreated interfaces.

Spacebar exposes that renderer as an agent tool: it returns an image of the app with its shared state, allowing vision-capable agents to inspect the rendered interface alongside its code and state. Visual inspection can reveal layout and presentation problems that a successful publish or state readback alone cannot establish.

Evaluator captures retained with the October 7–8 artifact reviews. These show the captured viewport and state, not every part of each app. They demonstrate a visual inspection path; they do not establish that every benchmarked model used it successfully. Displayed business figures remain subject to the artifact findings below. Open any image at full resolution.

Planning controls, explicit unknown-cost warnings, and monthly and scenario charts.

A dense review layout with shared assumptions, charts, risks and a recommendation.

A compact dashboard combining vendor controls, KPI cards and cost charts.

Large vendor-selection cards and a shared decision-status control.

Prominent decision controls, editable assumptions and cost cards.

Sectioned shared inputs, KPI cards and comparison charts.

A useful result has to be evaluated alongside the time and cost of producing it. The chart below adds together the recorded workflow execution segments for each result. It measures elapsed workflow time, including tool use—not just model inference latency.

Figure 5. Cumulative execution time for the evaluated artifact set. Earlier experiments on discarded artifact sets are excluded. Runtime is measured, not estimated.

Cost covers the same execution history as the runtime above. A tilde denotes a usage-based estimate; ≥ denotes a lower bound where charge coverage is incomplete. ‡ Sol uses published Standard-tier prices applied t