Open-weight RAG stack scores 90% on FinanceBench, all 150 answers published
Each score as reported by its authors, with their own judging. Different libraries, judges and question edits prevent a single like-for-like ranking. VectoGrid takes about a minute per question because every answer is p…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Each score as reported by its authors, with their own judging. Different libraries, judges and question edits prevent a single like-for-like ranking. VectoGrid takes about a minute per question because every answer is produced twice, by two independent methods, and checked before it is shown.
Not listed: LinqAlpha reports 97.2% on 144 of the 150 questions, after leaving out the six that name no company or fiscal year, without stating how documents were provided to the system and without publishing its answers.
Ask in your own words. VectoGrid searches every report at once and brings back the right passage, in the right report, for the right year, even when the table looks the same as last year's.
It reads what a number depends on, fetches the second figure from another statement or another report, and does the arithmetic in code, not in its head.
Two independent methods answer every question and a referee chooses. Every answer comes with its source quotes, so checking it takes seconds, not an afternoon.
An AI cannot reason from evidence it never finds. So before answering a single benchmark question, we spent the work on retrieval, against a test of our own: 1,000 questions, scored by recall@3: a question counts as found only when the passage holding its answer comes back in the first three results. The questions span about 40 documents in English and French, some real reports and some built specifically to trip search up. Every change had to raise the count, or it was thrown out.
Plenty was tried and thrown out along the way: a reranker eight times larger (worse overall, twelve times slower), translating every question (no gain), a vision model re-reading pages (one fix, two breaks).
On top of retrieval we built two ways of answering. A pipeline plans the question, reads each value twice and computes in code. An agent decides what to search and read next, with a calculator and a financial reference built from public material: the SEC's Beginners' Guide to Financial Statements, public ratio definitions from Wikipedia, and the US GAAP taxonomy. A referee chooses between their answers, after two checks: an answer resting on a different year's report than the one asked about loses, and so does a verdict that contradicts its own figures. The agent's quotes are matched word for word to the pages it actually read.
gpt-oss-120b and Qwen 3.6 27B answer, Mistral Large 3 referees, bge-m3 embeds and NVIDIA's Nemotron reranks. All of them can run on your own servers, so your documents never have to leave the building.
This is the heart of VectoGrid. An answer is only as good as your ability to check it, so every figure comes with the passage it was read from, and with a picture of the page itself, the line highlighted. You see what the AI saw, in the original document, in one glance.
What is the FY2018 capital expenditure amount (in USD millions) for 3M?
The FY2018 capital expenditure (purchases of property, plant and equipment) for 3M was $1,577 million, as reported in the Consolidated Statement of Cash Flows under Cash Flows from Investing Activities.
All 150 answers are in the download: the question as published, the reference answer, our answer, the passage it rests on, and our verdict, with a note wherever it differs from the reference.
Every model is open-weight: answering, refereeing, embedding and reranking. The whole stack can run on your own servers, so your documents never have to leave the building.
Every figure is traced to the passage it was read from. Quotes are checked word for word against the pages the system actually read; any that can't be found are dropped, and the published answers show exactly where that happened.
This benchmark used public company reports, with metered calls to hosted copies of these same open-weight models. In deployment, they run wherever you decide.
Bring your documents. Bring the questions your team spends hours answering.
Measure VectoGrid on the work that matters to you: accuracy, evidence quality, speed and cost.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。