gBench is an open-source benchmarking and evaluation suite for open large language models (LLMs) and vision-language models (VLMs). It measures serving performance on the hardware it runs on, and model capability in thr…

以下正文同步自 Hacker News 最新，版权归原站所有，已转换为易读排版。

It works with any OpenAI-compatible /v1 server through --remote-endpoint (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls), or starts and stops a local vLLM server itself.

Key features

Documentation

Installation

Quick start

Three capability evaluation tiers

Developer guide and contribution workflow

Authors

License

Key features

OpenAI API compatibility: measure any OpenAI-compatible /v1 endpoint with --remote-endpoint <URL>/v1 (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls; see Remote Endpoint Mode).

Three capability tiers:

Academic and domain evals (--evals): 86 canonical Python suites in --evals all (plus ui_control_osworld and swe_lancer, which run only when named) across 6 core capability pillars (General Knowledge & Scientific Reasoning, Mathematics & Proofs, Coding & Algorithmic Design, Long Context & Retrieval, Tool Use & Agentic Workflows, Multimodal Vision & Grounding) with the model's reasoning mode on request (--eval-thinking), images sent as their original bytes where the dataset stores them (no re-encoding), plugin discovery (--eval-plugins-dir) and custom JSONL datasets (--eval-custom-jsonl).

Golden Set smoke tests (--golden-only): 16 deterministic capability checks (code_canonical, function_call_single, structured_json, etc.) with two-sided presence assertions and python_exec unit-test matching, including a labelled chat template subset that separates a broken chat template from a weak model.

Agentic quality suite (--quality-only): the GemmaClaw / OpenClaw QA suite (the QA scenarios of the pinned GemmaClaw commit, 60 at d8bc6989; scenarios that need another model provider are listed apart and not scored) testing multi-turn tool calling, session memory and plugin routing, run in gBench's local gbench-quality Docker image (built once per GemmaClaw commit).

Model discovery and GPU sizing: model properties read from the HuggingFace config.json (parameter count, MoE topology, vision/audio tokens), a recommended GPU tier per model size (1 GPU up to 20B, 2 GPUs up to 80B, 8 GPUs above) that is checked against the required --num-gpus (a warning when off the tier, a refusal when the model will not fit), and a server startup timeout that scales with model size (max(600, min(3600, (300 + params_b * 10) * 3 * (1.3 if MoE)))).

Repeatability checks: several iterations per configuration with mean, median, standard deviation, P50, P95, P99 and the coefficient of variation, checked against --max-cv-percent (default 5.0).

Open-loop stress sweep (--stress-test): one merged Poisson schedule per rate point at increasing rates (x2 per step) until a point fails; each of --stress-reps repeats gives a bracket [pass, fail], and the median repeat's limit is then certified with probes sized for the P99 SLO (a cohort of >= 1099 requests, about 1400 sent, the same rule on every device) to +-9% when that resolution is reached (otherwise a certified lower bound with its resolution, a lower bound censored at --stress-max-qps, or no limit when certification refutes the sweep), or reported UNCERTIFIED with the time certifying would need (--stress-certify-budget). A point passes when, on the requests planned after the point's ramp-up: the queue is not growing, >= 90% finish within the end-to-end time the SLOs imply, P99 TTFT and P99 ITL meet their SLOs, failures stay under 2% and the client kept to its schedule. The queue-growth test reads TTFT; overload that shows only as longer end-to-end times (for example KV-cache preemption) is left to the completion-fraction and ITL gates. SLOs are per campaign (5000 ms TTFT and 200 ms ITL without --campaign), overridden by --stress-threshold and --itl-slo-ms.

Documentation

Using gBench

git clone https://github.com/google-deepmind/gbench.git cd gbench python3 -m venv venv && source venv/bin/activate

pip install -e . # base package (Python 3.11+): laptops and remote endpoints such as Ollama

optional, per use:

pip install transformers # --remote-endpoint: prompt sizing (--tokenizer), Golden Set chat-template

cases and the mmlu, wmdp, ruler and longbench_v2_code suites

pip install vllm # local GPU serving, and remote real-dataset workloads (--campaign chat-like,

--dataset custom/hf); gBench never installs vllm/torch itself

pip install -e ".[evals]" # the eval suites' client, scorer and harness dependencies

The installation guide covers the requirements, conda, the optional installs per suite and the test environment.

1. Quick local performance smoke test (auto-starts local vLLM server)

gbench --models google/gemma-4-E4B-it --num-gpus 1 --preset quick

2. Run a specific workload performance campaign (e.g. agentic long-context)

gbench --models google/gemma-4-12B-it --num-gpus 1 --campaign agentic

3. Run the Golden Set capability smoke tests on a local vLLM server

gbench launches; its tool cases need the model's tool-call parser (Gemma 4 example)

gbench --golden-only --models google/gemma-4-12B-it --num-gpus 1 --tool-call-parser gemma4 --reasoning-parser gemma4

4. Run GemmaClaw agentic quality suite over remote endpoint

gbench --quality-only --models google/gemma-4-12B-it --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-12B-it

5. Run academic evaluations with Chain-of-Thought reasoning against remote server

gbench --evals-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --evals gpqa mmlu --eval-thinking --max-output-tokens 65536

6. Stage model artifacts from HuggingFace to Google Cloud Storage

gbench --models google/gemma-4-12B-it --stage-to-gcs gs://my-gbench-bucket/models/

7. Dry run: the plan, each server's vllm serve command, port, GPUs and base URL (validated)

gbench --models google/gemma-4-12B-it --num-gpus 1 --preset default --dry-run

Three capability evaluation tiers

Tier 1: academic and custom evals (--evals)

The full catalog of the 86 suites in 6 pillars, with each suite's protocol and options, custom plugins and JSONL suites: docs/eval-suites.md. Each suite has a setup and scoring guide in docs/evals/.

16 deterministic capability checks (code, math, tool calling, structured output, chat templates, factual knowledge, translation, safety, vision and audio) with two-sided presence assertions and python_exec unit-test matching, including a chat-template subset that separates a broken chat template from a weak model. Details: docs/golden-set.md.

Evaluates multi-turn agent behavior with the GemmaClaw / OpenClaw QA suite: the QA scenarios of the pinned GemmaClaw commit (60 at d8bc6989); scenarios that need a model provider other than the model under test and the Gemini API are listed apart and not scored.

Run specific GemmaClaw QA scenarios (by scenario id) from a local checkout

gbench --quality-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --scenarios memory-recall mcp-plugin-tools-call --gemmaclaw-path /path/to/gemmaclaw

Developer guide and contribution workflow

Install development environment (in a test environment, see tests/README.md): pip install -e ".[dev,evals,omnidocbench]" transformers and python -m nltk.downloader punkt punkt_tab; the system tools the tests exercise are bubblewrap, a Java runtime and Docker (the Unit Tests workflow installs exactly this)

Run code style check: ruff check gbench/

Run type checker: mypy gbench/

Run unit test suite: pytest tests/ (the Gemma 4 chat-template artifact check is not a unit test; it lives in tools/template_checks/, see its README)

The pull-request gate is the unit-test suite (pytest tests/, see CONTRIBUTING.md). A change to the Golden Set keeps its two-sided presence assertions; the Golden Set itself measures a model and never gates a gBench merge.