返回资讯中心
外部精选
软件工程
#开发者实时流

CacheVerifier, check if verifying LLM cache hits helps your traffic

Python client for CacheVerifier — a hosted API that

Hacker News 最新ChengyouXin2026/10/19 · 28 天前15 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Python client for CacheVerifier — a hosted API that verifies semantic-cache hits. Given a query and a candidate cached answer, it approves or rejects serving that answer from cache, so a similarity match that is close but wrong doesn't become a silent error in your app.

CacheVerifier does not run your cache or do similarity search. Your cache backend does its own lookup first; you call verify() only on the candidates in the similarity "gray zone", where a plain threshold match might be wrong.

pip install cacheverifier

with the GPTCache adapter:

pip install "cacheverifier[gptcache]"

with the offline Health Check (adds torch + sentence-transformers):

pip install "cacheverifier[healthcheck]"

Requires Python 3.9+. The only runtime dependency is httpx — the extras above are opt-in.

Get a free API key at https://www.cacheverifier.com (self-serve verify and fine-tuning are free forever, no card).

from cacheverifier import CacheVerifier

cv = CacheVerifier(api_key="cv_...")

query = "how do I cancel my subscription" candidate = "Go to Settings > Billing > Pause subscription for a month." # from your cache

result = cv.verify(query, candidate) if result.approved: answer = candidate # verified hit — skip the LLM call else: answer = call_your_llm(query) # not trustworthy — fall through

Later, once you know if it was actually right (thumbs-down, reopened ticket, ...):

cv.feedback(query, answer, was_correct=True, similarity_score=0.86)

verify() returns a VerifyResult:

verify() runs inline with your traffic, so its defaults are conservative:

cv = CacheVerifier(api_key="cv_...", verify_timeout=1.0, fail_open=False) # the defaults

fail_open=True instead returns approved=True on an outage — only if a

stale-or-near-miss answer is acceptable for that traffic:

cv = CacheVerifier(api_key="cv_...", fail_open=True)

There is no formal uptime SLA yet, which is the other reason the fallback path is a built-in default rather than left to you.

Drop the verifier into a GPTCache pipeline as its similarity evaluator — no fork required:

from gptcache import cache from cacheverifier.integrations.gptcache import CacheVerifierEvaluation

evaluator = CacheVerifierEvaluation(api_key="cv_...") cache.init(similarity_evaluation=evaluator, ...)

when you learn a served hit's real outcome:

evaluator.report_feedback(query, answer, was_correct=False, similarity_score=0.9)

See examples/gptcache_example.py.

Once you have ~20+ feedback rows (the service found fine-tuning is often a net negative below ~1,000 on the hardest data — see the research), train a verifier on your own gray-zone labels:

job = cv.finetune() # or cv.finetune(target_risk=0.01, cost_ratio=5.0) job = cv.get_finetune_job(job["id"]) # poll until status == "done" print(job["auc_baseline"], job["auc_tuned"])

a model can finish as "held_for_review" — promote it explicitly:

if job.get("result_model_version"): cv.activate_model_version(job["result_model_version"])

cv.dry_run([...]) reports the same baseline-vs-tuned AUC on examples you pass directly, without writing anything or deploying a model.

cv.dry_run() still uploads your examples to the API. If that's a blocker — a compliance review, or just not wanting production traffic to leave your network — run the identical stock-vs-fine-tuned evaluation entirely on your own machine:

pip install "cacheverifier[healthcheck]"

cacheverifier healthcheck traffic.jsonl cacheverifier healthcheck traffic.jsonl --emit-summary summary.json

traffic.jsonl is a JSON array or JSONL of {"query", "candidate_answer", "was_correct"} rows in arrival order (the train/calibrate/test split is chronological, matching the hosted service so the numbers are comparable). Optional per row: "stale": true.

Nothing is sent anywhere — the base model downloads once from Hugging Face, then it's fully offline. --emit-summary writes an aggregate-only JSON file (AUCs, counts, rates — no query or answer text) that's safe to share for a human read.

Choosing the base model. --base-model takes the same values as the hosted service's base_model fine-tune parameter, so local and hosted numbers stay comparable:

The English bases read most Chinese characters as unknown tokens, so on Chinese traffic their result doesn't mean anything — the CLI warns when it sees mostly CJK text on an English base. You can also pass any Hugging Face model id or a local model directory. If Hugging Face is unreachable from your network, set HF_ENDPOINT to a mirror, or download the model elsewhere and pass its directory:

cacheverifier healthcheck traffic.jsonl --base-model multilingual HF_ENDPOINT=https://hf-mirror.com cacheverifier healthcheck traffic.jsonl --base-model multilingual_small cacheverifier healthcheck traffic.jsonl --base-model ./models/bge-reranker-base

results

train / calibrate / test: 3349 / 419 / 419 stock verifier held-out AUC: 0.6120 fine-tuned held-out AUC: 0.7080 (delta +0.0960) label-noise proxy (disagreement): 11.4% ceiling status: still_improvable

verdict

IMPROVED -- fine-tuning on your own data helps this traffic

API surface

method endpoint

verify(query, candidate_answer) POST /v1/verify

verify_batch(pairs) POST /v1/verify/batch

feedback(...) / feedback_batch(items) POST /v1/feedback / /batch

finetune(...) / dry_run(examples, ...) POST /v1/finetune/jobs / /dry-run

get_finetune_job(id) / list_finetune_jobs() GET /v1/finetune/jobs[/id]

activate_model_version(id) POST /v1/finetune/model-versions/{id}/activate

drift_status() GET /v1/monitor/drift-status

gray_zone_threshold() GET /v1/monitor/gray-zone-threshold

usage() / savings() GET /v1/usage/status / /savings

Non-2xx responses raise CacheVerifierError (.status_code, .detail).

MIT — see LICENSE. (The research repository is separately licensed; this client is not.)

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →