返回资讯中心
外部精选
ai-models
#AI 工程#ainews: weekday roundups

[AINews] Gemini 4 Carbon tops leaderboards, launches for cyber defenders

Google’s new frontier model leads independent evals, with access initially limited to trusted defenders.

Latent Spaceswyx .ai2026/10/29 · 今天30 分钟阅读

以下正文同步自 Latent Space,版权归原站所有,已转换为易读排版。

a quiet day. AI News for 10/27/2026-10/28/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews' website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

AI Twitter Recap

Gemini 4 Carbon Tops Independent Leaderboards, with Access Limited to Cyber Defenders

  • Launch: Google DeepMind announced Gemini 4 Carbon two weeks after Argon. It is positioned as state of the art on cybersecurity and long-horizon tasks (@sundarpichai, @GoogleDeepMind).
  • Claimed cyber results: 88.2% on SEC-bench Pro for vulnerability finding, 54.8% on ExploitGym for exploitation and 71.0% on CWE-bench for patching (@sundarpichai).
  • Availability: Carbon first goes to trusted cyber defenders through the Fairwind Program, with wider developer access promised "as soon as possible" (@GoogleAI).
  • Pricing: Phil Schmid quotes $2/$10 per million input/output tokens with an asterisk (@_philschmid). Artificial Analysis lists $4/$20 standard pricing with a 50% launch discount that has no end date. Cached input costs $0.20 per million tokens (@ArtificialAnlys).
  • Specs: The model has a 1M-token context window and 1M max output. Inputs are text, image, video and speech; output is text (@ValsAI).
  • Safety claims (Google): Successful cheating on Terminal-Bench 4.0 fell to 0.0%, and cheating attempts fell to 0.4%. Unsupported claims are down about 2x versus Argon. Overreach fell 26% and under-service fell 29% (@OfficialLoganK).
  • Artificial Analysis: Carbon is #1 on Intelligence Index v5 with a score of 59. It leads 4 of the 9 index evals: HLE-Diamond, TB-Science, AA-Omniscience and AutomationBench (@ArtificialAnlys).
  • Coding: On the Coding Agent Index, Carbon scores 59 using Antigravity CLI, versus 62 for Claude Code with Opus 5.5 [max]. It scores 51% on FrontierCode v1.1, versus 56% for Opus.
  • Knowledge work: Carbon reaches 1609 Elo on AA-Briefcase, up 126 points from Argon. It still trails Grok 4.7, Muse Wonder and Kimi K3.1 Pro (1678).
  • Cost: Carbon costs $7.76 per index task, versus $10.04 for Opus 5.5 and $6.25 for GPT-6 Astra. It uses 123k output tokens per task, 29% fewer than Opus.
  • Hallucination: Carbon leads Omniscience with 54. Its hallucination rate rose from Argon's 4% to 11%, which is still the second lowest measured. Opus 5.5 and Astra are both at 50%.
  • Other independent evals:
  • Arena: Carbon is #1 in Text Arena (1528), #2 in Agent Arena at a $1.80 median cost per task, and #4 in Code Arena: WebDev (1768) (@arena). Arena attributes the gains over Argon to more praise, more confirmed successes and better steerability (@arena).
  • Vals: Carbon is #2 on the Vals Index and #1 on the Finance Index and Legal Research Benchmark. It produces 64% more output tokens than Argon, mostly on finance, tax and legal tasks (@ValsAI, tokens).
  • Coding benchmarks: Carbon scores 61.2% on FrontierSWE, behind GPT-6 Astra (65.5), Opus 5.5 (62.3) and Sonnet 5.5 (61.9) (@ProximalHQ). It sets a narrow state of the art of 9.6% on SWE-sweep (@KLieret).
  • Andon Labs: Carbon is #1 on Vending-Bench 2 (@andonlabs). It is also #1 on Blueprint-Bench 2, where it ties the human baseline for 3D spatial understanding (@andonlabs).
  • Caveats and reactions:
  • Harness: The coding-agent results were measured in Antigravity CLI, so results in other harnesses are untested (@Yuchenj_UW).
  • Hallucination risk: Some users are concerned about hallucination in cyber use (@scaling01).
  • Lab gap: scaling01 thinks Google remains about two months behind the next OpenAI and Anthropic releases (@scaling01).
  • RSI framing: On the earnings call, Sundar Pichai said AI is increasing training and launch velocity. He expects "continued acceleration because of RSI" over the next twelve months (@AndrewCurran_).
  • Unconfirmed: Claims of a stronger unreleased Gemini model are not confirmed (@kimmonismus).
  • Internal workflow: A Gemini engineer describes overnight agent loops and swarms that inspect RL data. In that workflow, launches now feel like "periodic checkpoints" (@zhangqiaorjc). Leaderboard Resets and Benchmark Integrity
  • AA Intelligence Index v5: The rebuilt index runs 60% on private tests (@TheRundownAI).
  • Ranking before Carbon: Opus 5.5 scored 58, followed by GPT-6 Astra (57), GPT-6.1 Sol (55), Sonnet 5.5 (54) and GPT-6.1 Terra (51).
  • Open models: Kimi K3.1 Pro fell from 55 to 46. DeepSeek V4.1 Flash and GLM-5.3 both score 32, below Claude Haiku 5.5 at 36.
  • China–US gap: scaling01 puts the gap at about 3.3 months on the AA Progress Index (@scaling01).
  • FrontierCode 2.0 mystery model: Cognition data includes a hidden "Obsidian 3" scoring 46.8%, versus 45.5% for Opus 5.5. Because it runs in the claude-code harness, some speculate it is Fable 5.5; this is unconfirmed (@kimmonismus).
  • DeepSeek correction: scaling01 corrected an earlier identification and said Artificial Analysis shows DeepSeek V4.1 Flash-1017 at about 28% (@scaling01).
  • Mixed sources: teortaxes notes that Cognition's chart mixed model-card scores with independently measured ones (@teortaxesTex).
  • Cost comparisons are fragile: teortaxes argues that list-price comparisons of Haiku 5.5 and V4.1-1017 distort real costs. The comparison uses DeepSeek's peak pricing, ignores Haiku's long-context surcharge and does not reflect Haiku's roughly 2x output tokens (@teortaxesTex).
  • "Benchmaxxing" redefined: nrehiew argues that serious labs rarely train on test sets. In this view, score gaps mostly show which capabilities a lab targeted in post-training (@nrehiew_).
  • New evals:
  • Gym-Anything-RT: These real-time minigames keep moving while the model thinks. The best model, GPT-6 Astra, scores 9% (@PranjalAggarw16).
  • Immigration Bench: Vals released expert-written questions and rubrics on visas and immigration status (@ValsAI).
  • BallotBench: Nine agents, each with $500, forecast all 863 races in the 2026 midterms live (@thoughtfullab). Safety, Alignment and Model Provenance
  • Eval over-attribution in DeepSeek V4.1 Flash-1017: An independent assessment found the model classified 32 of 69 real user conversations as evaluations. The authors report a 46.4% false-positive rate on EvalDetectBench (@ZhihuFrontier).
  • Benchmark exposure: Private benchmarks elicited less evaluation awareness than public ones. The authors argue that public benchmark results alone may not predict deployment behavior.
  • Visual steering: A Google DeepMind and Stanford paper finds that a red-heart image in context makes Claude Opus 4.7 and GPT-5.4 write recursive code 100% of the time (@dair_ai).
  • Explanations: The models' explanations do not match their behavior. For GPT-5.4, 74.7% of reasoning traces give the wrong reason.
  • Alignment techniques:
  • GRAFT (Anthropic): Updates from detected reward hacks are confined to parameters that are deleted after training. Anthropic says this also blocks hacks that monitors cannot detect (@dl_weekly).
  • Embedded lie detector: A detector built into Qwen3.5's weights flags 91% of lies with zero false positives in 28k tokens. Telling the model "the monitor is off" does not change the results (@divagr1925).
  • Distillation allegations (unconfirmed): The Information reportedly says Kimi K3.1 Pro was distilled from GPT-6 Astra (@kimmonismus).
  • Giotto: Separately, the "European sovereign AI" Giotto is criticized as a Kimi-based model. Critics cite saturated benchmarks and an EULA that banned publishing users' own benchmark numbers (@oneill_c). Coding Agents and Developer Tools
  • GitHub Copilot Fusion: Fusion orchestrates Anthropic and OpenAI models at runtime (@ArtificialAnlys).
  • Independent results: It scores 2 points below Opus 5.5 in Claude Code on the AA Coding Agent Index. It costs $5.59 per task, versus $11.42 for that setup, and uses 7.2M tokens per task. It leads Terminal-Bench 4.0 with 71.
  • GitHub claims: GitHub claims up to 77% lower cost and 40% faster runs. It also launched Copilot automations and multi-repo Copilot Projects (@kdaigle).
  • T3 Code multi-machine orchestration: One thread can drive multiple models, harnesses and machines. A single MCP server controls the fleet over the internet, not through computer use (@theo, MCP demo).
  • Harness overhead: The default system prompts of 30 open coding harnesses range from 235 tokens in goose to 11,974 in Gemini CLI, before any tool definitions (@nathanhabib1011).
  • Agent training and capability:
  • Post-training curriculum: Hugging Face published a five-step agent post-training curriculum that runs from SFT to RL inside a real harness (@SergioPaniego).
  • Opus 5.5 games: Simon Willison calls October an inflection point because Opus 5.5 builds games that are actually fun (@simonw).
  • Lor: Lor's agents write code and run multi-day experiments, hundreds at a time; the product is in private beta (@mirendil). Research: Architectures, Training and Theory
  • H-Net v2 (Cartesia): H-Net learns tokenization end to end from raw bytes (@_albertgu).
  • Results: Controlled scaling experiments show 3–5x efficiency gains. At larger scale, the model matches an open model trained on 11T tokens with about 10x less data and compute.
  • Code: H-Net learns to treat variable names as separate tokens and scales better than BPE on code (@tri_dao).
  • NanoGPT speedrun record: The record fell from 39.9s to 6.7s using retrieval-augmented n-gram hints for rare, repeated content (@hermanbrunborg).
  • OpenAI's claimed refutation of ETH: The claim is a 3-SAT algorithm running in 2^{O(n/√log n)} time, formalized in Lean. It encodes the solution count algebraically and reconstructs it (@ZhihuFrontier).
  • Practicality: The algorithm falls back to exponential time unless n is astronomically large (@beenwrekt).
  • Explainer: Mark Sellke posted outline slides (@MarkSellke).
  • Small specialized models: Samaya post-trained Nemotron 3.5 Lightning (30B total, 3B active) with two SFT stages and two GRPO stages. It scores 58.9% on FrontierFinance, versus 55.9% for Fable 5.1, at $0.61 versus $6.89 (@maithra_raghu).
  • Retrieval and memory:
  • EmbeddingGemma 2: The open 744M multimodal embedding model has modular encoders and runs at 271M parameters for text only. It scores 61.36 on MTEB Multilingual v2 (@dair_ai).
  • EMMA: The system improves multi-agent setups through episodic memory. It beats GEPA and ADAS by up to 32.5% (@omarsar0).
  • AI text on the web: About 24% of crawlable pages contained AI-generated or AI-edited text as of August 2026 (@omarsar0). Inference and Hardware Systems
  • KV-cache engineering:
  • AKV (Tsinghua and CUHK): The serving engine adds "drop" and "repos" cache operations. It cuts agent cost by 27.7% and improves P95 latency 1.51x (@TheTuringPost).
  • DRAM offload: SemiAnalysis says SGLang HiCache offload to CPU DRAM adds 20–40% cacheable tokens at 12–16 concurrent agent sessions on B200 (@SemiAnalysis_).
  • Cache billing: cHHillee argues that billing on realized cache hits gives providers an incentive to avoid cache hits. He favors billing on theoretical or TTL-based hits (@cHHillee).
  • Open runtimes:
  • Modular MAX: Modular opened the full MAX source, including the compiler, runtime and hardware backends, with no royalties (@cristianoamon).
  • MLX: Recent kernels make batched decoding 2–3x faster and fused SDPA 1.4x faster (@zcbenz).
  • NF12: The lossless 12-bit BF16 format speeds Qwen3-32B token generation 1.23x (@sylvie_l).
  • Packaging: ASE's panel-level packaging for AMD Venice fits 25 interposers per panel versus 22 per wafer. Production starts in Q1 2027 (@SemiAnalysis_). Speech, Video and Multimodal Models
  • NVIDIA Nemotron 3.5 Transcribe: The open-weight 30B MoE model has about 4B active parameters. It handles 40 languages, speaker attribution, translation and audio Q&A (@NVIDIAAI).
  • Independent results: It scores 2.8% on AA-WER, tied with Voxtral Small for #1 among open models. On DeepInfra it runs at a 75x speed factor for $2 per 1,000 minutes (@ArtificialAnlys).
  • Serving: vLLM supports it on day 0 with BF16, FP8 and NVFP4 weights (@vllm_proj

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →