返回资讯中心
外部精选
软件工程
#开发者实时流

An AI agent read a Shift_JIS page as 208,953 replacement characters

fix(link-understanding): non-UTF-8 pages reach the model as garbled text - #188766

Hacker News 最新Yigtwx2026/10/31 · 17 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Code

Issues5k+ (5k+)

Pull requests2.9k (2.9k)

Actions

Projects

Security and quality722 (722)

Insights

More items

fix(link-understanding): non-UTF-8 pages reach the model as garbled text - #188766

#188766

Merged

obviyus merged 5 commits intoopenclaw:mainopenclaw/openclaw:mainfrom Yigtwxx:fix/link-understanding-charsetYigtwxx/openclaw:fix/link-understanding-charsetCopy head branch name to clipboard

Oct 30, 2026

ConversationCommits5 (5)ChecksFiles changed

Merged

fix(link-understanding): non-UTF-8 pages reach the model as garbled text#188766

obviyus merged 5 commits intoopenclaw:mainopenclaw/openclaw:mainfrom Yigtwxx:fix/link-understanding-charsetYigtwxx/openclaw:fix/link-understanding-charsetCopy head branch name to clipboard

Conversation

Yigtwxx

commented

Oct 30, 2026

•

edited by obviyus

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

What Problem This Solves

Fixes: linked pages served in a declared legacy charset reach link processors and agent input as garbled UTF-8 replacement characters. This changes incorrect user-visible output, not just allocation or performance.

User impact: an agent reading a linked Shift_JIS page now receives and returns its readable Japanese title. UTF-8 pages are unchanged. Unknown charset labels keep the existing non-fatal UTF-8 fallback; no configuration change is needed.

Reuse the response decoder already owned by web_fetch instead of the link fetcher's unconditional UTF-8 decoder. This preserves the existing BOM/header/document declaration order without adding another parser. The added export is internal, not a Plugin SDK surface. Fetch authority, cancellation, response limits, processors, prompt definitions and stored conversation history contracts are unchanged.

@Yigtwxx's original commit is preserved. The separate 13-line test table is consolidated into the existing transport/URL-selection regression; the shared decoder retains its declaration and fallback coverage. Production +1, tests 0, with no test-line allocation or ceiling change. Earlier closed attempt: #111773.

Built Gateway before/after evidence, native receipts and replay details.

Built Gateway agent turns on current-main baseline 1d2e0b28b60b and candidate 86a4b241c744: two Shift_JIS turns and two UTF-8 controls on each. Real public HTTP pages pass through the normal STRICT guarded fetch, a real CLI processor and the real OpenClaw agent runtime. A deterministic OpenAI-compatible model fixture records the actual input and echoes its title; no paid live-model claim.

Current head 73ec930363d6 has the identical source tree as the proven candidate (6b9d2585e88508e4f86e5d5c5f78834cf54f390e): the contributor's intervening allocation commit was preserved, then its unused allocation removed. All proof inputs are unchanged.

Main's Shift_JIS input and final replies are garbled twice (13 replacement characters each); the candidate produces 阿部寛のホームページ twice with 0 replacement characters. The UTF-8 control remains 気象庁 Japan Meteorological Agency throughout. All eight turns complete with one model request each, no rerouting, native visible-delivery receipts, matching assistant history and clean Gateway shutdown.

Focused tests: 33 passed (transport 5, shared decoder 14, prefix stability 14). The same final transport suite on main fails only the intended decoding regression (4 passed / 1 failed). Type-aware scoped lint, format and whitespace checks pass.

Two-way door: a revert fully restores the old decoder; there is no migration, dependency, schema, credential or configuration change. Blast radius is decoded linked-page text supplied to configured link processors and agent input; UTF-8 fallback and the shared web_fetch decoder remain the existing owners.

There was an error while loading. Please reload this page.

fix(link-understanding): non-UTF-8 pages reach the model as garbled text

…

38de2ab

Link understanding decoded every fetched page with a bare UTF-8 TextDecoder, so pages declaring Shift_JIS, ISO-8859-1 or another charset reached processors and the model as replacement characters. Reuse web_fetch's response decoder, which honors BOMs, Content-Type, meta and XML declarations and falls back to UTF-8 for unknown labels.

clawsweeper Bot

commented

Oct 30, 2026

•

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

🦞👀

ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper finished reviewing this revision. The review result is being finalized.

There was an error while loading. Please reload this page.

openclaw-barnacle Bot

added

agents

Agent runtime and tooling

size: XS

labels

Oct 30, 2026

clawsweeper Bot

added

clawsweeper:intake:bug

Preliminary submission classification; full source review pending.

clawsweeper:intake:needs-environment

Preliminary submission classification; full source review pending.

P2

Normal backlog priority with limited blast radius.

rating: 🦐 gold shrimp

Decent PR readiness signal, but merge confidence is limited.

status: 📣 needs proof

The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

and removed

clawsweeper:intake:bug

Preliminary submission classification; full source review pending.

clawsweeper:intake:needs-environment

Preliminary submission classification; full source review pending.

labels

Oct 30, 2026

clawsweeper Bot

commented

Oct 30, 2026

•

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Codex review: needs maintainer review before merge.

This PR makes link understanding reuse the existing charset-aware web response decoder.

Example: An agent reads the Shift_JIS page at http://abehiroshi.la.coocan.jp/.

Kind: Bug fix · Worth it: Yes · Fix scope: Complete

User problem: Agents receive unreadable text from linked pages declaring supported legacy encodings.

Reason: The repair restores readable page content through an established decoder without adding configuration, dependencies, or a competing policy.

This PR fixes a current decoding bug with a minimal shared-owner change; the earlier finding is resolved, no new defects were found, and CI is still running.

Priority: P2

Reviewed head: 73ec930363d6b9a218a091b5f0c31f57101b6ea8

Link understanding fetches message URLs through the guarded HTTP transport, decodes bounded response bytes, and supplies text to configured CLI processors and agent input.

flowchart LR A[Message URL] --> B[Guarded HTTP fetch] B --> C[Bounded response bytes] C --> D[Shared charset decoder] D --> E[CLI link processor] E --> F[Agent model input] F --> G[Assistant reply]

Loading

Technical review

Best possible solution:

Merge the shared-decoder repair once the existing CI gates pass.

Do we have a high-confidence way to reproduce the issue?

Current-main source establishes unconditional UTF-8 decoding, and the supplied Gateway comparison demonstrates the resulting garbled title before the repair.

Reusing the existing decoder is the smallest complete fix: it removes the divergent decoding decision while preserving transport limits, cancellation, and permissive fallback.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against e279b64ffe86.

fix(link-understanding): non-UTF-8 pages reach the model as garbled text #188766 (comment): isolated built Gateways fetched public pages through the normal guard, ran a real CLI processor, and carried corrected titles through actual model requests, replies, and history. The deterministic model fixture echoes the supplied title without decoding or repairing it; UTF-8 controls remain unchanged. (73ec930363d6)

Proof applies to the current tree: The tested candidate and reviewed head both resolve to tree 6b9d2585e88508e4f86e5d5c5f78834cf54f390e, verifying the recorded proof's unchanged source inputs. (73ec930363d6)

Earlier finding resolved: The separate regression table was removed and its meaningful charset assertion consolidated into the existing transport case, eliminating the test growth that prompted the earlier allocation finding. (src/link-understanding/runner.transport.test.ts:123, 73ec930363d6)

PR surface

Source +1, Tests 0. Total +1 across 3 files. Added test files: 0.

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 30, 2026, 7:41 PM ET / 23:41 UTC (Revision 2).

There was an error while loading. Please reload this page.

obviyus

and others added 4 commits October 31, 2026 04:54

test(link-understanding): cover charset decoding in the existing tran…

…

86a4b24

…sport case

test(link-understanding): budget the charset regressions

3cf4ae6

Merge remote-tracking branch 'origin/pr-188766-latest' into fix/link-…

…

b8329bd

…understanding-charset

test(link-understanding): remove the unused charset line allocation

73ec930

obviyus

commented

Oct 30, 2026

Copy link

Copy Markdown

Collaborator

Correctness gate and shipped Gateway proof

This is not performance-only: the same public page reaches both the actual model request and the final assistant reply with the wrong characters on main, and readable Japanese after the repair.

The build and test proof ran at 86a4b241c744b6e2de9f70a93fbe23c6a70a692f. Before publication, the guarded push detected the contributor's new allocation commit 3cf4ae615b2f; I merged it intact and removed that now-unused 13-line allocation because the consolidated tests have zero growth. The resulting final source tree is identical to the proven tree: both Git tree IDs are 6b9d2585e88508e4f86e5d5c5f78834cf54f390e. No behavior, source, test, dependency or build input changed; only commit ancestry changed, so the recorded proof is reused rather than rebuilt.

Shipped route: operator WebSocket sessions.create → chat.send → normal OpenClaw agent preprocessing → STRICT guarded public HTTP fetch → real configured CLI link processor → real OpenClaw agent runtime → OpenAI-compatible model fixture → agent.wait terminal reply and chat.history.

The CLI processor reads the real fetched document from stdin, records its decoded title and replacement-character count, and outputs only LINK_TITLE=<JSON title>. The de

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →