
An AI agent read a Shift_JIS page as 208,953 replacement characters
fix(link-understanding): non-UTF-8 pages reach the model as garbled text - #188766
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Code
Issues5k+ (5k+)
Pull requests2.9k (2.9k)
Actions
Projects
Security and quality722 (722)
Insights
More items
fix(link-understanding): non-UTF-8 pages reach the model as garbled text - #188766
#188766
Merged
obviyus merged 5 commits intoopenclaw:mainopenclaw/openclaw:mainfrom Yigtwxx:fix/link-understanding-charsetYigtwxx/openclaw:fix/link-understanding-charsetCopy head branch name to clipboard
Oct 30, 2026
ConversationCommits5 (5)ChecksFiles changed
Merged
fix(link-understanding): non-UTF-8 pages reach the model as garbled text#188766
obviyus merged 5 commits intoopenclaw:mainopenclaw/openclaw:mainfrom Yigtwxx:fix/link-understanding-charsetYigtwxx/openclaw:fix/link-understanding-charsetCopy head branch name to clipboard
Conversation
Yigtwxx
commented
Oct 30, 2026
•
edited by obviyus
Loading
Uh oh!
There was an error while loading. Please reload this page.
Copy link
Copy Markdown
Contributor
What Problem This Solves
Fixes: linked pages served in a declared legacy charset reach link processors and agent input as garbled UTF-8 replacement characters. This changes incorrect user-visible output, not just allocation or performance.
User impact: an agent reading a linked Shift_JIS page now receives and returns its readable Japanese title. UTF-8 pages are unchanged. Unknown charset labels keep the existing non-fatal UTF-8 fallback; no configuration change is needed.
Reuse the response decoder already owned by web_fetch instead of the link fetcher's unconditional UTF-8 decoder. This preserves the existing BOM/header/document declaration order without adding another parser. The added export is internal, not a Plugin SDK surface. Fetch authority, cancellation, response limits, processors, prompt definitions and stored conversation history contracts are unchanged.
@Yigtwxx's original commit is preserved. The separate 13-line test table is consolidated into the existing transport/URL-selection regression; the shared decoder retains its declaration and fallback coverage. Production +1, tests 0, with no test-line allocation or ceiling change. Earlier closed attempt: #111773.
Built Gateway before/after evidence, native receipts and replay details.
Built Gateway agent turns on current-main baseline 1d2e0b28b60b and candidate 86a4b241c744: two Shift_JIS turns and two UTF-8 controls on each. Real public HTTP pages pass through the normal STRICT guarded fetch, a real CLI processor and the real OpenClaw agent runtime. A deterministic OpenAI-compatible model fixture records the actual input and echoes its title; no paid live-model claim.
Current head 73ec930363d6 has the identical source tree as the proven candidate (6b9d2585e88508e4f86e5d5c5f78834cf54f390e): the contributor's intervening allocation commit was preserved, then its unused allocation removed. All proof inputs are unchanged.
Main's Shift_JIS input and final replies are garbled twice (13 replacement characters each); the candidate produces 阿部寛のホームページ twice with 0 replacement characters. The UTF-8 control remains 気象庁 Japan Meteorological Agency throughout. All eight turns complete with one model request each, no rerouting, native visible-delivery receipts, matching assistant history and clean Gateway shutdown.
Focused tests: 33 passed (transport 5, shared decoder 14, prefix stability 14). The same final transport suite on main fails only the intended decoding regression (4 passed / 1 failed). Type-aware scoped lint, format and whitespace checks pass.
Two-way door: a revert fully restores the old decoder; there is no migration, dependency, schema, credential or configuration change. Blast radius is decoded linked-page text supplied to configured link processors and agent input; UTF-8 fallback and the shared web_fetch decoder remain the existing owners.
There was an error while loading. Please reload this page.
fix(link-understanding): non-UTF-8 pages reach the model as garbled text
…
38de2ab
Link understanding decoded every fetched page with a bare UTF-8 TextDecoder, so pages declaring Shift_JIS, ISO-8859-1 or another charset reached processors and the model as replacement characters. Reuse web_fetch's response decoder, which honors BOMs, Content-Type, meta and XML declarations and falls back to UTF-8 for unknown labels.
clawsweeper Bot
commented
Oct 30, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Copy link
Copy Markdown
Contributor
🦞👀
ClawSweeper picked this up.
Pull request received. I will update this pull request when review starts.
ClawSweeper finished reviewing this revision. The review result is being finalized.
There was an error while loading. Please reload this page.
openclaw-barnacle Bot
added
agents
Agent runtime and tooling
size: XS
labels
Oct 30, 2026
clawsweeper Bot
added
clawsweeper:intake:bug
Preliminary submission classification; full source review pending.
clawsweeper:intake:needs-environment
Preliminary submission classification; full source review pending.
P2
Normal backlog priority with limited blast radius.
rating: 🦐 gold shrimp
Decent PR readiness signal, but merge confidence is limited.
status: 📣 needs proof
The PR needs real behavior proof before ClawSweeper can clear the contributor ask.
and removed
clawsweeper:intake:bug
Preliminary submission classification; full source review pending.
clawsweeper:intake:needs-environment
Preliminary submission classification; full source review pending.
labels
Oct 30, 2026
clawsweeper Bot
commented
Oct 30, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Copy link
Copy Markdown
Contributor
Codex review: needs maintainer review before merge.
This PR makes link understanding reuse the existing charset-aware web response decoder.
Example: An agent reads the Shift_JIS page at http://abehiroshi.la.coocan.jp/.
Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Agents receive unreadable text from linked pages declaring supported legacy encodings.
Reason: The repair restores readable page content through an established decoder without adding configuration, dependencies, or a competing policy.
This PR fixes a current decoding bug with a minimal shared-owner change; the earlier finding is resolved, no new defects were found, and CI is still running.
Priority: P2
Reviewed head: 73ec930363d6b9a218a091b5f0c31f57101b6ea8
Link understanding fetches message URLs through the guarded HTTP transport, decodes bounded response bytes, and supplies text to configured CLI processors and agent input.
flowchart LR A[Message URL] --> B[Guarded HTTP fetch] B --> C[Bounded response bytes] C --> D[Shared charset decoder] D --> E[CLI link processor] E --> F[Agent model input] F --> G[Assistant reply]
Loading
Technical review
Best possible solution:
Merge the shared-decoder repair once the existing CI gates pass.
Do we have a high-confidence way to reproduce the issue?
Current-main source establishes unconditional UTF-8 decoding, and the supplied Gateway comparison demonstrates the resulting garbled title before the repair.
Reusing the existing decoder is the smallest complete fix: it removes the divergent decoding decision while preserving transport limits, cancellation, and permissive fallback.
AGENTS.md: found and applied where relevant.
Codex review notes: model internal, reasoning medium; reviewed against e279b64ffe86.
fix(link-understanding): non-UTF-8 pages reach the model as garbled text #188766 (comment): isolated built Gateways fetched public pages through the normal guard, ran a real CLI processor, and carried corrected titles through actual model requests, replies, and history. The deterministic model fixture echoes the supplied title without decoding or repairing it; UTF-8 controls remain unchanged. (73ec930363d6)
Proof applies to the current tree: The tested candidate and reviewed head both resolve to tree 6b9d2585e88508e4f86e5d5c5f78834cf54f390e, verifying the recorded proof's unchanged source inputs. (73ec930363d6)
Earlier finding resolved: The separate regression table was removed and its meaningful charset assertion consolidated into the existing transport case, eliminating the test growth that prompted the earlier allocation finding. (src/link-understanding/runner.transport.test.ts:123, 73ec930363d6)
PR surface
Source +1, Tests 0. Total +1 across 3 files. Added test files: 0.
6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.
ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.
Reviewed October 30, 2026, 7:41 PM ET / 23:41 UTC (Revision 2).
There was an error while loading. Please reload this page.
obviyus
and others added 4 commits October 31, 2026 04:54
test(link-understanding): cover charset decoding in the existing tran…
…
86a4b24
…sport case
test(link-understanding): budget the charset regressions
3cf4ae6
Merge remote-tracking branch 'origin/pr-188766-latest' into fix/link-…
…
b8329bd
…understanding-charset
test(link-understanding): remove the unused charset line allocation
73ec930
obviyus
commented
Oct 30, 2026
Copy link
Copy Markdown
Collaborator
Correctness gate and shipped Gateway proof
This is not performance-only: the same public page reaches both the actual model request and the final assistant reply with the wrong characters on main, and readable Japanese after the repair.
The build and test proof ran at 86a4b241c744b6e2de9f70a93fbe23c6a70a692f. Before publication, the guarded push detected the contributor's new allocation commit 3cf4ae615b2f; I merged it intact and removed that now-unused 13-line allocation because the consolidated tests have zero growth. The resulting final source tree is identical to the proven tree: both Git tree IDs are 6b9d2585e88508e4f86e5d5c5f78834cf54f390e. No behavior, source, test, dependency or build input changed; only commit ancestry changed, so the recorded proof is reused rather than rebuilt.
Shipped route: operator WebSocket sessions.create → chat.send → normal OpenClaw agent preprocessing → STRICT guarded public HTTP fetch → real configured CLI link processor → real OpenClaw agent runtime → OpenAI-compatible model fixture → agent.wait terminal reply and chat.history.
The CLI processor reads the real fetched document from stdin, records its decoded title and replacement-character count, and outputs only LINK_TITLE=<JSON title>. The de
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。