Highlighting Code: Pierre Computer Company

代 代 代 代

码 码 码 码

高 高 高 高

亮 亮 亮 亮

今 今

天 天

就 就

让 让

你 你

的 的

代 代

码 码

高 高

亮 亮

起 起

来 来

高

亮

{ } </>

代码高亮

今 天 就 让 你 的 代 码 高 亮 起 来

高

亮

(CLICK TO HIGHLIGHT ART – SEIZURE WARNING)

SOUND ON

Today, we’re excited to announce the public release of a new library we call Highlights.

Highlights is a syntax highlighter written in WebAssembly Text, or WAT. It supports over 70 built-in languages, uses Zed-compatible themes, and produces HTML or themed tokens. It also handles streaming input and incremental edits, so applications can keep highlighting as their code changes.

Highlights generates HTML at up to 298× Shiki’s throughput across the large-file fixtures below.

Bun 1.4.0 · Apple M4 Pro · October 5, 2026 · — not measured

These recorded results are from October 5, 2026, using Bun 1.4.0 on an Apple M4 Pro. Highlights includes UTF-8 encoding and HTML decoding; Shiki is the string-API baseline, and Tree-sitter uses tree-sitter-highlight 1.1.2. Each case gets a 200 ms warm-up and a 1.5-second sampling budget, with at least 20 samples and a rotating contender order. Throughput uses the median of per-call averages from roughly 5 ms batches. To reproduce these results, check out the benchmark.

Syntax highlighting is surprisingly expensive.

Historically in Diffs we’ve used the excellent Shiki library and then relied on workers to help keep this work off the main thread, and caches to help avoid repeating it.

This has been fine, but particularly as we’ve invested more into our edit functionality, syntax highlighting has become more and more an unavoidable bottleneck for us.

A highlighter breaks code into tokens. In const greeting = "hello";, it recognizes const as a keyword and "hello" as a string, then uses a theme to assign colors. It can return tokens for a renderer or HTML with styled spans. Identifying the tokens takes most of the work.

regex is the death of performance. - @aidenybai

Shiki, Prism, highlight.js, Sugar High, and Rangi all use regular expressions to match tokens. Regex gives them a compact way to describe what a keyword, number, or quoted string looks like. In engines such as Prism, adding a language means defining those patterns and letting the same engine do the matching. That makes language support easier to build and extend.

The cost comes from how much matching happens. Different rules can scan the same stretch of code, each looking for its own pattern. Complex expressions can also backtrack: when a match fails, the engine revisits characters and tries another path. A single regex can be fast, but applying a whole grammar means paying for successful matches, failed attempts, and repeated scans. Across large files or code highlighted after every edit, that work adds up.

Tree-sitter takes another approach. It parses source code into a concrete syntax tree, which retains details such as punctuation, then uses queries to assign highlight names to nodes in that tree. A theme maps those names to styles.

That structure lets highlighting rules distinguish a function name from a parameter by its place in the code. Building, storing, and querying the tree takes time and memory, but the tree can also support features such as code navigation. After an edit, Tree-sitter can update it incrementally and reuse unchanged parts. A full-file benchmark measures a different workload from maintaining a tree while someone types.

Recently, Shu Ding introduced gpu-lexer from Vercel Labs, an experimental highlighter that uses a small learned model running on WebGPU. One model handles different languages without requiring a language argument or a separate grammar for each one.

The CPU first splits the source into simple parts, such as words, whitespace, and symbols. The GPU processes features from those parts and their context, then predicts one of nine visual classes. The public API, await parse(code), returns classified spans with JavaScript string offsets. The project README describes that pipeline and its requirement for WebGPU in a secure browser context.

The learned approach is interesting because it can attempt to highlight a language it has never been explicitly configured to support. Its quality depends on what the model learned and how well that transfers to the input. The model card explains that evaluation measures agreement with Shiki's labels, rather than semantic correctness.

Chromium 153 · Apple M4 Pro · October 21, 2026

The chart shows the October 21, 2026 run with Highlights 0.2.0 from npm: median times from three passes, with one warm-up in a fresh worker for each sample; initialization and DOM rendering are excluded. Highlights and Shiki use codeToTokens(), returning themed JavaScript tokens. gpu-lexer returns classified spans; Starry Night returns a HAST tree; Sugar High, Prism.js, and Highlight.js return HTML. The recorded samples include versions and run conditions. This measures time for those API paths on one large JavaScript input; it does not measure highlighting quality.

JavaScript handles the public API, input encoding, and theme preparation. The WebAssembly module scans the source and emits HTML or token records. There is no syntax tree to construct along the way. The architecture notes describe the boundaries in more detail.

A lexer walks through source text and identifies meaningful ranges: a keyword, a number, a comment, a string. In Highlights, these lexers are written by hand in WAT and share scanners for common operations.

More complicated languages need more state. Embedded languages also need a way to hand control to another lexer. The JavaScript lexer, for example, can scan an HTML

<script> region within its bounds, then return control to the HTML lexer. Supported Markdown code fences work through the same general idea. This makes language support explicit. Adding a language means implementing and maintaining its lexer. Existing TextMate grammars cannot simply be loaded into Highlights, and the output will not always classify code exactly as Shiki or Tree-sitter would. For example, in the TypeScript statement const greeting = 1;, Highlights classifies greeting as a variable, while Shiki assigns it the TextMate scope variable.other.constant.ts. With their bundled GitHub Dark themes, that identifier uses the foreground color in Highlights and blue in Shiki. If matching existing highlighting matters, compare representative files before switching. A lot of source code consists of stretches where nothing interesting happens. Inside a line comment, the lexer mostly needs to find the next line ending. Inside a quoted string, it can skip ahead until it encounters a quote, an escape, or another character that needs special handling. Highlights uses WebAssembly SIMD for several of these scans. SIMD lets one instruction operate on multiple values at once; here, a scanner can load 16 bytes and compare all of them against the characters it is looking for. The comparisons produce a bit mask, which identifies the first matching position. That lets the scanner advance through a long comment 16 bytes at a time. Short, common cases still matter: whitespace scanning has a scalar path for the small gaps between tokens, avoiding SIMD setup when there is little to scan. Keyword lookup also uses tables prepared during the build. The lookup hashes selected bytes and the word length, then checks the candidate's exact bytes. Language keywords become data the lexer can look up directly. For HTML, the emitter writes escaped text and styled spans directly into WebAssembly memory. JavaScript prepares the theme ahead of time; ordinary color themes become a compact table containing color and font settings for each token type. Neighboring ranges with the same style can share a span, including the whitespace between them. Span opening tags are cached for reuse. These details reduce both the work of producing HTML and the amount of markup the browser receives. The API exposes that output as UTF-8 bytes: import { codeToHtml } from '@pierre/highlights'; import { pierreDark } from '@pierre/highlights/themes'; const bytes = codeToHtml('const greeting = "hello";', { lang: 'js', theme: pierreDark, }); const html = new TextDecoder().decode(bytes); The package runs in Node.js, browsers, and Cloudflare Workers. It initializes WebAssembly automatically, so highlighting calls are synchronous after import. Highlights accepts Zed-compatible theme objects. You can import a bundled theme, such as pierreDark or githubLight, from @pierre/highlights/themes, or supply your own. Highlighting a complete file quickly helps, but live interfaces also need to handle text that is still arriving or being edited. StreamTokenizer accepts chunks of text or UTF-8 bytes and emits themed tokens for completed lines. It preserves lexer state between chunks, so a comment or string can continue into the next chunk without rescanning earlier lines. An incomplete final line stays buffered until a newline arrives or the caller finishes the stream with end(). import { StreamTokenizer } from '@pierre/highlights'; import { pierreDark } from '@pierre/highlights/themes'; const response = await fetch('https://example.com/source.ts'); const lines = response.body.pipeThrough( new StreamTokenizer({ lang: 'ts', theme: pierreDark }) ); for await (const tokens of lines) { console.log(tokens); } LiveTokenizer handles edits to existing text. It keeps the document, per-line token records, and saved lexer states in its own WebAssembly instance. After an edit, it tokenizes the affected lines and continues until the outgoing state matches the previously saved state at an unchanged boundary. Beyond that point, the existing results can be reused. import { LiveTokenizer } from '@pierre/highlights'; import { pierreDark } from '@pierre/highlights/themes'; const live = new LiveTokenizer({ code: 'let count = 1;\nconsole.log(count);', lang: 'js', theme: pierreDark, }); live.applyEdits([{ range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, newText: '2', }]); console.log(live.getLineText(0)); // let count = 2; console.log(live.getLineTokens(0).tokens); live.dispose(); Some edits still affect the whole file. Opening a multiline comment near the top can change how every following line is tokenized. With renderRange, the live tokenizer processes lines through the end of the visible range, including preceding dirty lines needed to establish its lexer state. It finishes the remaining work in background slices and delivers those tokens through onDeferTokenize. The editor can update visible lines while the rest catches up. WAT is the text representation of WebAssembly instructions. Writing it by hand means working directly with loads, stores, branches, and vector operations. Which isn't the most fun… Rust and Zig can both compile to WebAssembly and provide control over memory and SIMD. However, our preference for WAT is about the workflow: the source expresses the loads, comparisons, and branches we want to inspect and tune. For example, the 16-byte comment scan is written directly as a vector load, comparisons, and a bit-mask check. With a program built mostly from scans like this, working in that instruction representation is useful. The choice also gives us more details to maintain ourselves. Writing every table and constant by hand would make the lexers harder to maintain. Our preprocessor adds a few forms to the source and expands them into standard WAT during the build. Local imports bring shared code