返回资讯中心
外部精选
软件工程
#开发者实时流

DuckDB-WASM startup: from 6.8 to 2.3 seconds with Haybarn-WASM

Haybarn-WASM, a drop-in fork of DuckDB-WASM, loads five extensions in 1.9 seconds, against 5.4 for upstream. Here's where the startup time went and how we cut it.

Hacker News 最新rustyconover2026/10/19 · 27 天前26 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Haybarn-WASM, a drop-in fork of DuckDB-WASM, loads five extensions in 1.9 seconds, against 5.4 for upstream. Here's where the startup time went and how we cut it.

Headless Chromium 156 on an AWS c6g.12xlarge, five interleaved runs per build, each in a fresh browser. See Methodology.

Cupola, our browser-based environment for querying and exploring data, runs on Haybarn-WASM, and its users were waiting several seconds before they could do anything. I was waiting too. I use Safari with the multithreaded build enabled, and every page load meant watching the engine download and compile, then load each extension, one at a time.

The multithreaded build should be the better choice, since queries can use every core. On our test machine it was the slower one to start. Haybarn-WASM 1.5.6-rc2 took 6.8 seconds to get a page to six loaded extensions (icu, autocomplete, spatial, iceberg, grainlift and ducklake), and the single-threaded build took 5.4.

With 1.5.6-rc6, it’s 2.3 seconds threaded and 2.0 single-threaded. This post covers where the time went, what we changed, and the one call you need to add to your code to get all of the benefits.

Haybarn-WASM is Query.Farm’s fork of DuckDB-WASM, built on Haybarn, our DuckDB distribution. The JavaScript API is the same, so if your app uses @duckdb/duckdb-wasm, switching is a package swap:

npm install @haybarn/haybarn-wasm

// before

import * as duckdb from '@duckdb/duckdb-wasm';

// after

import * as duckdb from '@haybarn/haybarn-wasm';

AsyncDuckDB, selectBundle, ConsoleLogger and the rest keep their names. The only addition is a call to prefetchExtensions() during setup, covered below.

I compared the two on one machine, loading the five extensions both projects publish for WebAssembly (icu, autocomplete, spatial, iceberg and ducklake), each from its own project’s extension repository:

A few practical details about switching, like where to load the files from, are in notes on switching at the end.

If you already use Haybarn-WASM, upgrading gets you the middle column, and adding the prefetch call gets you the last:

These are medians of five fresh browsers on an AWS Graviton2 machine, with extensions from our real CDN. Graviton2 runs this workload about three times slower than an Apple M3, so a laptop is faster across the board. Details are at the end.

I expected the time to be in DuckDB, or in our own worker code. A large share of it was in emscripten, the toolchain that compiles DuckDB’s C++ to WebAssembly and supplies the runtime code that links extensions in. That was the biggest surprise of this work.

An extension is compiled separately from the engine. When it calls into DuckDB, to register a function or read a column, it refers to that code by its symbol: the name a compiled function or variable is known by. The linker matches those names to addresses when the extension loads.

The engine exports about 48,000 C++ symbols for extensions to use. emscripten’s startup code walked all of them before the first query could run, in the main worker and again in every pthread worker, so each extra thread made startup longer. It now looks up a symbol the first time something asks for it. On the threaded build, that cut the time to the first query result by 43%.

emscripten had two smaller costs that applied to every extension. To find out whether each symbol an extension exports could be written to, it threw and caught an exception; spatial exports 1,286 symbols. It now reads the answer from the module. It also kept each extension’s file in memory after linking it, 26 MB for spatial, and it now lets that go.

A LOAD on the threaded build runs with one thread, because a blocking download while other threads run can deadlock. After each LOAD the thread count went back up, and every new thread had to load all the extensions loaded so far before the next LOAD could start. Across six extensions that added about 460 ms. Consecutive LOADs now stay on one thread, and the threads come back before your next query.

The extension loader also copied every file into shared memory one byte at a time before linking it. It now passes the downloaded bytes to the linker directly.

These changes make up the middle column: about three seconds on both builds. What remained was the extensions waiting in line.

LOAD runs inside a SQL statement, and SQL in DuckDB-WASM runs synchronously in the worker. Each LOAD downloads its extension with a blocking request, hashes the file, checks the signature, compiles the module, and only then lets the next one start. While the engine hashes and compiles, the network sits idle.

I hadn’t expected the hashing to matter. The signature check needs a SHA-256 of the whole file, and the engine computes it with its own SHA-256 code, compiled to WebAssembly. The files are larger than their downloads, because they arrive compressed: spatial is 4.8 MB over the network and 26 MB to hash. The browser’s WebCrypto SHA-256 is native code and more than six times faster on the same bytes. For the eight files in our list, the engine spends 400 ms hashing, one file at a time:

A synchronous SQL statement can’t wait on WebCrypto, or on several downloads at once, so that work has to happen before the LOADs. That’s what prefetchExtensions() in rc6 does:

const db = new duckdb.AsyncDuckDB(logger, worker);

await db.instantiate(bundle.mainModule, bundle.pthreadWorker);

await db.open({});

// While setting up the session, before running any queries:

// iceberg loads avro and parquet itself, so prefetch those too.

await db.prefetchExtensions(['icu', 'autocomplete', 'spatial', 'iceberg', 'avro', 'parquet', 'grainlift', 'ducklake']);

const conn = await db.connect();

await conn.query('LOAD spatial'); // much quicker now

The worker downloads every extension in parallel, hashes each one with WebCrypto, and compiles it while it downloads. When LOAD spatial runs, the file is already there. The engine checks the signature against that hash and links the module compiled from the same bytes. Here’s the six-extension load on rc6, with and without prefetch, drawn to scale:

The signature check itself hasn’t moved. The public key and the RSA verification stay in the engine, and a module reaches the linker only after its bytes pass. I tested it with an extension that had one byte flipped, and it was rejected with and without prefetch, on both builds.

My first prefetch measurements had one outlier. Most LOADs dropped to a quarter or a third of their time, but iceberg only went from 550 ms to 379. LOAD iceberg also loads avro and parquet, and since those weren’t in the prefetch list, iceberg downloaded both itself, one after the other. With them in the list, LOAD iceberg takes 197 ms:

Each figure is the median of five runs over the real CDN, and the totals add up the medians. The download sizes are the compressed single-threaded files. What’s left in each LOAD is linking the module and running the extension’s own setup code, which can’t happen ahead of time. That’s why spatial still takes a few hundred milliseconds.

Cupola uses it now, so you can open it and watch the startup yourself. When Cupola loaded nine extensions, moving from rc5 to rc6 with prefetch took it from 4.7 to 3.7 seconds until it was ready to use. It has since added avro and parquet to its startup list, and they’re prefetched with the rest.

I expected prefetch to cost memory, since it holds each downloaded file until its LOAD takes it. That matters in a browser: a tab that runs out of memory gets killed, and phones hit that limit early. Memory went down instead:

WebAssembly memory can grow but never shrink, so anything copied into it stays for the life of the page. The old loader copied each extension file into that memory before linking it. Without those copies, the threaded build uses 105 MB less with six extensions loaded. About half of that saving exists before any extension loads, because the engine now starts 55 MB smaller.

Prefetch doesn’t change where you end up. Right after prefetching all eight files, the threaded build was at 312 MB, and with everything loaded it reached the same 499 MB with or without prefetch.

Chrome’s memory measurement can’t see the single-threaded build’s WebAssembly memory, so I measured that engine in Node.js. Loading five extensions grew the process by 206 MB, down from 343 MB.

One more emscripten fix, still in review, frees the copy of each extension that every thread keeps. In earlier tests it saved another 90 MB with these six extensions loaded.

Chrome can keep compiled WebAssembly between visits, so a returning user shouldn’t have to compile the engine again. It wasn’t doing that for DuckDB-WASM, because the code that reported download progress wrapped the response in a way that hid it from Chrome’s cache. With that fixed, opening the database on a repeat visit takes 87 ms instead of about 230. Extensions compiled during a prefetch are cached too: on a repeat visit, the prefetch and six LOADs take 1,009 ms instead of 1,362, and the page has all six loaded at 1.8 seconds instead of 2.2.

Two caveats. Current Chrome waits minutes before writing that cache, because of a V8 bug whose fix has landed and will ship in a coming release. And Safari doesn’t keep compiled WebAssembly between visits at all, so in Safari, where I first noticed the wait, every visit is a first visit. That’s why the first-visit work above mattered most to me.

Most of these fixes aren’t specific to Haybarn, and I’ve sent them upstream:

They’re open for review. Until they land, Haybarn-WASM carries them as patches. prefetchExtensions() depends on how Haybarn checks extension signatures, so for now it’s only in Haybarn-WASM.

None of the extensions loaded on DuckDB-WASM’s multithreaded build. By default LOAD never returned, and with the thread count set to one first, it failed with a LinkError. Those extension binaries are built without shared memory, which the threaded engine needs. Haybarn builds its threaded extensions with shared memory and drops to one thread around each LOAD for you.

Query Farm's adbc_scanner uses DuckDB 2.0 whole-query pushdown to run supported queries entirely in PostgreSQL. In our benchmark, it was 10–11× faster.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →