Show HN: Typed data pipeline – in TypeScript and DuckDB
croft is one tool that pulls data from APIs and files, transforms it, keeps it fresh on a schedule, and serves it to your app. Pipelines are TypeScript, with SQL where you want it, and everything lives in one DuckDB fil…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
croft is one tool that pulls data from APIs and files, transforms it, keeps it fresh on a schedule, and serves it to your app. Pipelines are TypeScript, with SQL where you want it, and everything lives in one DuckDB file. No Python, no servers, no glue between tools.
A croft project is a folder of TypeScript and SQL files. Each file becomes one table, and croft runs them in order.
From Stripe, GitHub or any API, or from CSV exports. Incremental, typed, and safe to rerun.
// assets/stripe_charges.ts export default ingest({ key: "id", schedule: "every hour", async *rows({ http }) { … }, });
Transform
Shape it
A SQL query or a TypeScript function per table, with checks that run before anything is saved.
-- assets/revenue_by_month.sql -- check: revenue >= 0 SELECT month, sum(amount) AS revenue FROM stripe_charges GROUP BY month
Schedule
Keep it fresh
A schedule in plain words. Everything downstream updates after it, in order.
$ croft schedule on $ croft status ASSET NEXT stripe_charges every hour revenue_by_month after inputs
Serve
Use it in your app
Query the tables from any Bun or Node app, with no database server to run.
import { query } from "@zabaca/croft/read";
const rows = await query( "from revenue_by_month order by month", );
Typed across SQL and TypeScript
Every table gets a TypeScript type, generated from its real columns. Change a SQL query, and croft checks the TypeScript that reads it, before anything runs.
// .croft/types/revenue_by_region.d.ts, written by croft export type RevenueByRegionRow = { region: string; // VARCHAR, the key orders: number | bigint | null; revenue: number | null; // DOUBLE };
After renaming revenue to total in the SQL:
$ croft validate --types error UNKNOWN_INPUT_COLUMN assets/region_labels.ts:9:65 revenue_by_region has no column "revenue" fix: revenue_by_region has these columns: region, orders, total, _loaded_at
What it replaces
A typical data stack is several tools, in Python and YAML, glued together. croft is one, in TypeScript.
croft does these jobs; it does not read those tools' projects. It is built for one machine and one DuckDB file. If your team already runs a warehouse with dbt, keep it.
Tell Claude Code what you want. It writes the pipeline, and croft checks every step.
Some data can't be fetched again. croft never throws it away on its own.
Each update is saved whole or not at all, and a failed check keeps the old data.
Anything replaced or deleted goes to the trash first, and comes back with one command.
One file on your computer. No account, no server, no cloud to pay for.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。