Quick start · API · Benchmarks · Configuration · Development

Huntsman is a small, self-hosted web scraper that turns any page into clean Markdown for LLMs and search. It renders JavaScript with Lightpanda, strips ads, trackers and cookie banners, and serves a Firecrawl-compatible /v1/scrape API from one Go binary and one container. No Redis, no Playwright, no Chrome.

Fast and light. On the same 8-vCPU machine it scrapes 4× as many pages per second as Firecrawl, and a 2-vCPU machine handles 45 pages at once.

Clean Markdown. It keeps the main content and drops navigation, footers, ads and cookie banners, using Firecrawl's onlyMainContent rules and the EasyList filters.

Drop-in API. Requests and responses follow Firecrawl's /v1 shapes, so most clients only need a new base URL.

JavaScript when it matters. Lightpanda renders single-page apps and waits until the page goes quiet. PDFs are fetched directly, and pages that block headless browsers are retried as plain HTML.

Batches and crawls. A SQLite-backed queue handles up to 1,000 URLs per batch and whole-site crawls, with a per-host concurrency limit and a result cache.

Safe by default. It blocks private, loopback and link-local addresses at dial time, obeys robots.txt, runs as a non-root user, and supports a bearer token.

Quick start

Run the published image, for linux/amd64 or linux/arm64:

docker run -d --name huntsman -p 127.0.0.1:8080:8080 -v huntsman-data:/data ghcr.io/tulipfarm/huntsman:latest

Scrape a page:

curl -s localhost:8080/v1/scrape
-H 'Content-Type: application/json'
-d '{"url": "https://example.com"}'

{ "success": true, "id": "d91d020c462b68ccb8126dee5a86cfad", "cached": false, "data": { "markdown": "This domain is for use in documentation examples without needing permission. …\n\nLearn more", "links": ["https://iana.org/help/example-domains"], "metadata": { "title": "Example Domain", "sourceURL": "https://example.com/", "statusCode": 200, "fetchedAt": "2026-10-27T06:10:48.552457254Z", "engine": "lightpanda" } } }

In production, pin a version such as ghcr.io/tulipfarm/huntsman:0.1.0, or its digest, and set HUNTSMAN_TOKEN or HUNTSMAN_TOKENS.

git clone https://github.com/TulipFarm/huntsman.git && cd huntsman docker compose up -d --build

API

Endpoint What it does

POST /v1/scrape Scrape one URL and return markdown and links

POST /v1/batch Queue 1–1,000 URLs: {"urls": [...], "scrapeOptions": {...}}

GET /v1/jobs/{id} Status and result of a queued job

POST /v1/crawl Crawl a site: {"url", "limit", "maxDepth", "scrapeOptions"}

GET /v1/crawl/{id}?offset=N Crawl progress and pages

GET /health Liveness and storage check

Scrape options: url, formats (only markdown), waitFor (ms), waitSelector, timeout (ms), maxAge (ms, cache) and onlyMainContent. Unknown fields are rejected, so typos fail loudly instead of being ignored.

Authentication: if HUNTSMAN_TOKEN or HUNTSMAN_TOKENS is set, every endpoint except /health needs Authorization: Bearer <token>. HUNTSMAN_TOKENS gives each client its own key, such as HUNTSMAN_TOKENS=web:t0k3n1,staging:t0k3n2. Each request is logged with its key name, never the token, so you can tell clients apart and revoke one key without the others.

curl -s localhost:8080/v1/crawl
-H 'Content-Type: application/json'
-d '{"url": "https://example.com", "limit": 50, "maxDepth": 2}'

→ {"success": true, "id": "…"}

curl -s 'localhost:8080/v1/crawl/<id>?offset=0'

Coming from Firecrawl

Point your client at Huntsman's base URL and keep using /v1/scrape. Huntsman deliberately does less: it returns Markdown and links only, with no screenshots, HTML, JSON extraction, actions or search. Pair it with a search engine such as SearXNG if you need one.

The most pages each engine can scrape at once, on AWS Graviton (ARM) machines, each engine self-hosted on its own machine. The input is 1,000 URLs sampled from the test split of huntsman-scrape-dataset-v1. Each cell shows the most pages at once that passed, pages per second at that level, and the success rate.

A level passes if the engine doesn't crash or run out of memory, its success rate stays within 10 points of its best level, and its median page time stays under 3× the time at 8 pages at once. A binary search finds the exact limit. Huntsman stopped on page time each time, with every CPU core in use; it never crashed and never ran out of memory. Each figure is a single run against live sites, with robots.txt checks off for every engine and Firecrawl in scrape-only mode. Methods and raw numbers are in benchmark/README.md and benchmark/results/.

huntsman-scrape-dataset-v1 is on Hugging Face under CC BY 4.0: 10,000 URLs balanced across 63 languages, script direction, rendering, page type, region and topic. The pipeline that builds it is in dataset/huntsman-v1/.

src/ the scraper: cmd/huntsman (server) and internal/engine benchmark/ speed, memory and accuracy against Firecrawl, Crawl4AI and Spider dataset/ URL benchmark datasets and the scorer, eval.py site/ the huntsman.tulipfarm.ai landing page (Astro)

Development

You need Go and, for browser tests, a Lightpanda binary.

go build ./... go test ./... # browser tests are skipped without a browser HUNTSMAN_TEST_BROWSER=/path/to/lightpanda go test ./... docker build --target test . # full test suite with Lightpanda inside Docker

Releases

Maintainers release from their own machine; the build runs on GitHub Actions. You need the GitHub CLI and, for a preview of the version and notes, git-cliff.

make release # version from the commit messages: feat → minor (before 1.0), fix → patch make release BUMP=minor # or patch, major, or an exact version such as 1.0.0

The command checks that CI passed on main, shows the changes and next version, and asks before it starts the Release workflow. That workflow:

The script then watches the run until it finishes.

Issues and pull requests are welcome. Read CONTRIBUTING.md and the Code of Conduct first. Please report vulnerabilities privately, as described in SECURITY.md.

Huntsman is licensed under Apache-2.0. The Docker image also ships third-party components under their own licenses: the Lightpanda browser (AGPL-3.0, unmodified, run as a separate process) and the EasyList filter lists (GPLv3 / CC BY-SA 3.0). The Firecrawl dataset is MIT and the Huntsman dataset is CC BY 4.0. See NOTICE for details.