MiniLake is a local Databricks API emulator with real SQL, real Delta Lake and real Spark job execution. Point databricks-sdk, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at http://localhost:8000 — no account, no API key, no cloud bill.

Copy

$ docker run -p 8000:8000 ghcr.io/dmux/minilake:latest GHCR

Copy

Get started

See it in code

Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.

Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real spark-submit jobs in containers — so a green test locally means something.

Unity Catalog, for real

Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.

SQL that actually runs

Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.

Real Delta Lake files

EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().

Jobs that really execute

spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.

Unity Catalog protocol for Spark

spark.table("cat.sch.tbl") resolves against MiniLake through the official UCSingleCatalog connector — the same protocol Spark speaks to a real workspace.

A built-in SQL workspace

An Athena-style editor at /ui: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab.

Terraform & Asset Bundles

Point the databricks provider at localhost and terraform apply creates real catalogs. Verified on every test run: the real provider is driven through apply, plan and destroy, and a second plan has to come back empty. databricks bundle deploy and bundle run work end to end — jobs execute for real.

An MCP server for agents

67 tools, 8 resources and 4 prompts at /mcp — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.

Designed for CI

One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.

Workspace, DBFS, Files, Secrets

File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks.

Air-gapped by design

DuckDB's delta extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with docker run --network none.

Databricks-shaped errors

Failures come back as {"error_code", "message"}, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud 501 NOT_IMPLEMENTED instead of a convincing lie.

MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.

Any token is accepted. No sign-up, no PAT, no OAuth dance to develop against.

Types round-trip. STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo.

Physical truth wins. Declared column types that disagree with the files lose, so mismatches are visible instead of believed.

Persistence when you want it. MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.

Python SDK Spark & Delta Terraform Asset Bundles curl

quickstart.py

Copy

from databricks.sdk import WorkspaceClient from databricks.sdk.service.catalog import ColumnInfo, TableType

The only line that differs from production

w = WorkspaceClient(host="http://localhost:8000", token="dev")

w.catalogs.create(name="sales") w.schemas.create(name="store", catalog_name="sales") w.tables.create( name="orders", catalog_name="sales", schema_name="store", table_type=TableType.MANAGED, data_source_format="DELTA", storage_location="/data/sales/store/orders", columns=[ ColumnInfo(name="id", type_text="BIGINT"), ColumnInfo(name="customer", type_text="STRING"), ColumnInfo(name="amount", type_text="DECIMAL(10,2)"), ], )

wh = w.warehouses.create(name="dev", cluster_size="Small") w.statement_execution.execute_statement( warehouse_id=wh.id, statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)", )

result = w.statement_execution.execute_statement( warehouse_id=wh.id, statement="SELECT customer, amount FROM sales.store.orders", ) print(result.result.data_array) # [['Ana', '459.90']] — really stored

from pyspark.sql import SparkSession

UC = "http://localhost:8000"

spark = ( SparkSession.builder .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")

Unity Catalog connector — three-part names resolve against MiniLake

.config("spark.sql.catalog.spark_catalog", "io.unitycatalog.spark.UCSingleCatalog") .config("spark.sql.catalog.spark_catalog.uri", UC) .config("spark.sql.catalog.sales", "io.unitycatalog.spark.UCSingleCatalog") .config("spark.sql.catalog.sales.uri", UC) .getOrCreate() )

Real Delta files: parquet + _delta_log/, written by real Spark

spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"])
.write.format("delta").mode("overwrite").save("/data/delta/sales/store/events")

spark.sql("INSERT INTO sales.store.events VALUES (3, 'c')") print(spark.table("sales.store.events").count()) # 3

...and the same bytes are readable through MiniLake's SQL API:

SELECT e.name, o.customer

FROM sales.store.events e JOIN sales.store.orders o ON o.id = e.id

provider "databricks" { host = "http://localhost:8000" token = "dev" }

resource "databricks_catalog" "sandbox" { name = "sandbox" comment = "Local sandbox catalog" }

resource "databricks_schema" "things" { catalog_name = databricks_catalog.sandbox.name name = "things" }

resource "databricks_sql_endpoint" "compute" { name = "local-compute" cluster_size = "Small" }

terraform apply -> creates real catalogs and real DuckDB databases

terraform destroy -> removes them

Resources backed by APIs MiniLake does not emulate fail loudly with 501,

instead of silently pretending to succeed.

bundle: name: my_pipeline

targets: dev: mode: development workspace: host: http://localhost:8000

resources: jobs: daily_load: name: daily_load tasks:

• task_key: main spark_python_task: python_file: ../src/load.py

$ databricks bundle deploy -t dev

$ databricks bundle run daily_load -t dev

The CLI syncs files through the Workspace API, the provider creates the job,

and bundle run executes it on real Spark in a sibling container —

returning the job's actual output.

Is it up, and which services answered?

curl http://localhost:8000/_minilake/health

Create a catalog through the Databricks API surface

curl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs
-H 'Content-Type: application/json'
-d '{"name": "sales"}'

Run SQL and get real rows back

curl -X POST http://localhost:8000/api/2.0/sql/statements
-H 'Content-Type: application/json'
-d '{"warehouse_id": "dev", "statement": "SELECT 42 AS answer"}'

Give the next test a clean workspace — no restart needed

curl -X POST http://localhost:8000/_minilake/reset

How it works One process. One port. Real engines underneath.

A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.

CLIENTS

databricks-sdkWorkspaceClient (Python)

Databricks CLIbundle deploy · bundle run

Terraform providerapply · destroy

PySpark / JupyterLabspark.table("cat.sch.tbl")

LLM agentsModel Context Protocol

BrowserSQL workspace at /ui

MINILAKE · ONE PROCESS

MiniLake FastAPI · uvicorn · :8000

/api/2.0 · /api/2.1 Databricks REST surface

/ui · /jupyter SQL workspace + notebooks

/mcp 67 tools for agents

/_minilake/* health · ready · reset

REAL ENGINES

DuckDB one database file per catalog

Delta Lake on disk parquet + _delta_log/

Docker → Apache Spark spark-submit, sibling container

Spark writes the files — DuckDB reads the same bytes

For LLM agents Give an agent a lakehouse it can't break.

A Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server closes those gaps — and it costs nothing when the agent gets it wrong.

Plus 8 resources — including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB — and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's error is exactly the error an SDK client would have seen.

One line to register it with Claude Code

claude mcp add --transport http minilake http://localhost:8000/mcp

Then just ask:

"Use minilake: create a sales catalog with a seeded

orders table, and tell me the total."

The agent does this — three calls, no glue code:

setup_fixture({"catalog": "sales", "schema": "store"})

seed_table({"full_name": "sales.store.orders", ...})

run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})

Enable it on the server side:

MINILAKE_MCP=1 minilake --port 8000

Coverage What's emulated, and how honestly.

Every row below is either really executed or clearly labelled. Anything out of scope answers 501 NOT_IMPLEMENTED rather than faking a success your production code will not get.

These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.

Any token is accepted and there is exactly one user. Never expose it beyond localhost.

The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.

Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.

The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.

DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.

They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.

Being explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.

Free and MIT-licensed. No account, no API key, no cloud bill — and nothing to uninstall from your cloud account when you're done.