Lumen 1.4.0 Release

Lumen is an open-source framework for conversational data exploration and analysis. You ask a question about your data, and an LLM proposes the SQL, the transformations and the chart. Every step stays visible, editable and re-runnable. Lumen is built on Panel and the rest of the HoloViz ecosystem, so the results are interactive tables, plots and dashboards. If you are new to it, the Lumen 1.0 announcement explains the design.

We are excited to announce the 1.4.0 release of Lumen! This release makes Lumen easier to start and cheaper to run well. You no longer need an API key to try it, answers rest on better table context, and a report can write its own story. Here’s a high-level overview of the highlights:

Lumen 1.4 requires Python 3.11 or later, Panel 1.9 or later and Bokeh 3.9 or later.

Many thanks to the contributors to this release: @ghostiee-11, @philippjfr, @ahuang11, @vignesh-chaturvedi, @SajalDevX, @DevForge255, @7487, @rimsha152007-dotcom, @fly1d, @Vamshi1404, @hoxbro, @AmirF194, @yiboban, @raashish1601, @nightcityblade, @mayuriphad, @mahdibani, @latent-9, @Zayn329, @SuMayaBee, @Mr-Neutr0n, @Coderambling, @umi008, @jadhavgaurav.

A special welcome to our new contributors: @vignesh-chaturvedi, @SajalDevX, @DevForge255, @7487, @rimsha152007-dotcom, @fly1d, @Vamshi1404, @AmirF194, @yiboban, @raashish1601, @nightcityblade, @mayuriphad, @mahdibani, @latent-9, @Zayn329, @Mr-Neutr0n, @umi008, @jadhavgaurav.

Without an API key, lumen-ai serve used to start against a provider that was not there. In 1.4 there are four ways to get going without one.

If you are signed in to Claude Code, Codex or the GitHub Copilot CLI, Lumen can use it as its LLM:

lumen-ai serve penguins.csv --provider claude-code

The other values are codex-cli and copilot-cli, and antigravity-cli is available too. The Claude Code and Copilot providers run the CLI with its own tools switched off and in an empty temporary directory, so it answers Lumen’s prompts without reading the folder you launched from.

One limit: Lumen’s own tools are not forwarded to these CLIs. An action that needs the model to call a tool, such as fetching a URL from the chat, cannot run, so pass your data on the command line.

lumen-ai serve penguins.csv --provider kilo

Kilo’s free tier needs no account. It is a shared service, so expect it to be slow and to have limits. In our tests a question that ended in a chart took five to six minutes, and after roughly 40 minutes of steady use it answered with a 429 that points to a free sign-up. The free router also changes which model it uses over time, and some of those models reject the structured requests the planner makes. When Kilo answers with an upstream error, Lumen now falls back to asking for JSON in plain text and keeps doing that for the session.

For a local model, Lumen reads OLLAMA_HOST, and the local providers now default to a 9B model, which you can fetch with ollama pull qwen3.5:9b.

If nothing is set up, Lumen no longer guesses. It picks a provider only when its API key is set, and otherwise prints what would work on your machine:

ERROR: No LLM provider found: no API key is set. Pick one of these to get started:

• Free models, no signup: --provider kilo
• Use a coding CLI you are signed in to: --provider claude-code
• Sign in to one of these coding CLIs first: --provider copilot-cli, --provider antigravity-cli
• Set an API key, e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY or GEMINI_API_KEY
• Use an OpenAI-compatible server: --provider openai --provider-endpoint URL
• Ollama is running with no models: ollama pull qwen3.5:9b

Docs: https://lumen.holoviz.org/configuration/llm_providers/

Maps and Geospatial Data

A table that has a geometry column can now be drawn as a map directly. Start Lumen on a GeoJSON file, a WKT file or a zipped shapefile, and ask for the map:

lumen-ai serve countries.geojson --provider claude-code

Map each country’s estimated population as a choropleth on a log colour scale using the Equal Earth projection, with a legend.

The chart is a plain Vega-Lite spec, shown above the map. If something is off, edit the YAML or ask for the change.

Maps are interactive: hover a state to see its values, and the view follows the light or dark theme.

Around that feature, geospatial support got a round of fixes:

Maps are one case of a general ability: describe a chart and Lumen writes the SQL and the spec. Here is a prompt with several moving parts, run on the Gapminder data:

For the ten most populous countries in 2005, draw a connected scatter plot of fertility against life expectancy from 1955 to 2005, with bubbles sized by population at each point and the country name at the 2005 end of each path.

A language model writes each chart, so a rerun gives a different one, and a first pass sometimes needs a follow-up. In this one, a few labels near the bottom right overlap. The specs and SQL behind this figure are in the repro/ folder of this post.

Follow-ups work on the chart you already have. Here Lumen draws UK electricity per capita from fossil fuels, nuclear and renewables as a stacked area, and a follow-up can redraw it, for example as a line chart comparing just fossil fuels and renewables:

The Vega-Lite agent also got better at stacking and ordering: asked for the US athletes with the most Olympic medals, it draws a stacked bar by medal type (#1992).

Two more prompts on data-centre electricity demand: a slope chart that highlights the United States in red, and a ranking bar chart that labels each bar with its value.

A Lumen report collects the tables and charts from your explorations. In 1.4 you can ask it to write the story around them. The Annotate button uses an LLM to write an overall summary at the top and a short note under each section. The narrative shows in the report and flows into the Notebook and HTML exports.

A report can now also be exported as a Word document. The .docx carries the headings, the narrative, native tables and the charts as images, respects the section selection, and includes the story and notes (#1924). The story’s prose is editable with Panel’s TextEditor (#2017), so you can fix a sentence instead of regenerating the whole thing. The Word export needs python-docx, which Lumen now installs.

Most of the engine work in this release is about giving the model better facts and catching mistakes cheaply.

The table context agents see used to be built from a shuffled 100-row sample with no SQL types or keys. It could list wrong ranges and incomplete value lists, it dropped NULLs, and it changed on every run, which defeated prompt caching. It is now profiled with the cheapest accurate strategy for each source: engine metadata first, one exact pass for tables up to a million rows, and a seeded sample above that (#2105). This is what the model sees for the penguins data:

import asyncio

from lumen.ai.schemas import get_metaset from lumen.sources.duckdb import DuckDBSource

url = "https://datasets.holoviz.org/penguins/v1/penguins.csv" source = DuckDBSource( uri=":memory:", tables={"penguins": f"read_csv('{url}', nullstr='NA')"}, ) print(asyncio.run(get_metaset([source], ["penguins"])).compact_context())

penguins (344 rows) species VARCHAR {Adelie:152, Gentoo:124, Chinstrap:68} island VARCHAR {Biscoe:168, Dream:124, Torgersen:52} bill_length_mm DOUBLE 32.1..59.6 nulls <1% bill_depth_mm DOUBLE 13.1..21.5 nulls <1% flipper_length_mm BIGINT 172..231 nulls <1% body_mass_g BIGINT 2700..6300 nulls <1% sex VARCHAR {male:168, female:165} nulls 3% year BIGINT 2007..2009

Types, value counts, ranges and null shares are in one line per column, and the output is the same on every run. On the 11 BIRD Mini-Dev databases, the rendered schema context shrank from 22.4k to 16.1k tokens while carrying more facts, according to the pull request that introduced it.

A tool call that failed, named a tool that does not exist, or sent invalid arguments used to end the attempt. It is now returned to the model as the tool’s result, so the model can correct itself in the same loop. Identical calls to read-only tools are answered from a cache, and SQLAgent submits its query as a tool call, so a rejected query is fixed inside the loop (#2104).

Every SQL result is profiled for missing values, duplicate rows, placeholder numbers, untrimmed text, numbers stored as text and outliers. The profile needs no LLM call. Only when something is worth fixing does SQLAgent spend one more call rewriting the query (#2000).

import pandas as pd

from lumen.ai.data_quality import lint_data

readings = pd.DataFrame({ "station": ["north ", "south", "east", "west", "north ", "south"], "temp_c": [12.5, 13.0, -9999, 14.0, -9999, 15.0], }) for finding in lint_data(readings): print("-", finding) print("mean with the placeholders:", readings["temp_c"].mean())

• Placeholder numbers [-9999.0] appear as data in "temp_c" (2 rows); they almost certainly encode missing values.
• Untrimmed or empty text in "station" (2 padded, 0 empty). mean with the placeholders: -3323.9166666666665

Without the check, the placeholders would have gone straight into an average and a chart. Pass SQLAgent(clean_data=False) to turn the pass off.

LLM providers now count the tokens they report, and, if you give them prices, what those tokens cost. This example needs no API key, because it uses a free model on Kilo:

import asyncio

from lumen.ai.llm import Kilo

MODEL = "nvidia/nemotron-3-ultra-550b-a55b:free"

async def main(): llm = Kilo( model_kwargs={"default": {"model": MODEL}}, usage_pricing={MODEL: {"input": 0.0, "output": 0.0}}, # USD per million tokens ) with llm.capture_usage() as usage: await llm.invoke([{"role": "user", "content": "Name one penguin species."}]) print(usage.input_tokens, usage.output_tokens, usage.cost_usd)

asyncio.run(main())

22 39 0.0

The token counts vary from call to call. usage_pricing takes prices in USD per million tokens, keyed by the model name the provider returns. capture_usage() collects everything inside its scope, including tool rounds, retries and calls from child tasks, while llm.usage totals the lifetime of the instance (#2100).

To see what happened inside a request, with llm.trace() as events: records every provider round trip and completed tool call in order, with arguments, responses, errors and durations (#2101).

Two opt-in building blocks for spending less landed as well: model routing, where a model_kwargs entry with a "routing" key picks a model per call by task complexity (#2015), and a DecisionModel interface for closed-choice questions such as follow-up classification (#2109). Nothing in the coordinator uses the decision model yet.

A Source no longer has to return pandas. Any dataframe library that narwhals supports is accepted, and schema generation, Filter and the common transforms run on it natively (#2008):

import polars as pl

from lumen.pipeline import Pipeline from lumen.sources.base import InMemorySource from lumen.transforms import Aggregate, Filter

url = "https://datasets.holoviz.org/penguins/v1/penguins.csv" penguins = pl.read_csv(url, null_values="NA")

pipeline = Pipeline( source=InMemorySource(tables={"penguins": penguins}), table="penguins", transforms=[ Filter(conditions=[("island", "Biscoe")]), Aggregate(method="mean", by=["species"], columns=["body_mass_g"], with_index=False), ], ) print(type(pipeline.data)) print(pipeline.data)

<class 'polars.dataframe.frame.DataFrame'> shape: (2, 2) ┌─────────┬─────────────┐ │ species ┆ body_mass_g │ │ --- ┆ --- │ │ str ┆ f64 │ ╞═════════╪═════════════╡ │ Adelie ┆ 3709.659091 │ │ Gentoo ┆ 5076.01626 │ └─────────┴─────────────┘

The frame stays polars from the source to pipeline.data. Some things still convert to pandas, and Lumen logs why each time: Aggregate converts unless with_index=False, as above, and anything that touches the pandas index always does. Views convert too, because hvPlot, Tabulator and Vega only read pandas. Set dataframe_backend on a Pipeli