Use Temporal Foundation Model to predict drawing

Predict the next 2 seconds of human hand sketch using Timestral, a time-series foundation model by Mistral. It draws all 21 of the model's quantiles, from 0.01 to 0.99, each in its own rainbow colour from red (0.01) to violet (0.99). Each line starts at the end of your stroke and fades to transparent toward the end of the forecast. Lines also start fainter the further their quantile is from the median: the median (0.5) starts solid, and 0.01 and 0.99 start at 10%.

Your strokes stay on the canvas and build up. Each stroke starts violet, the colour of the highest quantile. As you draw, it shifts toward the colour of whichever forecast line your pen is closest to: following the median turns it green, and veering toward the low or high side turns it red or violet.

Every forecast is also kept: as each one arrives it is drawn faintly into the canvas, underneath your strokes, so the alternatives the model considered build up into a glow around each stroke.

frontend/ is a plain HTML canvas page. It records the stroke, resamples it to a uniform 60 Hz grid, sends it to the backend, and draws the forecast.

backend/server.py loads the model and serves POST /forecast. It also serves the frontend, so you only run one process.

Requirements

uv

Python 3.14 or later (uv can install it for you)

A web browser

No GPU is needed. The backend uses CUDA if it finds a GPU and the CPU otherwise.

uv sync # install dependencies uv run backend/server.py # start the server

The first run downloads the model weights from the Hugging Face Hub and caches them. When the server prints ready, open http://127.0.0.1:8000.

Open http://127.0.0.1:8000/?record=true to record a video of the canvas. Recording starts when you begin your first stroke. Pressing Escape stops it, downloads the video and clears the canvas; the next stroke starts a new recording. The video is 1080×1920 (9:16 portrait), in MP4 (H.264) where the browser supports recording it, and WebM otherwise.

To check the backend without a browser, run uv run smoke_test.py while the server is running.

The default model is mistralai/Timestral-28M. To use another one, pass --model:

uv run backend/server.py --model mistralai/Timestral-5M # 21 MB, fastest uv run backend/server.py --model mistralai/Timestral-209M # 837 MB, best quality

--host and --port (default 127.0.0.1:8000) set where the server listens.

The forecast uses the whole stroke until it's longer than the model's context window, which is 32,768 steps (about 9 minutes at 60 Hz). After that, only the most recent 32,768 steps are used, as a rolling window. Longer contexts make each forecast slower. To trade accuracy for speed, shrink the window:

uv run backend/server.py --max-context 600 # last 10 s of the stroke

The frontend reads the window from /info and trims the stroke before sending it.

POST /forecast takes uniformly sampled coordinates and a horizon in steps (1–1024):

{"x": [0.1, 0.11, ...], "y": [0.5, 0.52, ...], "horizon": 120}

It returns [xs, ys] paths for all 21 quantile levels (0.01, 0.05, 0.1, 0.15, …, 0.9, 0.95, 0.99), and the inference time:

{"quantiles": {"0.01": [[...], [...]], "0.05": [[...], [...]], ..., "0.99": [[...], [...]]}, "ms": 41.2}

GET /info returns the loaded model name and its context window in steps (max_context).

These are set at the top of frontend/app.js:

Curved strokes can jump abruptly in the forecast, possibly every 32 steps (the model's patch size).

The quantile lines are computed separately at each step, so they are not real paths. Treat them as decoration.

See docs.md for the full plan and next steps.

The Timestral weights use a Modified MIT license, which differs from the Apache 2.0 code license. Read the weights' license before any commercial use.