
A Whole-Cortex Encoding Model of Vision, and Language for In-Silico Neuroscience
A Whole-Cortex Encoding Model of Vision, Audition, and Language for In-Silico Neuroscience
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Like 1
Other
PyTorch
Safetensors
neuroscience
fmri
brain-encoding
naturalistic-stimuli
fsaverage
video
audio
License: cajal-1-research
Model card Files Files and versions xet
Community
Copy to bucket new
Cajal-1 The story
How it works
Results
What it learned about the brain
What it is useful for
Quick start
Inputs and outputs
Model
Training data
Limitations
Licensing and credits
Testing
What's next: Cajal-2
Files
Cajal-1
A Whole-Cortex Encoding Model of Vision, Audition, and Language for In-Silico Neuroscience
Watch a film. See the whole cortex respond.
Cajal-1 is a brain encoding model. Give it any video with sound and it predicts, every 1.49 seconds, how each of the 20,484 points of the human cortical surface will respond — the same signal an fMRI scanner records. On the films Meta held out to test TRIBE v2, it predicts the real brain activity of the same viewers 16% better than TRIBE v2.
Left: Purkinje neurons of the pigeon cerebellum, drawn by Santiago Ramón y Cajal in 1899. Right: the areas Cajal-1 predicts respond most to faces, places and bodies.
Try it in five minutes: open demo.ipynb in Google Colab (File → Upload notebook). It downloads the model and a sample film and draws the brain for you.
In the 1890s Santiago Ramón y Cajal stained single neurons with silver and drew them by hand. His drawings showed that the brain is made of separate cells wired into circuits, and earned him the 1906 Nobel Prize. A century later fMRI lets us watch the whole living cortex at work, about once a second, while a person watches a film.
Cajal-1 learns the mapping between the two worlds: what is on screen and in the soundtrack, and what the cortex does about it. It was trained on about 3,100 fMRI recordings of people watching TV episodes, films and video clips, viewing photographs and listening to stories. Once trained, it needs no scanner: a video file is enough.
Test set: TRIBE v2's own held-out movie10 segments (The Bourne Supremacy, The Wolf of Wall Street, Hidden Figures, chunks 06 and 10), 40 fMRI runs from the CNeuroMod movie10 dataset. Score: Pearson r between predicted and measured activity per run, averaged over runs, after a 128 s high-pass filter. "Parcel" averages vertices within 1,000 Schaefer parcels first; "vertex" scores every vertex.
Brackets are 95% bootstrap intervals over runs (10,000 resamples). TRIBE v2 was run from its released checkpoint with its own video and audio feature extractors. Both models were scored on the same rows of the same runs by the same code. A three-seed ensemble of Cajal-1 reaches 1.19×; this release is the best single seed.
A note on sub-06: TRIBE v2 never saw this viewer, while Cajal-1 trained on sub-06's other recordings (never these test films). The sub-06 row is therefore a comparison on a viewer Cajal-1 knows, not a test on a new person.
These numbers are reproduced by the packaged code: tests/test_e1_parity.py re-scores all 40 runs and matches the original evaluation to within 0.001 per run (mean 0.1822 vs 0.1822).
The brain has areas that prefer faces, places, bodies and written words. Neuroscientists find them with a "localizer": a sequence of pictures shown one at a time in the scanner. We showed Cajal-1 the same kind of localizer video (396 images, one every 8 seconds) and contrasted its predicted response to each category with the others. No brain data enters these maps; they are entirely the model's.
This is an exploratory demonstration, not the formal comparison against measured localizer maps, which is still to be run.
What it is useful for
Experiments before the scanner. Run a localizer, a new stimulus set or a film through the model to see which networks it engages, and refine a design before booking scanner time.
Comparing stimuli. Ask which of two clips drives face, voice, motion or language areas more, at the scale of thousands of clips.
Testing what makes a model brain-like. Swap or ablate feature streams and see how the predicted cortex changes.
Teaching and outreach. Show students the cortex responding to a film, second by second, in colour.
A starting point. Its shared readout and per-viewer readouts make it a base for fitting new people or new datasets.
It is a research model. It predicts an average viewer's response, not a specific person's, and it is not a diagnostic tool.
Quick start
from huggingface_hub import snapshot_download import sys path = snapshot_download("ffh92r32rm0/Cajal-1") sys.path.insert(0, path) import cajal
model = cajal.load(path) # GPU if available brain = model.predict_video("my_video.mp4") # features + prediction in one call
brain.plot() # film frames above the predicted cortex, TRIBE v2 style brain.show(at=42) # every side of the cortex at 42 s brain.top_regions(at=42) # the most active areas then, with plain-language names brain.movie("brain.mp4") # the film and the cortex side by side brain.save("my_video_brain") # .npz with the (time points, 20484) prediction and times
brain.pred is the raw (time points, 20484) array if you want to analyse it yourself. The lower-level path (cajal.features.extract_features then model.predict) is unchanged, for example to reuse features across runs or to choose a training viewer's readout per call.
Install: pip install -r requirements.txt (or pip install torch "transformers>=4.53" safetensors huggingface_hub nilearn av matplotlib h5py sentencepiece plotly, plus faster-whisper for the optional transcript stream and colorcet for the exact colour map).
The Colab notebook does all of this with a sample film (Tears of Steel, CC BY 3.0) or your own upload, and adds a rotatable 3D view and region time courses. A 3-minute clip takes about 5-10 minutes on a free T4 GPU.
Rendering. cajal.plotting draws the cortex the way the TRIBE v2 figures do: half-inflated fsaverage5 surface, sulcal shading and a fire colour scale, with smooth per-pixel shading. Each view is rasterised once, after which a frame takes tens of milliseconds, so whole-film brain movies are practical on a laptop.
Inputs and outputs
Input. Four feature streams on a 1.49 s grid, one row per fMRI volume. cajal.features computes them from a video file with the exact recipe used in training:
Text is off by default because the scores above were measured without it.
Output. (T, 20484) float32: left hemisphere vertices 0-10,241, right hemisphere 10,242-20,483, in fsaverage5 order. Values are z-units: each vertex relative to its own mean and spread over the run, so a prediction says where and when activity rises and falls. Rows with no valid input (past the end of the video) are NaN.
Context. The model uses up to 61 preceding rows (91 s) of context. The first 61 rows of a clip are predicted from the context available, which is less accurate; predict(..., include_warmup=False) returns NaN for them instead.
Readout. subject="average" (default) uses the shared population readout. Readouts learned for 41 individual training viewers or pooled datasets are also available (model.subjects, e.g. "movie10:01").
Model
218.8 M parameters. Per-stream projections, a learned haemodynamic response per dataset, and a 12-layer causal transformer trunk (width 1,152, 12 heads, attention reach 61 rows) with rotary positions in seconds.
A shared low-rank readout to 20,484 vertices, plus a small residual readout per training viewer.
Trained 36,180 steps with a mean-squared-error loss on z-scored, 128 s high-pass-filtered targets, learning rate 5e-4 with plateau decay, weight decay 0.1, dropout 0.2 and 30% per-modality dropout. This checkpoint is a weighted average of four late checkpoints of one seed.
Training data
About 3,100 fMRI recordings from ten public datasets, all projected to fsaverage5:
Limitations
Category areas. Places and bodies land in the right areas; the fusiform face area is found only weakly, and the visual word-form area and object-selective lateral occipital cortex are not recovered (see above).
Subcortex. The checkpoint has subcortical output columns, but they were not trained; this release returns cortex only.
Population. Training viewers were healthy adults. Predictions describe an average viewer, not a specific person, unless a training viewer's readout is chosen.
Not a diagnostic tool. This is a research model.
Licensing and credits
Before using or redistributing this model, check the terms of:
Images in this card (all from Wikimedia Commons):
Brain figures are rendered from Cajal-1's own outputs by tools/make_readme_figures.py.
Testing
Every release file was checked before publication. Results are in tests/results/.
Not tested here: a real Colab session. The notebook was executed with the same code on a Linux CPU machine. On a GPU without native bfloat16 (such as Colab's T4), feature extraction runs in float32. That is the precision used in the feature parity test above.
What's next: Cajal-2
Cajal-2 is being prepared on the same architecture with a larger and cleaner corpus: hundreds more Friends episodes of the same viewers, a rebuilt natural-speech dataset covering ten listeners, further recordings of the same people (entering only if they pass registered quality gates), and fixes from a full data-quality audit. Its evaluation plan and validation set are fixed in advance, and it will be scored the same way as Cajal-1.
Files
File Content
model.safetensors weights (218.8 M parameters, float32)
norm_stats.safetensors feature normalisation statistics
config.json architecture, readout keys, window settings
cajal/ model code, feature extraction, plotting, brain atlases
demo.ipynb Colab demo
assets/readme/ the figures in this card
tests/ unit tests, the 40-run TRIBE v2 test-set reproduction, feature-extraction checks
tools/ checkpoint export, notebook generation, README figures
Downloads last month -
Safetensors
Model size
0.2B params
Tensor type
F32 ·
Files info
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。