TabPFN and TabICL vs. tuned XGBoost: the model that doesn't train won 14/14

ModelsBenchmarksGPUData

TabPFN and TabICL against tuned XGBoost: the model that does not train won on fourteen tables out of fourteen

The claim behind TabPFN and TabICL is that they predict on a table without ever training on it and still beat tuned boosting. I measured it on fourteen datasets from the Grinsztajn benchmark, with the same split and the same clock for everyone. The one that does not train wins, the advantage holds up to 32,000 rows instead of breaking, and the most-cited model can no longer be downloaded without an account.

Efrain Garay 17 August 2026 18 August 2026

Listen to the summary0:00Playing summary

The claim has been going around for months and it is concrete enough to be measurable: a tabular foundation model predicts on a table without ever having trained on it and still beats tuned boosting.

If that is true, half a decade of practice changes shape. Searching hyperparameters stops being a mandatory step and becomes a luxury that sometimes does not pay off.

So I put it to the test on my own card, with fourteen datasets, four contenders and the same stopwatch for everyone.

A tabular foundation model is pretrained on millions of synthetic tables generated on purpose. When a new table arrives, it does not adjust a single weight: it receives the training rows as context and produces the predictions in one forward pass.

It is the same in-context learning idea we already know from language models, moved from words to columns. That is why the verb “train” sits oddly: the code still calls it fit, but inside there is no gradient descent, there is a copy of data to the card.

That also explains why the cost shows up where you do not expect it. Fitting is nearly free and prediction is what pays, exactly the reverse of a tree.

Put that way it sounds abstract, so here is the full journey of one row: a request arrives with an empty cell, the model weighs it against everything that already happened, resolves it in one pass and returns a future. Pick any of the six cases I measured to see it with their real data:

2 100 applications already settled · 10 columnsclf_num/credit.csv

2 100 calls already made · 7 columnsclf_num/bank-marketing.csv

2 100 previous discharges · 7 columnsclf_num/Diabetes130US.csv

2 100 cases already closed · 11 columnsclf_cat/compas-two-years.csv

2 100 previous periods · 7 columnsclf_num/electricity.csv

2 100 molecules already assayed · 419 columnsclf_num/Bioresponse.csv

2 100 applications already settled · 10 columnsclf_num/credit.csv

2 100 calls already made · 7 columnsclf_num/bank-marketing.csv

2 100 previous discharges · 7 columnsclf_num/Diabetes130US.csv

2 100 cases already closed · 11 columnsclf_cat/compas-two-years.csv

2 100 previous periods · 7 columnsclf_num/electricity.csv

2 100 molecules already assayed · 419 columnsclf_num/Bioresponse.csv

Banking’s most repeated case: deciding who gets lent to.

All three datasets go to the foundation model. On heloc TabPFN takes 0.022 and TabICL 0.014: in credit risk that is not decoration.

Context: clf_num/credit.csv from the inria-soda/tabular-benchmark suite, trimmed to 3,000 rows with seed 0 and split 70/30 with stratification.sha256 22a759296600c39a56884dafd84eb346be018c537ff096adc8c2ae7e0520f2a9