For more than a decade, the consensus in the data science community has remained largely unshaken: if you are working with tabular data, you use gradient-boosted decision trees. This "default answer" has seen libraries like XGBoost, LightGBM, and CatBoost dominate Kaggle competitions and enterprise production environments alike. However, recent developments in the field of "Tabular Foundation Models" (FMs) are fundamentally inverting this paradigm. According to the latest data from TabArena, a prominent community benchmark, every single-model entry currently outperforming the most highly tuned gradient-boosted trees is a pretrained transformer. These models, loosely categorized as "tabular LLMs," are now demonstrating the ability to predict missing columns or labels in any spreadsheet with zero-shot accuracy, mimicking the way a large language model completes a sentence.
The emergence of these models represents a shift from traditional training-heavy workflows toward in-context learning. In this new framework, the step previously known as "training"—which involved extensive hyperparameter optimization and gradient descent—is replaced by a single forward pass during inference. This evolution was recently put to the test through an independent audit of TabICLv2, one of the strongest open-weight models in this class. The audit, conducted on independent hardware, confirmed that these models are not only statistically superior in accuracy on diverse datasets but are also significantly more efficient in terms of setup time and computational cost.
The Architecture of In-Context Learning for Tables
A tabular foundation model is a pretrained transformer designed to treat a spreadsheet as a context-rich environment rather than a static dataset. Unlike a text-based LLM, which processes word-pieces from a fixed vocabulary, a tabular model works with raw numerical and categorical cells. Because every table has unique columns and features, the model must be capable of inferring relationships on the fly.
TabICLv2, a leading example of this architecture developed by Inria’s SODA team, utilizes a three-stage transformer process to achieve this. The first transformer acts as a set transformer, reading down columns to understand the distribution of values. This allows the model to differentiate between identical numbers used in different contexts—for instance, recognizing that "450" in a price column represents a different statistical entity than "450" in a postal code column.
The second stage processes across rows, collapsing the features of a single entry into a fixed-length vector. Finally, the third transformer performs the actual prediction via in-context learning. During this phase, unlabelled test rows attend to labelled training rows provided in the "prompt." The model identifies patterns and similarities, effectively performing a highly sophisticated, learned version of k-nearest-neighbors. This eliminates the need for gradient-based training on the user’s specific data, as the model’s weights are already tuned to understand general tabular structures through pretraining on millions of synthetic tables.
Chronology of Development: From TabPFN to TabFM
The timeline of this technological shift began in earnest in 2022 with the release of TabPFN (Tabular Prior-Data Fitted Network). Developed by Prior Labs, TabPFN established the feasibility of using transformers for in-context learning on small-to-medium-sized tables. This was followed by a rapid succession of iterations from various research groups.
In 2023 and 2024, the Inria SODA team introduced TabICL and its successor, TabICLv2, which addressed scaling issues and improved attention mechanisms. Layer 6 contributed TabDPT, while the landscape shifted again in June 2026 with Google Research’s announcement of TabFM. TabFM currently sits at the top of the TabArena leaderboard, though it remains a subject of scrutiny within the community due to the lack of a published technical paper and its non-commercial licensing.
The rapid pace of these releases has caught many practitioners off guard. As of July 15, 2026, the TabArena leaderboard reflects a complete overhaul of the "best-in-class" rankings. While AutoGluon’s heavy ensemble pipelines (which run for several hours) still hold competitive positions, the single-model frontier is now entirely dominated by foundation models.
Independent Verification and Benchmarking Results
To validate these claims, independent researchers recently re-evaluated the TabArena results using the TabICLv2 model. The audit was conducted on an AWS A10G instance, covering 51 datasets ranging from binary classification to complex regression tasks. The datasets varied in size from roughly 750 rows to 150,000 rows.
The findings were striking. The independent run achieved an Elo rating of 1559, closely matching the official leaderboard’s 1575. For 16 of the 51 datasets, the metric values were identical to four decimal places. The median relative difference across the entire sweep was a mere 0.08%, a margin typically attributed to GPU nondeterminism rather than methodological errors.
The efficiency of the foundation model approach was also highlighted by the cost of the audit. The entire 51-dataset sweep, including environment setup, was completed in 2.1 hours for a total cost of approximately $2. In contrast, traditional gradient-boosted decision tree (GBDT) protocols often require hours of hyperparameter tuning per dataset to reach peak performance. TabICLv2’s median "fit" time—which is essentially just data loading and context storage—was 4 seconds per 1,000 rows, roughly 100 times faster than tuned GBDT protocols.
Comparative Performance: The Elo Standings
The TabArena benchmark uses an Elo rating system to compare models, where a standard Random Forest is anchored at 1000. A 400-point gap indicates a 10:1 win probability in head-to-head comparisons.
As of the mid-2026 snapshot, the standings are as follows:
- TabFM (Google Research): 1713 Elo
- AutoGluon (Ensemble): 1615 Elo
- TabICLv2 (Inria): 1590 Elo
- TabPFN-2.6: 1582 Elo
- LightGBM (Tuned & Ensembled): 1432 Elo
- CatBoost (Tuned): 1417 Elo
- XGBoost (Tuned): 1405 Elo
The data reveals a significant performance gap. Even after giving the GBDT models a full tuning budget and ensembling their outputs, they remain more than 150 Elo points behind the untuned, zero-shot foundation models. This suggests that the structural priors learned by transformers during pretraining are more effective at capturing complex tabular relationships than the iterative partitioning used by decision trees.
The Regime Split: Where Trees Still Hold the Ground
Despite the dominance of foundation models in aggregate scores, the audit identified specific "regimes" where traditional gradient-boosted trees remain superior. This "regime split" provides a crucial roadmap for engineers deciding which tool to deploy.
The first major factor is dimensionality. In datasets with more than 100 features, the performance of foundation models begins to degrade. For example, on the Bioresponse dataset (1,776 features) and the QSAR-TID-11 dataset (1,024 features), GBDTs consistently outperformed TabICLv2. Below the 100-feature threshold, however, the foundation model won nearly 88% of the matchups.
The second factor is the combination of scale and high-cardinality categorical variables. On the Amazon_employee_access dataset—a classic benchmark characterized by high-cardinality features—the foundation model suffered its largest loss, with a 25% error gap compared to the best tree. Furthermore, while foundation models are highly effective on small-to-medium datasets, their win rate against trees drops from 89% for tables under 3,000 rows to approximately 64% for tables exceeding 20,000 rows.
Data Integrity and the Question of Contamination
A recurring concern in the evaluation of any foundation model is benchmark contamination—the possibility that the model "saw" the test data during its pretraining phase. In the realm of tabular data, this risk is mitigated by the use of synthetic pretraining.
Models like TabICLv2 and the original TabPFN are pretrained exclusively on synthetic data generated from random structural causal models (SCMs). These are essentially mathematically generated "fake" datasets that follow plausible cause-and-effect patterns. Because no real-world data enters the pretraining pipeline, there is no risk of leakage from the TabArena benchmark.
However, the industry is seeing a trend toward "hybrid" pretraining. Newer models like RealTabPFN-2.5 have begun incorporating curated corpuses of real-world datasets from OpenML and Kaggle to boost accuracy. While these researchers employ rigorous deduplication pipelines—checking feature names, row hashes, and metadata—the community remains cautious. The audit notes that models with the cleanest provenance (synthetic-only) are currently being out-scored by those using real-world data, creating an "incentive gradient" that may complicate future objective benchmarking.
Industry Implications and the Hybrid Future
The rise of tabular foundation models suggests a looming shift in how enterprise machine learning pipelines are constructed. For many organizations, the ability to get state-of-the-art predictions in seconds without a hyperparameter search is a compelling value proposition, particularly for "cold-start" problems where data is limited.
Analysis of "router" systems—which attempt to choose the best model for a given dataset automatically—indicates that the future may lie in hybrid ensembles. While a deployable router picking between two different foundation models can successfully boost Elo scores, routing between a foundation model and a GBDT is currently more difficult. The validation errors of the two model families are often not directly comparable, leading to sub-optimal selections.
For now, the recommendation for practitioners is clear: for tables with fewer than 100 features and modest row counts, models like TabICLv2 offer a new "gold standard" for open-source performance. For high-dimensional, complex categorical data, the tuned gradient-boosted tree remains an essential tool. As Google’s TabFM and other proprietary models continue to push the ceiling of what is possible, the data science community is entering an era where the transformer may finally do for the spreadsheet what it has already done for text and image.
