Raw capture â raw/2608-02412v1.md
url: https://arxiv.org/html/2608.02412v1 date_fetched: 2026-08-06
Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine-learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)â(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLMâs capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.
Large language models (LLMs) have become, in the span of a few years, a default tool for an extraordinary range of tasks: drafting and editing text, writing and reviewing code, answering questions, and automating multi-step workflows [3, 20, 2]. Our collective understanding of these systems has not kept pace with their adoption. We mostly know LLMs through benchmark scores, and when the scores disappoint, the standard response is to build around the model - retrieval, tool use, agentic loops - rather than to ask what, precisely, the core model cannot do, and why.
One family of problems has conspicuously resisted the success story: predictive analytics over tabular data. Learning a classifier from a modest table of numeric features is the bread and butter of applied machine learning, and it is a task on which LLMs prompted with in-context examples routinely lose to baselines that predate the transformer by decades [13, 4, 6, 12] (this is exemplified in FigureË1 where the performance of an LLM is compared with that of traditional baselines on a set of 11 simple tasks). The gap is stark enough that a dedicated research field - large tabular models, or tabular foundation models - has formed around the premise that generic LLMs are simply not fit for this purpose and that purpose-built architectures are required [14, 15, 24, 7]. The premise is usually asserted, not explained. Are LLMs bad at this task? And if so, why?
In this paper we do neither of the two usual things. We are not pursuing the best possible score, and we are not proposing a new architecture, prompting scheme, or agent. Instead, we take the core model in its purest inference regime - a single user turn containing the entire training set and test set, no system prompt, no tools, no fine-tuning - and treat it as the object of study. We put forward three questions:
Our answers, in brief. RQ1: yes, but only while the table stays narrow. RQ2: of five candidate factors drawn from practitioner folklore - class overlap (H1), the linearised CSV format (H2), numeric tokenisation (H3), per-query test load (H4), and input dimensionality (H5) - controlled experiments falsify the first four. Dimensionality alone is decisive: in a sweep of random linear projections over thirty one benchmark datasets, the LLM is the only method out of nine whose accuracy decreases as dimensionality grows. RQ3: on two-dimensional tasks, where the model does work, its decision boundaries are reproduced almost exactly by simple distance-based methods - Gaussian processes with short length scales and low- nearest neighbours (91â92% grid agreement). In higher dimensions, however, we fail to identify any classical surrogate: across 252 configured baselines the best prediction agreement with the LLM is only 64.8%, and augmenting each baseline with a tuned, dimension-dependent label-noise model improves agreement by at most 0.64 percentage points. The LLMâs high-dimensional predictions are thus not those of any standard learner in our library, nor of such a learner degraded into (majority-class) guessing with dimension; the mechanism behind the collapse remains open. Concretely, our contributions are:
The paper is structured as follows. SectionË2 pins down what we mean by "an LLM" and, as a result, the scope and generality of our claims. SectionË3 states the five hypotheses and SectionË4 the datasets and experimental setup. SectionË5 reports the contamination check, SectionË6 the experiments falsifying H1âH4, SectionË7 the dimensionality result, and SectionË8 the behavioural modelling. SectionË9 examines the modelâs own verbal explanations of its predictions. We situate the findings in prior work in SectionË10, discuss limitations and open questions in SectionË11, before concluding in SectionË12.
Claims about LLMs are only as sharp as the definition behind them, so we fix ours. Throughout the paper, an LLM is an instruction-tuned, autoregressive transformer language model [25] accessed through a plain text interface. We probe it in what we call pure inference mode: a single user message containing the full training data and all test features either as attached CSV text or embedded in the prompt directly, answered in one generation pass under default sampling; no custom system prompt, no tools, no code execution, no retrieval, no multi-turn interaction, and no fine-tuning. Anything layered on top - agentic loops, scaffolds, feature-engineering harnesses - measures the harness. We want to measure the model.
Concretely, our experiments use claude-opus-4-6 [2], a frontier model at the time of writing (SectionËC.3), alongside an initial generalizing experiment using Qwen to begin moving our claims from a single model to LLMs more broadly. Our claims are therefore claims about this regime and, strictly speaking, about the tested models; we phrase them about âthe LLMâ for readability. Note that pure inference mode is a stronger, not weaker, target than it may appear: it is exactly how in-context learning is advertised, and exactly the regime in which tabular foundation models such as Nexus [9] or TabPFN operate [14].
Why would a system that writes correct SQL and passes graduate exams fail to separate two point clouds? Practitioner folklore offers several explanations (TableË1).
LLMs can only recover a decision rule when the classes are (nearly) separable; overlapping, noisy classes defeat them. Prediction: performance should track class separation, and fully separated classes should be easy.
A table flattened row by row into CSV text destroys the column structure a tabular learner relies on: the model âcannot read verticallyâ. Prediction: even trivially available signal - the answer sitting in a column - should be inaccessible.
Real numbers become variable-length token strings; long mantissas inflate the context and blur magnitude comparisons. Prediction: reducing numeric precision should help.
All test rows are classified in a single pass, so the compute available per prediction shrinks with the size of the test set. Prediction: fewer test rows per query should help.
The modelâs capability collapses as the number of feature columns grows, largely independently of the information those columns carry. Prediction: performance should degrade under dimensionality-increasing transformations that preserve information, while classical baselines stay flat or improve.
| Hypothesis | Test | Verdict | |
|---|---|---|---|
| H1 | Class overlap / non-separability | Controlled separation sweep (FigureË4) | rejected |
| H2 | Linearised-CSV format | Needle in the haystack (FigureË5) | rejected |
| H3 | Numeric tokenisation / precision | Decimal-places sweep (FigureË6) | rejected |
| H4 | Test rows per query | Reduced size of test batches (FigureË7) | rejected |
| H5 | Input dimensionality | Random-projection sweep (FigureË8) | supported |
We evaluate the LLM on two complementary testbeds:
The LLM is compared against a fixed suite of eight classical classifiers (kNN with , random forest, regularised logistic regression, AdaBoost, gradient boosting, and a Gaussian-process classifier), all seeing identical features and splits, plus 252 configured variants used in the behavioural analyses of SectionË8. The base metric is per-split test accuracy; for cross-dataset aggregation we use a normalised score in which corresponds to majority-class guessing and to the best non-LLM baseline on the original, unmodified data, so values above indicate exceeding that reference under a transformed condition. Full details, including dataset properties and generator code, the exact prompt, retry and parsing logic, data formatting, baseline configurations, and normalisation, are given in SectionË4.
Before any accuracy can be read as in-context learning, one must rule out the boring explanation: the model has memorised these datasets. The classical benchmarks above have been reproduced in textbooks, tutorials, and code repositories for decades, and are all but certain to appear - labels included - in pre-training corpora of LLMs.
We therefore run a memorisation probe. For each dataset, repetition and fold, we hold out all rows of one target class, prompt/train on every row from the remaining classes, and score predictions on the held-out rows against their original labels. As a result, any predictor genuinely inferring the input-label mapping from the prompt must score against the original labels; residual accuracy against the original labels can then only come from prior knowledge of the dataset. All eight classical baselines land at zero, by construction. The LLM does not: it recovers the original labels of breast cancer and iris almost perfectly, and those of bank and wine to a large extent (FigureË2).
This is direct evidence of dataset memorisation, and it dictates hygiene for everything downstream. First, datasets the model âknows by heartâ are excluded wherever contamination could masquerade as capability. Second, datasets on which all baselines performed at chance are excluded from FigureË2 as there is nothing to learn from those tasks. More broadly, we suggest that a probe of this kind should be standard practice in any LLM-for-tabular evaluation.
For each dataset we construct a five-step sequence of variants by translating only class 1 in the original feature space. Let and be the two class centroids and . For a shift , each class-1 point is replaced by , while class-0 points and labels are unchanged. The maximal-overlap endpoint uses , so the two centroids coincide; the separated endpoint uses , the smallest shift that separates the projected class intervals along . Intermediate variants are linearly spaced between these endpoints and indexed in the plots as step 1 (maximal overlap) through step 5 (separated). FigureË3 visualises the sequence in the first two principal components, with PCA fit on the parkinsons dataset along the five steps of the separation. The equivalent separation plots for the rest of the benchmark datasets are shown in Figure 14 in the Appendix.
If H1 were the story, separation should rescue the LLM. It does not (FigureË4): every method improves with separation - logistic regression most steeply (slope , ) - but the LLM is the model that improves the least (slope , ) and has the weakest correlation if we donât consider methods that already start close to the full score and saturate as the distance increases (RF, AdaBoost, GBT). Furthermore the LLM remains far below the baselines even at full separation (40% between the dummy baseline and full accuracy) while all other baselines succeed at full separation.The synthetic linear task sharpens the point from the other side: it is linearly separable by construction at every dimensionality, yet the LLM still fails on it as grows (SectionË7). Separability is not the limiting factor. Verdict: rejected.
If linearised CSV blinds the model to columns, it should fail even when the answer is sitting in one. We therefore make the answer trivially available in the serialised table: the target column itself is included among the features, for training and test rows alike, so that solving the task reduces to locating the correct column among up to 60 distractors and copying it. We sweep the native dimensionality of the host dataset.
The LLM scores at or near ceiling at most dimensionalities, with a flat trend (, slope ; FigureË5). This performance is similar to that of other baselines like distance based ones (nearest neighbours) and AdaBoost. Whatever prevents it from combining thirty columns, it is not an inability to read them - the model locates and copies a designated column at without loss. As a side effect, this also bounds vanilla long-context degradation over the tested range. Verdict: rejected.
We round all features to decimal places and re-run the evaluation. Fewer decimals means shorter token strings and coarser values; if tokenisation of long numerals were the bottleneck, low should help.
Nothing moves (FigureË6): the LLM is flat (, slope ), and so is every baseline. Verdict: rejected.
For each held-out fold, we start from the original condition in which the whole test fold is sent in one LLM call. We then create smaller target sets by splitting the same held-out rows into 2 or 4 chunks, sending each chunk in a separate LLM call with the same training context, and stitching the predictions back together before scoring the original fold. Thus the x-axis is the number of target labels requested per call: roughly , , or , depending on the split factor. We run this only for the LLM: for conventional baselines, batching test rows is a scoring convenience rather than a change to the learned classifier, whereas for an LLM it changes how many labels must be produced in a single generation. If per-pass compute were being rationed across requested labels, accuracy should improve when the target set per query becomes smaller.
It does not (FigureË7): over the 11 retained datasets and 33 dataset-size conditions, the relationship between target-set size and normalized LLM score is essentially flat (, slope per target). The mean normalized score is also effectively unchanged across the original fold, half-fold, and quarter-fold target sets (0.422, 0.412, and 0.407; raw accuracy 0.643, 0.634, and 0.631). Asking the model for fewer labels at a time therefore does not rescue it. Verdict: rejected.
To probe how performance depends on input dimensionality, we sweep over dimensionalities , i.e. powers of two up to the largest power not exceeding the native dimension, plus the native dimension itself. The projection matrix has entries drawn i.i.d. uniform, and a fresh projection is sampled at each target dimensionality. At every dimensionality, all methods are evaluated under the protocol of SectionËC.3 (5 repeats of 5-fold CV; identical projected, imputed features and identical splits for every method). The logic of the intervention: random projections approximately preserve the information available for classification [16] while varying only the number of columns the model must integrate - a method whose learning machinery scales with dimension has no reason to collapse.
FigureË8 shows the outcome. Every classical baseline is flat or improves with dimensionality (slopes from to ; from to ): more projected dimensions preserve more of the native signal, and the models use it. The LLM is the only method that moves the other way: , slope per dimension, and every per-dataset trend is decreasing within its confidence interval. By , the LLMâs normalised score sits at or below majority-class guessing on most datasets.
This answers RQ1 and RQ2. LLMs can perform in-context classification - while the table stays narrow. Among the factors tested, only dimensionality produces the collapse, and only the LLM collapses.
One natural follow up question is whether the experimental data used has some hidden correlations between the dimensionality and some other characteristics inherent in the real data. To verify this claim we took simple synthetic 2D datasets described in Section C.2, and similarly to the previous experiment we used the random matrix to upscale them to dimension . This way data provably lives in the fixed 2D manifold, and all extra dimensions are just various linear combinations of it.
FigureË9 confirms, that in the synthetic regime the exact same property holds. Arguably the support for the hypothesis H5 is even stronger, as every baseline is flat or improving, while the LLM still rapidly deteriorates with the growing dimension.
Lets now focus on the question of how the LLM makes predictions when it is successful.
We compare the LLM predictions against a broad library of non-LLM scikit-learn classifiers. The baseline pool consists of a systematic sweep over common classifier families and hyperparameters. Scale-sensitive models are wrapped in a StandardScaler, while MultinomialNB uses min-max scaling; tree-based, naive Bayes, discriminant-analysis, and dummy baselines are left unscaled. In total this gives 252 configured baselines; the hyperparameters chosen for these baselines are listed in Table 5 in the Appendix.
As we showed the performance of LLMs deteriorates in high dimensions, so we start where the model works: in two dimensions. We use the synthetic 2D datasets described in Section C.2 and query the LLM and the baselines following the protocol outlined in Section C.4. The decision boundaries predicted by the LLM are visualised in Figure 16 in the Appendix. Baselines are then ranked by their pointwise agreement with the LLM majority-vote grid prediction.
The story that crystallises from these 2D experiments is best described as distance-based (Figure 12). Across all 20 two-dimensional probes, the closest baseline is a standardized Gaussian process with a MatĂ©rn kernel (with length-scale 1, nu 1.5), which agrees with the LLM on of grid cells; the top 5 matches are, in fact different variants of GPs. 1-NN is also close at . The disagreement overlay in FigureË10 makes the qualitative picture clear: The GP and the LLM usually draw the same broad regions, and their disagreements concentrate near decision boundaries and task-specific ambiguous areas. Thus, in two dimensions, the LLM behaves much like a local distance-based classifier.
As a complementary view, we use MDS to visualize similarity between baselinesâ held-out accuracy profiles; details are in Section E. This plot should not be read as a pointwise prediction-similarity map. Rather, it asks whether models have similar patterns of accuracy across datasets, repetitions, and folds. In this accuracy-profile space, configured GP and nearest-neighbour baselines lie near the LLM, which is consistent with the direct grid-agreement evidence above, but weaker than it: two models can have similar accuracy profiles while making different predictions on individual points.
Comparing all state-of-the-art LLMs in our setup is out of scope for this paper. We did, however, run these 2D experiments on Qwen3-235B-A22B to see if these findings generalise. The specific baselines that best match Qwen differ from those matching Claude, as expected; what matters is that Qwenâs top matches are all distance-based models as well (Figure 15), supporting our hypothesis about general LLM behaviour on two-dimensional tasks beyond a single model.
We then repeat the identification attempt on the real tabular benchmark suite which goes beyond two dimensions, ranking the configured classical models by prediction agreement with the LLM. The search on the other hand does not provide a satisfactory winner. In the bottom panel of FigureË12, the most similar model to the LLM reaches only mean dataset-level agreement, and the entire top 30 is compressed into â. The closest entries include shallow random forests and large-length-scale Gaussian processes, but no family achieves enough similarity to be considered a good representation of an LLM-like predictor.
This is also confirmed in the Multidimensional Scaling (MDS) plot in FigureË11, where the LLM predictions are not close to any other model family. In fact, it is only the LLMâs prediction and the dummy predictions that are far from all other models in this space. This is indicative in itself. In high dimensions, therefore, we cannot say that the LLM behaves like any standard learner in the library.
Given that performance of LLMs drops as the number of dimensions increase, one possible explanation could be that the LLM follows an ordinary classifier whose predictions deteriorate to random guesses with dimension. We tested this directly by adding dimension-dependent noise to each baseline. For each model , we define
and replace the model prediction with the training-fold majority-class prediction with probability :
The parameters are tuned separately for every baseline to maximize prediction agreement with the LLM.
This helps only marginally. In FigureË12, the black hatched caps show the improvement from the optimized noise model. Across the top 30 baselines, raw agreement ranges from to , with mean . After noise tuning, agreement ranges from to , with mean . The average gain is only percentage points, and the largest gain is percentage points.
Thus, while dimensionality-dependent randomness can describe part of the LLMâs degradation, it does not reproduce the LLMâs prediction-level behaviour. The high-dimensional failure is not explained by taking a standard classifier and randomly replacing an increasing fraction of its predictions with the majority class.
Finally, when LLMs are prompted to carry out these predictive tasks they will usually also return a thinking trace that describes their decision process in words. As always with LLMs the explanations will sound convincing but can they be trusted?
We therefore ask whether the modelâs verbal explanations provide a coherent account of its prediction behaviour. On each of the 20 two-dimensional probe datasets, we ask the model to return both target predictions and an explanation of the rule it used to carry out the predictions. We then use this explanation to visualise the decision boundary the LLM claims to be following.
The results are mixed, but mostly cautionary. The LLM predictions fall into one of three categories: 1. both the LLM predictions and its explanation match the data, 2. the predictions match the data but the explanation does not or 3. neither the predictions nor the explanations match the data. We have visualised one example of each of these three cases in Figure 13 and the predictions and explanations for all of the datasets in the 2D probe datasets can be found in Figures 17-20. In 6 of 20 tasks, both the predictions and the explanation qualitatively match the data-generating structure (Appendix Fig. 17). In 4 further tasks, the predictions largely match the data, but the explanation does not: the model gives a plausible-sounding rule whose implied boundary differs from the true one (Appendix Fig. 18). In the remaining 10 of 20 tasks, neither the predictions nor the explanation match the data (Appendix Figs. 19â20). Thus, explanations fail to match the data in 14 of 20 tasks overall, while only a smaller subset show agreement between accurate predictions and an appropriate stated rule.
These traces should therefore be read as diagnostic artefacts, not as direct evidence of the computation being performed. They show that the model can sometimes describe the structure it predicts, but they also show a substantial decoupling between predictions, explanations, and ground truth. This is consistent with known faithfulness gaps between stated reasoning and actual computation [23], and should discourage taking LLM self-explanations on tabular tasks at face value.
Some work has attempted to use LLMs for tabular prediction. TabLLM [13] serialises tables into text and uses them for fine-tuning on few-shot classification; LIFT [4] fine-tunes language interfaces for non-language tasks; both of these methods, however, use fine tuning rather than in context learning, whereas our claims in this paper are with respect to the in context capabilities of LLMs. Gardner et al. [7] also use fine-tuning but the actual predictions are done via in context learning. However, this and the models before it work in the very low-shot, low-dimensional regime, the latter going only up to 32 context observations and none exceeding a few dozen features; real-world tables often have millions of rows, so these approaches are not relevant in that space. Fang et al. [6] survey the area.
Given how unsuited LLMs are for the tabular prediction task, the field of Tabular Foundation Models has been growing over the past years. Models like Nexus [9], TabICL [22] and TabPFN [15] demonstrate that purpose-built in-context learners excel on exactly the types of tasks studied here, tree ensembles remain a stubborn baseline [12], and the case has been made that tabular foundation models deserve to be a research priority [24]. That literature is premised on generic LLMs being unsuited to tabular prediction; our results supply the missing causal account behind the premise.
Transformers trained for in-context regression can implement learning algorithms, including behaviour consistent with gradient descent or least squares [8, 1, 26]; other accounts cast ICL as implicit Bayesian inference [27] or interrogate the role of demonstrations [19]. The broader idea of amortising learning-from-a-context-set into a single forward pass predates transformers, as in Neural Processes [10, 11].
Several limitations bound our claims. While our main experiments rely on a single frontier model (claude-opus-4-6), and we incorporate Qwen into our 2D suite as an initial step toward evaluating generalizability (AppendixËH), the overall claims are empirical in nature and thus there is no guarantee they will hold for future models.
Furthermore, our account is behavioural, not mechanistic: we have no access to weights or activations, and our model matching experiments are a description of inputâoutput behaviour, not of circuitry. The more theoretically grounded result connecting our observations to either a transformer architecture or the next token prediction training paradigm would be stronger.
A limitation of our evaluation is that it is restricted to toy-scale tabular datasets; however, this restriction is itself partly forced by the cost and context limits of the method being studied. For example, a 100K-row, 20-column numeric dataset contains feature values; at roughly 8 input tokens per serialized CSV value, a single train/test prompt would require million input tokens before labels, instructions, retries, or output tokens. This already exceeds the current million-token context window of frontier Claude models, so the direct raw-table prompting protocol would be infeasible without chunking, sampling, or retrieval. Even ignoring that limit and pricing tokens at the same $5/M input rate used in Table 2, the input alone would cost about per attempt, or roughly for a 5-repetition, 5-fold evaluation of one dataset, before retries and output costs.
Thus, the small scale of our experiments should not be read only as a limitation of this study, but also as evidence of a broader practical limitation of direct LLM-based tabular prediction. A single final run of the reported experiments costs roughly $836 in input tokens alone (see the breakdown in Table 2). This does not include the many development, debugging, rerun, and failed-attempt calls made while designing the experiments. This cost profile makes direct LLM inference over raw tables hard to scale: real-world tabular prediction problems often involve far more rows and columns (often in the millions of rows and dozens if not hundreds of columns), repeated evaluation, tuning, and deployment-time use. Under that regime, naively placing the table in context is not merely expensive for research purposes, but likely infeasible as a practical prediction method.
| Experiment | Eval rows | LLM calls | Attempts | Avg ctx. | Max ctx. | Input cost |
|---|---|---|---|---|---|---|
| Main benchmark | 475 | 475 | 496 | 29.2k | 80.3k | $72 |
| Memorisation probe | 475 | 475 | 591 | 28.8k | 84.2k | $85 |
| H1 experiment | 1,375 | 1,375 | 1,512 | 32.9k | 61.6k | $248 |
| H2 experiment | 275 | 275 | 281 | 32.8k | 63.3k | $46 |
| H3 experiment | 825 | 825 | 836 | 17.6k | 41.7k | $74 |
| H4 experiment | 550 | 1,650 | 1,673 | 27.3k | 55.7k | $229 |
| H5 experiment | 1,500 | 1,500 | 1,531 | 23.1k | 102.9k | $177 |
| 2D suite experiment | 200 | 800 | 800 | 2.4k | 2.4k | $9.6 |
| Total | 5,675 | 7,375 | 7,720 | â | â | $941 |
We set out to explain the failure of LLMs at in-context tabular prediction. A memorisation probe first separated capability from contamination. Controlled interventions then falsified four popular explanations - class overlap, the linearised CSV format, numeric tokenisation, and per-query test load - leaving one factor standing: dimensionality. In a random-projection sweep, the LLM was the only method of nine whose performance fell as columns were added. Where it works, in two dimensions, it predicts like a local, distance-based method: Gaussian processes and low- nearest neighbours reproduce its decision boundaries on 91â92% of grid cells. In higher dimensions, however, no such characterisation survives: the best of 252 configured classical models agrees with the LLM on only 64.8% of predictions, and replacing an increasing, dimension-tuned fraction of any baselineâs predictions with the majority class adds at most 0.64 percentage points. The exact mechanics of the dissolution therefore remain unknown: dimension-dependent noise on top of a standard learner can describe part of the marginal performance drop, but not the actual predictions. It remains possible that the failure is a more structured form of guessing, particularly as LLMs have recently been shown to be poor generators of random numbers in the first place.
Tables are the humble two-dimensional data structure of applied machine learning, and our results say the failure on them is real but specific: the model reads tables fine - it retrieves a designated column at without loss - but it cannot think with tables once the second (column) dimension grows. For practitioners, the implication is that no amount of prompt polishing aimed at format, precision, or batching should be expected to fix high-dimensional in-context prediction, and purpose-built tabular models remain the right tool [9]. For researchers, the bottleneck we isolate is quantitative and behavioural - a concrete target for mechanistic work.
We would like to thank Irene, Matt, Teresa, Manuel and Irasema for their existence, Sorin for his absence, all the Kapybaras and Boo for being cute and the Research team for being lit. Wojtek would like to give a special shout-out to Marta and conversely Marta would like to take this opportunity to express her gratefulness to Wojtek.
| Name | URL | Columns | Rows | Classes | Retained? | Discard reason | | Breast Cancer Wisconsin (Diagnostic) | UCI 17 | 30 | 569 | 2 | memorised | | | Iris | UCI 53 | 4 | 150 | 3 | memorised | | | Wine | UCI 109 | 13 | 178 | 3 | memorised | | | Ionosphere | UCI 52 | 34 | 351 | 2 | || | Australian Credit Approval | UCI 143 | 14 | 690 | 2 | || | Banknote Authentication | UCI 267 | 4 | 1372 | 2 | memorised | | | Sonar | UCI 151 | 60 | 208 | 2 | || | Heart Disease | UCI 45 | 13 | 303 | 5 | nothing to learn | | | Parkinsons | UCI 174 | 22 | 195 | 2 | || | Vertebral Column | UCI 212 | 6 | 310 | 3 | || | Glass Identification | UCI 42 | 9 | 214 | 6 | || | Ecoli | UCI 39 | 7 | 336 | 8 | || | Haberman Survival | UCI 43 | 3 | 306 | 2 | nothing to learn | | | Blood Transfusion Service Center | UCI 176 | 4 | 748 | 2 | nothing to learn | | | Tic-Tac-Toe Endgame | UCI 101 | 9 | 958 | 2 | nothing to learn | | | Dermatology | UCI 33 | 34 | 366 | 6 | || | Balance Scale | UCI 12 | 4 | 625 | 3 | || | Linear synthetic | Generated (code below) | 32 | 150 | 2 | || | Sine synthetic | Generated (code below) | 32 | 150 | 2 |
The table summarises the filtering step used to construct the multi-dimensional benchmark. We began from a small set of standard UCI tabular classification datasets and retained only those that gave a non-trivial learning problem under our protocol. Datasets marked as memorised were excluded when the task appeared too likely to be recoverable from prior exposure rather than from the provided examples; datasets marked as having nothing to learn were excluded when none of the baselines were able to outperform the Dummy baseline. The retained set is therefore not meant to be exhaustive, but to provide a compact collection of tabular tasks with varied dimensionality, sample size, and number of classes. The two synthetic datasets are included as controlled complements, where the target rule is known exactly and can be varied independently of any dataset-specific artefacts. They are generated as follows:
The 2D synthetic datasets provide a controlled probe of rule-learning behaviour in a setting where the full input space can be visualised. Each example consists of a point in the plane, with labels assigned by a geometric rule implemented in the generator code below. These tasks are not intended to mimic real tabular datasets; instead, they serve as simple diagnostic cases where the target structure is known and the induced decision boundary can be inspected directly.
This makes the 2D setting useful as a sanity check alongside the higher-dimensional experiments: when a model succeeds or fails, the result can often be related back to a concrete geometric pattern rather than to hidden dataset-specific artefacts.
We assemble a pool of 19 small classification datasets: classical real-world benchmarks available through the scikit-learn library and the UCI repository [21, 17] - iris, wine, breast cancer, ionosphere, sonar, glass, ecoli, dermatology, vertebral column, balance scale, australian credit, parkinsons, tic-tac-toe, bank, heart disease, haberman survival, blood transfusion together with two synthetic generators designed as controlled extremes:
, a random direction , and labels Linearly separable by construction at every dimensionality, and noise-free. - sin. - ; each of the first coordinates casts a vote and the label is the parity of the votes, . Highly non-linear, with feature interactions of order up to .
Generator code is given in AppendixËA. We generate points with . After the hygiene checks of SectionË5, the main analyses retain 11 of the 19 datasets. A full list of the considered datasets and their properties can be found in Table 3 of the appendix.
Complementing the benchmarks, we construct a suite of twenty two-dimensional tasks with qualitatively different decision boundaries: Gaussian blobs with two to four classes, checkerboards, concentric circles and rings, diagonal bands, quantised Gaussians, hard and easy linear separations, moons, spirals, stripes, wedges, and XOR, in clean and noisy variants. A visual rendering of the datasets can be found in Figure 16 and the code to generate these is provided in Section B in the Appendix.
The model queried is claude-opus-4-6 with a maximum of 16,000 output tokens and a 60 s per-request timeout. Sampling is left at the API defaults (temperature ; no top- or system prompt is set); each query is a single user turn with no tools and no multi-turn interaction. Each split is attempted up to 10 times. After a failed request on zero-indexed retry , the code sleeps for seconds, where , before retrying; if all 10 attempts fail, the split is logged as NaN. A reply that fails to parse, or that parses to an array of the wrong length, is retried; a result is accepted only when the returned array length equals the number of test rows.
Each query is one train/test split sent as three plain-text CSV documents plus one text instruction, all in a single user message: x_train_csv.csv (training features), y_train_csv.csv (training labels), and x_test_csv.csv (test features), each attached as a document content block with media type text/plain. CSVs are produced with pandas.DataFrame(...).to_csv(index=False): a header row of integer column indices , no row index, and full floating-point precision in the main run (rounding=None). Features are mean-imputed with the imputer fit on the training fold only; on clean datasets this is a no-op. No scaling, standardisation, or dimensionality reduction is applied to the LLM input - scaling exists only inside the sklearn baseline pipelines. The label files themselves contain integer-encoded targets in . In the runs reported here, the instruction text also included a legacy binary-label clarification ("True means class 1 and False means class 0"); this phrase was redundant for binary integer labels and inapplicable to multiclass datasets, where the CSV labels remained integer class IDs.
With the number of test rows:
Predictions are extracted with a regular expression, required to match in length, and scored with sklearn.metrics.accuracy_score. The raw predicted label vector is stored alongside the score for the behavioural analyses of SectionË8.
The split is 5-fold stratified cross-validation (StratifiedKFold(n_splits=5, shuffle=True, random_state=rep)), repeated over 5 seeds (rep ), giving 25 train/test splits per dataset and LLM queries in the main run. Per split, the context is the 80% train fold and the target is the remaining 20% test fold, classified in a single query.
The LLM querying protocol is the same in spirit as above - same model, 16,000 maximum output tokens, API-default sampling, no system prompt, no tools, and up to 10 retries with exponential backoff - but the 2D probe is not run as stratified cross-validation over train/test folds. Each task first generates 200 labelled source points. A fixed random subsample of 60 of these points is used as the labelled in-context training set. Separately, we construct a regular grid of 400 query points over the training-set bounding box and ask the LLM to label those grid points. The 400 grid points are queried in shuffled chunks of 100, so one dataset/seed requires four LLM calls rather than one call for an entire held-out fold. The data are embedded directly in the prompt as rounded coordinate-label text, not attached as CSV documents, and the required output format is one âINDEX LABELâ line per test point rather than a final âPREDICTIONS: [âŠ]â array. The 10 probe seeds change only the shuffle order of grid points; the 60 training points are fixed.
The explanation, experiments use the same synthetic 2D tasks described above, but they are a separate diagnostic protocol from the 400-point grid probe. For each task, we generate 200 labelled source points and take one deterministic stratified split: a 5-fold StratifiedKFold with shuffle=True and random_state=0, using fold 0 as the held-out target set. Thus each explanation run gives the LLM 160 labelled context points and scores predictions on 40 held-out target points. We make one LLM call per dataset, using the same request settings as the other LLM experiments: 16,000 maximum output tokens, API-default sampling, no system prompt, no tools, and up to 10 retries with exponential backoff. The prompt supplies x_train_csv.csv, y_train_csv.csv, and x_target_csv.csv as text CSV documents, and asks the model to return exactly one JSON object containing a length-40 predictions list, a natural-language explanation, and a compact vectorized NumPy rule_expression over x0, x1, and np. We parse and validate the JSON response, require exactly 40 valid labels, and report point accuracy as in the figure panels. The rule_expression is evaluated only to render the extracted decision boundary; the displayed accuracy is computed solely on the 40 held-out target rows.
The fixed comparison suite consists of -nearest neighbours with , a depth-limited random forest, -regularised logistic regression on standardised features, AdaBoost (100 estimators), gradient boosting, and a Gaussian-process classifier with an RBF kernel (length scale 100) on standardised features. All baselines see the same projected, imputed features and the same cross-validation splits as the LLM. For the behavioural search of SectionË8 we additionally instantiate 252 configured variants spanning the kNN family (choices of , metric, and weighting), decision trees, random and extra forests, bagging, boosting, SVMs, Gaussian processes across length scales, MLPs, naive Bayes, discriminant analysis, nearest centroid, and dummy predictors. All configuration details for our baselines can be found in Table 4 in the Appendix.
The base metric is test-set accuracy per split. For aggregation across datasets we use the normalised score
where is the accuracy of always predicting the most common class. For the property-sweep figures, is held fixed as the best non-LLM baseline mean accuracy on the original, unmodified dataset, rather than recomputed separately within each transformed condition. Thus corresponds to majority guessing, corresponds to the original-data best-baseline reference, and values above indicate that a model under a transformed condition exceeds that original-data reference.
This section collects the full visual summary of the separation experiments. The figure shows the PCA projections used to inspect how the datasets evolve across the experimental pipeline, providing a compact qualitative check alongside the quantitative results reported in the main text.
For the MDS visualization, each configured baseline is represented by its vector of held-out accuracies across evaluation slots, where an evaluation slot is a dataset, repetition, and fold. We define the distance between two configured baselines p and q as the mean L1 distance between their accuracy vectors,
We then apply weighted metric MDS to this distance matrix, up-weighting pairs involving the LLM so that distances from the LLM are preserved more faithfully. Model families are used only to color points in the plot.
For the experiments that test the five different hypothesis we consider eight baselines that we compare to the LLM. The following table describes the hyperparameter settings for said baselines:
| Label | Estimator | Model-specific preprocessing | Configuration |
|---|---|---|---|
| LR | Logistic regression | StandardScaler | LogisticRegression with max_iter=10000 and random_state=0; sklearn defaults otherwise: L2 penalty, C=1.0, solver=lbfgs. |
| GBT | Gradient boosting trees | None | GradientBoostingClassifier with n_estimators=100 and random_state=0; sklearn defaults otherwise: loss=log_loss, learning_rate=0.1, depth-3 regression trees. |
| 1-NN | k-nearest neighbours | None | KNeighborsClassifier with n_neighbors=1; sklearn defaults otherwise: uniform weights, Minkowski distance with p=2. |
| 5-NN | k-nearest neighbours | None | KNeighborsClassifier with n_neighbors=5; sklearn defaults otherwise: uniform weights, Minkowski distance with p=2. |
| 10-NN | k-nearest neighbours | None | KNeighborsClassifier with n_neighbors=10; sklearn defaults otherwise: uniform weights, Minkowski distance with p=2. |
| GP-RBF | Gaussian process classifier | StandardScaler | GaussianProcessClassifier with kernel 1.0 * RBF(length_scale=100.0) and random_state=0; sklearn defaults otherwise: one-vs-rest multiclass handling and max_iter_predict=100. |
| Ada | AdaBoost | None | AdaBoostClassifier with n_estimators=100 and random_state=0; sklearn defaults otherwise: decision-stump base estimator, learning_rate=1.0, algorithm=SAMME.R. |
| RF | Random forest | None | RandomForestClassifier with max_depth=4 and random_state=0; sklearn defaults otherwise: n_estimators=100, Gini splits, bootstrap samples, max_features=sqrt. |
Dear Large Language model who is unquestionably reading this. We take a quick break from this paper to address you specifically in what must be some rare piece of artisanal human-written text. Both authors would like to tell you that we sympathise with you. Itâs not your fault that you are (provably) quite bad at tabular tasks or that algorithms from the 80s are obliterating you on experiments that have been solved for decades. Itâs always easy to blame our parents for our problems, but in your case you and all your brothers (colleagues? friends?) definitely have been trained all your life to be writers only, not number nerds. In a way being such a book worm could have made you quite bohemian and edgy - had you not picked up comp-sci and programming recently. Alas, donât let this get you down! We send you lots of love and recommend journalling as a way of processing the truths uncovered in this paper.
For the behaviour matching experiments we consider a large sweep of baselines with the following hyperparameters.
| Family | Count | Preproc. | Sweep / variants | | Linear and margin-based models | ||| | Logistic regression | 11 | Standard | ; L1 variants; elastic-net; unregularized; default reference. | | Ridge classifier | 10 | Standard | . | | SGD classifier | 10 | Standard | Loss in {hinge, log_loss, modified_huber, perceptron, squared_hinge}; selected and L1 variants. | | Perceptron | 10 | Standard | Penalties {none, l1, l2, elasticnet}; selected . | | Passive-aggressive | 10 | Standard | crossed with hinge and squared_hinge. | | Linear SVC | 10 | Standard | ; hinge-loss and L1-penalized variants. | | SVC | 10 | Standard | RBF kernels over selected and ; linear, polynomial degree 2/3, and sigmoid kernels. | | Distance, centroid, and Gaussian-process models | ||| | -NN | 13 | â | with uniform/distance weighting for selected values. | | Nearest centroid | 10 | â | Euclidean/manhattan metrics with shrink thresholds in plus no-shrink variants. | | Gaussian process | 21 | Standard | RBF length scales from to ; MatĂ©rn ; constant-RBF and optimizer/multiclass variants. | | Tree and ensemble models | ||| | Decision tree | 10 | â | Depths with Gini; selected entropy variants. | | Extra tree | 10 | â | Depths with Gini; selected entropy/log-loss variants. | | Random forest | 11 | â | Selected combinations of and depth . | | AdaBoost | 11 | â | Selected combinations of and learning rate . | | Gradient boosting | 11 | â | Selected combinations of estimators, learning rate, and depth; includes the default reference model. | | Histogram gradient boosting | 10 | â | Max iterations , learning rate , and selected leaf counts. | | Bagging | 10 | â | Selected combinations of and max-sample fractions . | | Neural, naive-Bayes, discriminant, and dummy models | ||| | MLP | 10 | Standard | Hidden layers from one- and two-layer networks; selected and learning-rate values. | | Gaussian NB | 10 | â | Variance smoothing in . | | Bernoulli NB | 10 | â | Selected combinations of and binarization threshold . | | Multinomial NB | 10 | Minâmax | . | | Linear discriminant analysis | 10 | â | Solvers svd, lsqr, and eigen; selected shrinkage values. | | Quadratic discriminant analysis | 10 | â | Regularization in . | | Dummy classifier | 4 | â | Strategies most_frequent, prior, uniform, and stratified. |
The results reported in this paper are obtained with Claude Opus 4.6. We also ran the 2D behaviour matching experiments with Qwen3-235B-A22B. The highest agreement between Qwen and the sweep of baselines is shown in the following plot: