[ICML2026] Foundation Models for Structured Data and the Current State of Time Series Deep Learning

Introduction

Deep learning, which has evolved primarily around images and language, has achieved remarkable results; however, a significant portion of the data accumulated daily in industrial settings still takes the form of tabular and time series data. Sensor logs from manufacturing equipment, semiconductor yield data, financial transaction records, and hospital electronic health records are all sequences of numbers organized along rows and columns or a time axis. While this structured data is the most common type encountered in industrial settings, paradoxically, it is also the area where deep learning has made its entry the latest. This is evidenced by the fact that tree-based models such as XGBoost and LightGBM have served as the de facto standard for the past decade or so.

However, this trend has been rapidly shifting over the past two to three years. This is because foundation models, which are pre-trained on large-scale synthetic data and perform predictions with a single inference without requiring additional training, have begun to make a significant impact in the realm of structured data as well. In the realm of tabular data, TabPFN was published in Nature and demonstrated performance that surpassed existing tree-based models^[1], while in time series data, foundation models capable of zero-shot prediction have also emerged one after another. The long-held assumption that “a model must be retrained for each dataset” has begun to waver.

These changes were clearly evident at ICML (International Conference on Machine Learning) 2026, the most prestigious conference in the field of machine learning. LG AI Research participated in ICML 2026, held at COEX in Seoul from July 6 to 11, 2026, to present our research findings and review the latest trends in structured data and time series analysis. This post summarizes the research presented by LG AI Research, along with notable time series deep learning studies from the conference.

ICML 2026 held in Seoul

ICML 2026, marking its 43rd edition this year, was the first ICML conference ever held in South Korea and took place at COEX in Seoul. A total of 23,918 papers were submitted, of which 6,352 were accepted (an acceptance rate of 26.6%). The number of submissions nearly doubled compared to the previous year^[2].

Diffusion models took center stage at this year’s awards. The two Outstanding Paper Awards went to “The Flexibility Trap,” which reexamined the value of arbitrary-order generation in diffusion language models, and “High-Accuracy Sampling for Diffusion Models and Log-Concave Distributions,” which addressed high-precision sampling in diffusion models. The Test of Time Award went to the 2016 paper (“Asynchronous Methods for Deep Reinforcement Learning”), which served as a milestone in deep reinforcement learning^[3]. While generative models are garnering significant attention, research on structured data is quietly but steadily expanding its scope. Numerous papers on time series were presented at the main conference, and the workshop on structured data foundation models marked its second year this year.

Research from LG AI Research: Why Do Tabular Foundation Models Miss Minor Classes?

Issue: Class bias in TabPFN

The research presented by LG AI Research at ICML 2026 is titled “Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment”^[4]. This study identifies the phenomenon where tabular foundation models experience a decline in performance under class imbalance, and proposes a method to mitigate this issue through a simple adjustment at the time of inference.

TabPFN is a foundation model for tabular data. After being pre-trained on a massive amount of synthetic data, it takes the entire training dataset as context when solving real-world problems and makes predictions with just a single forward pass. This is known as the In-Context Learning (ICL) approach, which differs from existing models in that it involves no parameter updates whatsoever. It has garnered significant attention for outperforming XGBoost on small datasets.

However, we discovered a vulnerability in this model that had been overlooked until now: label shift. This refers to a situation where the probability distribution of features by class, P(X|Y), is the same in both the training and test data, but the class proportions, P(Y), differ. In real-world industrial data, this situation is actually quite common. This is because in many cases, like defect detection, anomaly detection, and disease diagnosis, the target to be identified is a minority class.

In fact, our analysis of 253 tabular datasets from OpenML revealed that approximately 85% (214 datasets) exhibited class imbalance. Furthermore, TabPFN directly incorporates the class imbalance provided in the context into its predictions. In extreme cases, we observed that 98.3% of the total test samples were predicted to belong to the majority class, causing the recall for the minority class to effectively collapse. The fact that these results occurred even though training and testing were performed on the same dataset underscores the severity of the problem.

Solution: Correction at the inference time rather than the training time, DistPFN

To address this issue, we proposed DistPFN. The core idea is surprisingly simple. The prediction probability output is multiplied by TabPFN by an additional correction factor that indicates “how much higher this prediction is compared to the class ratio in the training data.” This factor decreases for classes with a higher proportion in the training data and increases for classes with a lower proportion. As a result, the influence of the class ratio in the training data on the prediction is reduced, and the model’s prediction results are given relatively greater weight. Mathematically, this is expressed as follows.

                            ┌─ Correction factor ─┐

$\tilde{p}\_DistPFN\left(y\right)=Norm\left(\stackrel{\wedge }{p}\right(y)\times \stackrel{\wedge }{p}(y)/p\_train(y\left)\right)$

Here, $\stackrel{\wedge }{p}$(y) is the class probability predicted by TabPFN (posterior distribution), p_train(y) is the proportion of that class in the training data (prior distribution), and Norm is a normalization that adjusts the sum of the scaled values so that it equals 1 again. The greatest advantage of this method is that it does not alter the model architecture in any way, requires no retraining, and does not even need to estimate the test label distribution. This contrasts with existing label shift correction techniques (such as EME and BBE), which required separately estimating the test prior distribution. This approach is possible because TabPFN-family models explicitly reference the context of the training data. In other words, they can utilize the class proportions of the training data as-is, without the need for separate estimation.

Furthermore, we also proposed DistPFN-T, which adjusts the correction strength according to the situation. This method uses the Cross-Entropy between the predicted distribution and the training prior distribution as a temperature value; the greater the discrepancy between the two distributions, the smoother the predicted distribution becomes, thereby preventing over-adjustment.

Image 2. Overall structure of DistPFN. The predictions of TabPFN, which are skewed toward the majority class (left),
are mitigated solely by distribution correction at the time of inference (right). (Source: Lee ^[4], Figure 4)

Experiment Results

We conducted comparative experiments using 16 baselines across 253 OpenML datasets. Additionally, to allow for stepwise adjustment of the degree of label shift, we designed an evaluation benchmark (shift intensity β) using inverse-frequency oversampling, which samples rare classes more frequently. The key results are as follows.

● We observed consistent performance improvements across all three types of tabular foundation models (TabPFN-v2, TabICL, and LoCalPFN). For example, under conditions with strong shifts (β=5), DistPFN increased TabICL’s accuracy by 12.3%, and DistPFN-T increased it by 13.7% (based on relative improvement rates).

● Under typical conditions without shift, they maintain their original performance. In contrast, existing calibration methods such as EME and BBE actually showed a performance decline of up to 1.7%p when there was no shift. In other words, this method is effective when shift is present, while causing no performance degradation when shift is absent.

● There is virtually no additional computational cost. Based on TabPFN-v2, the average prediction time increased only from 1.002 seconds to 1.003 seconds.

● The model’s performance approached that of an Oracle, which assumes knowledge of the true distribution of test labels. Based on the average accuracy across six shift intensity levels, the Oracle achieved 0.796, while DistPFN-T achieved 0.792.

In summary, the strength of DistPFN lies in its ability to improve reliability in class-imbalanced scenarios by adding a single, simple adjustment to the output stage while leaving the existing tabular foundation model as-is.

ICML 2026 research trends in time series deep learning

The area that LG AI Research focused on in particular at this conference was time-series deep learning. The trends in time-series studies presented this year can be summarized into four main categories: ① how to tokenize time series, ② what prediction models actually learn, ③ how to correct cumulative prediction errors, and ④ how to handle irregular real-world data.

① Rediscovering Tokenization: Byte Pair Encoding for Efficient Time Series Forecasting

This study reimagines the very method of dividing time series into tokens^[5]. Most time-series foundation models to date have used a method called “patching,” which groups a fixed number of samples into a single token. However, this approach wastes a significant amount of computational resources because it consumes the same number of tokens even for simple patterns, such as intervals where the value remains unchanged for a long time.

Drawing inspiration from Byte Pair Encoding (BPE), which is widely used in natural language processing, the authors proposed the first pattern-centric tokenization technique that creates a vocabulary by collecting frequently occurring motifs (short pattern fragments that recur in time series) and uses this vocabulary to adaptively compress time series. According to the paper, this motif-based tokenization improved the prediction performance of state-of-the-art time-series foundation models by an average of 40% while also increasing efficiency. The authors report that when combined with conditional decoding, a post-optimization technique that can be applied without gradient calculations or additional computational overhead, the MSE is reduced by up to 48%.

This study stands out for its shift in perspective, the realization that “time-series tokens do not necessarily have to be the same length.” It is significant not only as an example of how mature design principles from language models have been extended to time series, but also because it goes beyond a simple port and precisely targets recurring motifs, which is a characteristic unique to time series.

Image 3. After quantizing the time series into discrete symbols, repeating variable-length patterns are merged into a single motif token. (Source: Gotz et al. ^[5], Figure 1)

② Is the Model Really Learning “Dynamics”?: Time Series Forecasting Through the Lens of Dynamics

While the number of time-series forecasting model architectures has exploded in recent years, there has been a lack of explanation as to “why some models perform well while others do not.” This study attempts to answer that question from the perspective of dynamics^[6].

The authors hypothesized that a good forecasting model must learn the direct connection, or dynamics, between past and future time points, and proposed a framework called PRO-DYN to classify existing models from this perspective. As a result, two key findings emerged. First, low-performing architectures were only partially learning these dynamics. Second, the location of the block responsible for dynamics within the model is critically important; it was most effective when placed at the end of the model.

Rather than proposing a new state-of-the-art (SOTA) model, this study summarizes the design principles common to existing models, making it highly valuable for practical reference.

Image 4. A PRO-DYN system that decomposes a time series forecasting model into a composition of general processing (PRO) and dynamic (DYN) functions. (Source: Brachet et al. ^[6], Figure 1)

③ The Return of Classical Techniques: Reviving Error Correction in Modern Deep Time-Series Forecasting

When forecasting long-term time series, an autoregressive approach is often used, in which the model’s forecast values are fed back as input to predict the next period. The problem is that errors from previous predictions are incorporated into the inputs for subsequent predictions, causing the errors to snowball as the sequence progresses. This accumulation of errors is a chronic challenge in the field of time series analysis. While classical statistical models had an Error Correction Mechanism (ECM) to address this, it was difficult to apply it directly to deep learning models.

This study reexamines the role and necessity of ECM in the context of modern deep learning and proposes UEC-STD, a general-purpose error corrector that decomposes forecast values into trend and seasonal components and corrects each separately^[7]. A key feature is that it can be added to existing predictors without retraining them, regardless of the structure of the prediction model. The results showed significant improvements in calibration accuracy and robustness across four backbone models and ten datasets.

Our research, introduced earlier, also shares the concept of leaving the pre-trained model as is and adjusting only the output afterward. While there is a difference, as DistPFN intervenes only at the time of inference without any training, whereas UEC-STD trains a separate corrector, it is noteworthy that the approach of “restoring performance without retraining the model” has emerged in both tabular and time-series problems.

④ Time-Frequency Combination for Irregular Time Series: Bridging Time and Frequency

Real-world time series are not observed at regular intervals, as they are in textbooks. Medical records and IoT sensor data are recorded at varying times and asynchronously across different variables. In such irregular multivariate time series (IMTS), it is difficult to apply not only existing models that assume equidistant sampling but also Fourier transform-based techniques, which are particularly effective at capturing periodicity. This is because the Fourier transform also assumes that data is observed at regular intervals.

TFMixer addresses this issue by modeling both the time domain and the frequency domain simultaneously^[8]. First, it introduces a trainable non-uniform discrete Fourier transform (Non-Uniform DFT) to create a global frequency module that directly extracts frequency components even from irregular observation time points. This is combined with a query-based local time module that compensates for varying information density across intervals and selects and aggregates meaningful time patches. Finally, the representations from both domains are combined to generate a forecast, and periodic components identified in the frequency domain are mapped back to the time axis via an inverse NUDFT, thereby explicitly extending seasonality. Rather than abandoning frequency analysis due to irregularity, this study solves the problem by redesigning the transformation itself to accommodate the irregularity.

Image 5. Structure of TFMixer. It captures global periodicity using the learnable non-uniform discrete Fourier transform (NUDFT)
and local patterns using query-based patch joining. (Source: Qiu et al. ^[8], Figure 3)

Where Tabular and Time Series Meet: Foundation Models for Structured Data Workshop

On July 11, the final day of the conference, the FMSD (Foundation Models for Structured Data) workshop, which focuses on foundation models for structured data, was held in the COEX Grand Ballroom^[9]. Following its inaugural session at ICML 2025 last year, this marked the second edition of the workshop. This workshop provided an opportunity to observe a research trend that views tabular and time-series data not as separate fields but as a single set of problems. In particular, this year’s workshop focused on multimodal structured foundation models that integrate structured data with other modalities, such as using images and tabular data together in medical diagnosis or combining text and time-series data in electrocardiogram (ECG) interpretation.

Among these, the following presentations are particularly noteworthy.

● Causal Foundation Models for Time Series based on Prior-Data Fitted Networks^[10] - This study extends PFN (Prior-Data Fitted Network, a structure pre-trained on synthetic data that makes predictions based solely on the given context at the time of inference), which forms the foundation of TabPFN, to causal inference for time series. It serves as an example demonstrating that the tabular data ICL and time series forecasting addressed by LG AI Research ultimately stem from the same roots.

● TiViT: Time Series Representations for Classification Lie Hidden in Pretrained Vision Transformers^[11] - This study reexamines the possibility of cross-modality transfer from the perspective that pre-trained Vision Transformers already contain latent representations useful for time-series classification.

● Do Tabular Foundation Models Learn Rules or Memorize Exemplars?^[12] / Tabular Foundation Models Are Effectively Shallow^[13] - These are studies that critically analyze what tabular foundation models actually learn. They demonstrate a growing trend toward understanding how these models work, moving beyond mere performance competition.

● Foundations without Fundamentals: Zero-Shot Blind Spots in Time Series FMs^[14] - Selected as a “Spotlight” presentation, this paper highlighted blind spots in the zero-shot performance of time-series foundation models. This aligns with the concerns raised by LG AI Research regarding label shift vulnerabilities discovered in tabular data.

Conclusion: From performance to reliability

A notable shift at this year’s ICML 2026 is that the focus in the field of structured data is shifting from “how accurate” to “how reliable.” In the realm of tabular data, studies have emerged that dissect what foundation models learn, while in time series, studies have focused on verifying whether models actually learn dynamics or identifying blind spots in zero-shot learning. Our presentation on DistPFN also aligns with this trend, as it originated from the question, “Under what conditions does TabPFN fail to perform well?”

Data from real-world industrial settings is always imbalanced and irregular, and its distribution can change over time. Bridging these gaps, which are difficult to capture based solely on the average accuracy of benchmarks, is precisely where research on structured data creates tangible value. LG AI Research will continue its research on foundation models that encompass both tabular and time-series data, and we will continue to develop AI technologies that are robust to changes in data distribution and reliable in real-world settings.