[AAAI-25] A Novel Augmentation Technique in Truncated SVD-Based Representation Space for Improving Tabular Data Prediction Performance

The Association for the Advancement of Artificial Intelligence (AAAI) is one of the most prestigious international conferences in the field of artificial intelligence, where the latest AI technologies and research are shared and discussed. 

LG AI Research's Data Intelligence (DI) Lab traveled to this year's AAAI-25 to present our research on tabular data and share insights on various papers with researchers from around the world. AAAI-25 featured a number of notable research presentations with academic depth and industrial applicability, covering a wide range of topics including Self-Supervised Learning (SSL), Large Language Models (LLMs), Multimodal Learning, Graph Neural Networks (GNNs), Reinforcement Learning (RL), Tabular Data, and more.

In particular, research on AI alignment and reliability gained traction, and discussions continued on how to make large-scale models safer and fairer to use in the real world. Topics such as fairness, robustness, interpretability, and human-AI collaboration emerged as key research directions and a variety of approaches were proposed to explore how AI systems can effectively collaborate with humans. In addition, as multimodal data processing and the utilization of large-scale models have become a major trend, we saw an increase in the number of applications of AI beyond existing image and natural language processing to fields as diverse as science, healthcare, urban analytics, and finance. This widespread use of AI has also led to increased interest in tabular data, which is widely used in everyday life. As a result, this year's conference featured more research on tabular data than in previous years.

This article will introduce the study Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data^[10] presented by LG AI Research at AAAI-25, and highlights notable research in the field.

LG AI Research’s “Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data^[10]” 

Real-World Challenges and Research Inspiration

While conducting AI research to improve the efficiency and productivity of LG affiliate factories, LG AI Research identified a key challenge: effectively utilizing the massive amounts of tabular data generated by manufacturing processes and inspections. In manufacturing, key decisions such as process optimization, quality control, equipment maintenance, and production forecasting are made based on tabular data consisting of sensor data, equipment logs, process variables, etc. However, this data is high-dimensional, heterogeneous, and contains complex interactions between features, making it difficult to learn effectively using existing methods.

Previously, tree-based models such as XGBoost^[1] were mainly utilized to solve these problems, but they had scalability issues when learning large-scale data and limitations in representation learning in high-dimensional data. As a result, LG AI Research explored deep learning-based approaches that enable more robust representation learning, especially by utilizing SSL techniques, which have recently gained attention^{[2, 3, 4]}. However, existing Contrastive Learning-Based SSL techniques are optimized for image or text data, making it difficult to effectively apply them to tabular data with large structural differences. In particular, existing data augmentation techniques (e.g., Masking^[5], Shuffling^[6], CutMix^[7] etc.) distort the relationship between key features in tabular data or cause information loss.

Against this background, LG AI Research wanted to develop a new methodology to overcome the limitations of existing data augmentation and learning methods and enable deep learning models to learn effectively from tabular data. To this end, we proposed the RaTab (Representation Space Augmentation for Tabular Data), which performs data augmentation in the representation space rather than the existing augmentation method in the input space and was accepted by AAAI-25.

Key Research Details 

RaTab is a new way to perform representation space augmentation to maximize the effectiveness of SSL on tabular data. Unlike existing data augmentation techniques that directly transform in the input space, RaTab focuses on transforming the weight matrix in the last MLP layer of the encoder to create a more meaningful representation.

Image 1. RaTab overview^[10]
a) Schematic Diagram of Self-Supervised Pre-training with RaTab Augmentation Technique Applied;
(b) RaTab Augmentation Module

1) Representation Space Augmentation

RaTab performs augmentation in the representation space based on truncated SVDs while preserving key information to effectively reflect the characteristics of the tabular data. Existing augmentation techniques in the input space have a high risk of damaging the important structure of the data by transforming individual features and distorting the interactions between features. To overcome these limitations, RaTab performs augmentation by transforming the weight matrix of the last MLP layer of the encoder and applying truncated SVD to achieve expression diversity while preserving the core information of the data.

2) Truncated SVD-Based Augmentation

RaTab decomposes the last layer weight matrix $W$ of the encoder using SVD.

$W=U\Sigma V^T$

 

Here, $U$ is the left singular vector matrix, $\Sigma$ is a diagonal matrix containing the singular values, and $V^T$ is the right singular vector matrix.

$W_k=U_k{\Sigma }_k{V}_k^T$

 

Here, $U_k,{\Sigma }_k,V_k^T$ is a truncated SVD version that selects the top $𝑘$ singular values from the original matrix, and the $𝑘$values are set as a hyper-parameter, which is kept at a certain percentage of the total rank (e.g.,  50%, 60%, 70%). The encoder that applies the weight matrix transformed in this way is called $\widehat{f_{en}}$ and a new augmented representation is generated using it.

$Z_{aug}=\widehat{f_{en}}\left(x\right)$

 

The key to this process is to preserve ${\Sigma }_k$ which serves to preserve important information in the data. As the singular value matrix Σ contains the main components of the learned representation, using transformed weights based on it can create a new representation while preserving the structure of the original data. Thus, unlike existing input space augmentation methods, RaTab utilizes a low-rank approximation in the representation space to maximize the augmentation effect while minimizing information loss.

Additionally, RaTab applies dropout to further increase the diversity of the representation space. While truncated SVD retains important information, it utilizes dropout to randomly remove parts of the weight matrix to encourage the generation of many variations. This allows the model to learn a more generalized representation and also serves to prevent it from overfitting to a particular pattern. As a result, applying dropout allows the model to learn a more robust representation, and allows it to experience a wider range of data variations.

This approach is differentiated from existing contrastive learning-based augmentation techniques. Whereas existing methods directly transform the input data, RaTab performs the transformations in the learned representation space itself, maximizing the effectiveness of the augmentation while preserving the intrinsic structure of the data. This enables more robust representation learning, even on tabular data, and makes contrastive learning more stable.

3) Results

Image 2. Performance Comparison of Various Existing Augmentation Techniques and the RaTab Augmentation Technique^[10]

Image 3. Performance Comparison between the Previous SoTA Algorithm and RaTab^[10]

The experimental results demonstrate that RaTab effectively improves the performance of contrastive learning-based SSL on tabular data. In comparative experiments with existing methods on several public datasets (e.g., OpenML), RaTab consistently outperforms augmentation techniques (e.g., Masking, Shuffling, CutMix, etc.) that are performed in the traditional input space. In particular, when combined with the latest tabular-specific neural network models such as FT-Transformer^[8] and T2G-Former^[9] , it has shown competitive performance with tree-based models (XGBoost, LightGBM).

In conclusion, we experimentally demonstrated that RaTab is an effective way to optimize representation learning on tabular data through augmentation in representation space, which can improve learning performance with minimal information loss compared to traditional input space augmentation methods.

Notable Tabular Data-Related Research at AAAI-25

Various approaches to effectively learn the complex structural properties of table data are being investigated, particularly in the following directions: data augmentation in representation space, learning interactions between columns, and modeling irregular target functions. In this context, this post will introduce two studies designed to effectively learn complex interactions between columns in table data or to model irregular target functions more precisely.

1) APAR: Modeling Irregular Target Functions in Tabular Regression via Arithmetic-Aware Pre-Training and Adaptive-Regularized Fine-tuning^[11]

Typically, regression problems in tabular data are characterized by rapid changes in labels even for small changes in input features. This characteristic makes deep learning models poorly learn the irregular patterns of the objective function and vulnerable to overfitting. To address these issues, APAR proposes a new framework that combines pre-training and fine-tuning techniques. The core of APAR consists of two steps.

Image 4. Explanation of arithmetic-aware pre-training^[11]

1. Arithmetic-Aware Pre-training: To address the problem that traditional SSL techniques do not work well with table data, APAR introduces a method that learns by leveraging arithmetic operations between samples: randomly selecting two samples, performing addition, subtraction, multiplication, and division operations on their label values, and then learning to predict them. This allows the model to learn the relationships between samples more precisely and enables more robust representation learning than existing methods.

Image 5. Adaptive-Regularized Fine-Tuning^[11]

1. Adaptive-Regularized Fine-Tuning: To avoid overfitting while preserving the relationship between features in table data, APAR applied an adaptive data augmentation technique that reflects feature importance. The multivariate Bernoulli distribution is used to learn gate vectors that reflect the correlation structure between features and use them to augment the data. In particular, it is designed to normalize the original data to maintain consistency between the augmented data and the original data, so that the model has more robust generalization performance. We conducted comparative experiments with existing methods on 10 public datasets (OpenML, etc.) and found that APAR achieved RMSE performance improvements of 9.43% to 20.37% over existing GBDT-based (XGBoost, LightGBM, CatBoost) and neural network (NN)-based models (MLP, AutoInt, FT-Transformer, TabNet, etc.).

2) ICE-T: Interactions-aware Cross-column Contrastive Embedding for Heterogeneous Tabular Datasets^[12]

Image 6. Explanation of ICE-T^[12]

To solve this problem, ICE-T considers each column as an independent modality and applies the concept of multimodal contrastive learning to tabular data. In other words, it encodes the multiple columns that make up a sample separately, selects one specific column, and anchors the representation of that column and the aggregate representation from the other columns to form a positive pair. For example, if the entire data consists of M columns, the representation of variable $m$, $ℎ(𝑚)$ is the latent representation obtained by passing through only the data corresponding to variable $m$ and the anchor $𝜇(\backslash 𝑚)$, is computed as the average of the representations of the remaining $M-1$ variables. These two representations are passed through the same neural subnetwork $g$ to be transformed into the final embedding vectors  $z\mathit{(}\mathit{𝑚}\mathit{)}$ and $z\mathit{(}\mathit{\backslash }\mathit{𝑚}\mathit{)}$, which are trained by maximizing the cosine similarity between them.

This approach has two strengths: First, it can implicitly reflect interactions between columns. While the anchor is simply the average of the representations of the other variables, the representation of a variable $m$ that is trained to be highly similar to it will inevitably contain information about its association with the other variables. Second, it is computationally efficient. While existing pairwise contrastive loss grows in computation as $𝑂\left(M^2\right)$ for the number of variables $M$, ICE-T only needs to compute the anchor once for each variable, resulting in a linear time complexity of $𝑂(𝑀)$. This approach also has the advantage of versatility in representation. After learning, ICE-T is designed to efficiently generate not only single-variable embeddings but also full-sample embeddings and partial embeddings for arbitrary subsets of variables, which can be very useful for downstream tasks such as missing value imputation or modality translation in the future. 

In addition, ICE-T consistently outperformed existing SSL-based methods such as SCARF, SubTab, and VIME on various tabular datasets. In particular, ICE-T achieved high prediction performance even with only a small number of labels and outperformed existing methods using only a linear classifier without fine-tuning. The performance improvement of ICE-T was more pronounced in datasets with significant inter-column interactions, suggesting that ICE-T's proposed method of generating positive pairs based on inter-column relationships actually contributed to learning meaningful representations. In addition, we experimentally demonstrated that ICE-T is more computationally efficient than existing contrastive learning techniques and has a high potential for downstream tasks such as missing value imputation, partial feature prediction, and modality translation. These results show that ICE-T is a practical and scalable SSL framework that reflects the structural characteristics of tabular data, rather than just improving performance.

Future Plans

LG AI Research is focused on extending RaTab to broaden the scope of tabular data research and develop technologies that can operate stably in real-world environments. Tabular data is essential in many industrial domains, including finance, healthcare, and manufacturing, but real-world challenges such as real-time data processing, variability in data scale, and domain shifts are exposing the limitations of existing approaches.

As a result, researchers are exploring techniques to effectively deal with various issues related to data quality, such as noise, missing values, unbalanced data, and shifting distributions, to build a more reliable learning environment. Beyond just improving performance, we are also looking at robustness and generalization throughout the learning process to ensure that our models can perform reliably in actual industrial sites.  

In addition, we are focusing on developing a general-purpose learning framework and tabular foundation model that can be adapted to various domains and tasks by fundamentally understanding the structural characteristics of tabular data beyond simple data augmentation techniques. This research is expanding to cover the entire process from data collection and preprocessing to learning, optimization, and generalization, with the goal of providing highly reliable and scalable AI solutions for industrial environments.

In the future, it is expected that the tabular foundation model will evolve to integrate various research topics such as Few-shot Learning, Explainability, Domain Adaptation, and Multimodal Learning, centered on the tabular foundation model. In particular, it will be an important challenge to combine the versatility of a single model to adapt to various environments and tasks with a transparent learning structure that is easy for users to interpret and control.  

Based on this direction, LG AI Research plans to continue to lead the development of next-generation tabular AI technologies that create real value in industrial settings.

AAAI-25 series

EP.1 [AAAI-25] Study on Denoising Distillation Model Using 3D Coordinates