[ICML 2024] Tabular Learning Research Trends

The Data Intelligence (DI) Lab at LG AI Research presented the paper “Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains”^[1] at ICML 2024. This paper was previously presented at the NeurIPS 2023 Table Representation Learning Workshop. For further details, please refer to the previous blog post and the poster in Image 1.

In summary, to overcome the limitations of deep networks in tabular learning, this paper utilizes a binning algorithm to convert numerical variables into discrete variables and performs a pretext task to predict the discrete variable corresponding to the bin index in unsupervised learning. The learned representation is evaluated via linear evaluation or fine-tuning depending on the data-specific downstream task. In the workshop paper, only MLP and FT-Transformer structures, which are relatively simple encoders, were used, whereas in this paper, one of the recently published tabular learning model structures, T2G-Former^[2], was also utilized. This paper further proposes various experimental results including visualization in the discussion to illustrate the performance improvement of this method.

Image 1. Paper Presentation Poster for “Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains”

A total of 2,634 papers were presented at ICML 2024, including 76 in the Position Track. Among these, 14 papers were presented as tabular domain-specific research, including our paper (2 Spotlight, 11 Poster, 1 Position Track). An additional 10 papers were presented that were not tabular domain-specific, but used tabular datasets for evaluation. Of the 13 studies specialized in the tabular domain excluding the Position Track, 3 studies utilized language models, 3 studies utilized inductive bias related to tree models, 2 studies related to graphs, and 3 studies related to interpretable AI. Compared to other domains, the proportion of studies that utilized large language models (LLMs) was low, and the majority of studies proposed new methodologies targeting specific problem setups (e.g., extremely small sample size, anomaly detection, unsupervised learning, etc.).

Two of the tabular domain-related research papers are outlined below.

1. Tabular Insights, Visual Impacts: Transferring Expertise from Tables to Images

This paper covers image-tabular multimodal learning, which aims to improve the performance of image-based prediction by using tabular data as an additional input to learning. This data is often obtained from experts but is relatively expensive to acquire. For tabular data, there are limitations in utilizing multimodal learning techniques between image and natural language in the current state because it contains a mixture of numerical and categorical variables. Furthermore, not all information in tabular data is relevant to a given image, so it is necessary to find out which information or variables are relevant to the wanted image.

With these limitations in mind, this paper proposes “Channel Tabular Alignment with Optimal Transport” (CHARMS) as a new methodology (Image 2). To summarize this methodology, the paper uses an approach that maximizes the mutual information (MI) between the visual prediction and the selected tabular attributes to align attributes in the tabular data with the image channels.

Let’s take a closer look at the details. First, for image data, ResNet is used to extract a representation. In this case, the representation corresponds to the layer before the final average pooling, or in other words, a representation of size H*W*C. In general, images may have redundant semantics depending on the channel, so this paper uses C′ channels, which are smaller than C, by grouping similar channels using K-Means clustering so that each channel has more independent meaning. Next, for tabular data, FT-Transformer^[3] is used to extract representations for each attribute, resulting in a representation of size (number of attributes) * (embedding dimension). As a result, for images, the end result is a representation of H*W size for C’ channels, and for tabular data, the end result is a representation of size (dimension of embedding) for as many attributes as (number of attributes).

However, it is difficult to measure the similarity of the two representations directly as they are not only different in modality but also in dimensionality and meaning. In this paper, the cosine similarity between samples for each channel (i) or attribute (j) is calculated for image and tabular representation, respectively, and defined as $S_i^{Image},S_j^{Tabular} \in {\mathrm{ℝ}}^{N\times N}$, and the difference between the two is defined as the cost matrix. In other words, the cost matrix is defined as  $C_{ij} = {{\left|| S_i^{Image}-S_j^{Tabular }|\right|}^2}_2$, which is the difference between the channel-wise similarity of the image and the attribute-wise similarity of the tabular data. Based on the defined cost matrix, the optimal transport method is used to calculate the optimal transport matrix $\hat{T}$ that minimizes the sample similarity between the different modalities. This makes it possible to mathematically measure the similarity between probability distributions and find the optimal way to move mass from one distribution to another. As a result, the representation obtained from the image can be multiplied by the optimal transfer matrix $\hat{T}$ to align it in the same space as the representation obtained from the tabular data. Further, selective application of important tabular attributes can also be achieved in this process. Lastly, a loss function that maximizes the mutual information (MI) between the two aligned vectors is also used. Indirectly taking advantage of the positive correlation between predictability and MI, the approach used in this paper is meant to maximize the predictability between the image representation and the tabular attributes multiplied by the optimal transfer matrix

Image 2. 2 Schematic Diagram of the Proposed Methodology CHARMS^[4]

To verify the effectiveness of this methodology, experiments were conducted on six multimodal datasets. Results showed that it not only outperformed other multimodal learning methods, but also outperformed learning methods on tabular or single domain of images. Furthermore, tabular knowledge was also shown to be beneficial in terms of interpreting the results.

This paper is a pioneering study that proposes a multimodal learning methodology for image-tabular data, a field that has not been well studied to date and is expected to be very helpful for follow-up research by making both code and datasets public. It shows that multimodal learning performance can be improved by maximizing the mutual information between image and tabular representations, suggesting the potential of using tabular data as auxiliary information.

2. Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

This paper proposes the FeatLLM framework, which utilizes a large language model (LLM) as a feature engineer to generate optimal input data. This data can then be used with simple machine learning models such as linear regression to achieve high performance with a small number of samples. In other words, by utilizing the power of LLMs to generate features, once a specific feature is discovered, accurate predictions can be made without the need for complex machine learning models, thereby improving inference time.

Image 3. Schematic Diagram of the FeatLLM Methodology^[5]

FeatLLM extracts rules corresponding to each class from a small number of samples, as shown in image 3, and processes them with LLM to generate new binary features. These binary features are input into a dot product operation with non-negative training weights, which is used to estimate the probability of corresponding to each class.

To extract rules, FeatLLM first designs prompts. It encourages the LLM to infer causal relationships or trends between features and task descriptions based on general knowledge or common sense, all without referring to example demonstrations. Next, it generates reasoning instructions that encourage the LLM to use the example demonstration and information from the previous step to infer rules for each class. Finally, to make the inferred rules easier to use, LLM structures its responses using logical operators such as AND or OR. It then uses the generated rules to create new binary features for each class. The final prediction is made by calculating how many of the defined rules per class are satisfied, further utilizing the ensemble through bagging. Through experiments on 12 datasets, performance improvement was confirmed to have increased by an average of 10% over the existing baseline.

This paper is significant as it confirms the feasibility of LLM in the tabular domain in a few-shot setting, as LLM has been recently shown to be useful in various domains. This study is different from previous studies in that it focuses on generating new rules, as opposed to previous studies that simply tried to process data or feature descriptions with LLMs. This suggests the possibility of utilizing various LLMs in the tabular domain.

As announced at the ICML 2024 Position Track^[6], research related to the tabular domain tends to be underrepresented, despite its importance. However, the number of research papers on tabular learning has increased significantly every year and is growing rapidly. Based on discussions with various researchers at ICML 2024, we believe that the tabular domain is currently at a point where it is necessary to develop technologies that can be applied in an understandable manner in specific problem situations. 

Image 4. Visitors, oberserving the poster presented by LG AI Research at ICML 2024 

LG AI Research also presented papers related to tabular learning at NeurIPS 2023 and ICML 2024 and had discussions with various practitioners and researchers on the future direction of tabular learning development. Currently, DI Lab at LG AI Research is working on developing technologies for process data-based prediction and applying them in the field, especially in relation to process automation, which is one of our DI Lab’s major challenges. We are actively conducting research from the perspective of defining problem situations that are important in the tabular domain and developing technologies for each problem situation. We are actively conducting research to develop technology that can respond to: (1) prediction in situations where there is no or very little labeled data, such as anomaly detection; (2) prediction in situations where the amount of data is very small and realistic data is generated; and (3) stabilization of unstable learning due to excessive number of hyperparameters in the model. If we can develop a model that performs well with tabular data, which is often used not only in research but also in various industries, we anticipate that we will be able to create significant value industrially. LG AI Research DI Lab plans to conduct more advanced tabular learning research based on the various research methodologies and results discussed at ICML 2024, as well as through our own various experiences and gained insights.

▶ICML 2024 Review Series : Solving Combinatorial Optimization with the Diffusion Model