[NeurIPS 2024 Series] B-splines-Based Input Value Normalization Method for Tabular Data

NeurIPS, the leading conference in machine learning, has hosted the Table Representation Learning (TRL) Workshop every year since 2022 to address research on effective learning methods for tabular data. As researchers’ interest in tabular data continues to grow, the size of the TRL Workshop has also expanded to keep pace with this rising interest.

At this year’s NeurIPS TRL Workshop, LG AI Research's Data Intelligence (DI) Lab presented a new normalization technique for numerical input that can maximize the learning performance of tabular data. The study, titled “Learnable Numerical Input Normalization for Tabular Representation Learning based on B-splines^[1],” proposes a normalization technique based on B-splines that can learn and respond accordingly to the complex and heterogeneous characteristics of tabular data.

Tabular data is characterized by heterogeneous and diverse distributions. The process of normalizing input values is particularly important as each column is composed of different features and requires a normalization technique suitable for each distribution and characteristic. Although conventional normalization methods such as min-max, standardization, and quantile transformation can be easily applied, they are often not optimized during the learning process of neural networks and are only effective for certain distributions. As a result, the performance of neural networks in tabular data is highly dependent on which normalization method is chosen. The process of finding and applying a suitable normalization method for the distribution of all variables requires considerable time and resources. Therefore, this study proposes a normalization technique that overcomes these limitations and dynamically adjusts the transformation according to the learning difficulty of the data sample.

LG AI Research’s Learnable Numerical Input Normalization for Tabular Representation Learning based on B-splines^[1]

A normalization technique is expressed as a process of converting non-normalized raw input into normalized input through a specific continuous function. The normalization technique for one variable can be represented by a corresponding curve. In this study, we implemented the normalization technique using B-spline, one of the parametrized curves, to implement a normalization technique that can change according to the characteristics of the data. The parameters of the curve are then optimized simultaneously with the learning of the neural network.

Image 1. B-spline-Based Input Normalization^[1]

The Image above is one of the normalization curves we found using the method proposed in this study. As shown, the normalization technique of converting raw inputs into normalized inputs can be represented by a single curve. Further, the slope of the curve determines how much of the normalized input is occupied by a particular range of raw inputs. By adjusting the slope of the curve, we can specify the region of raw inputs that the neural network should learn with high importance.

The B-spline was chosen for this study for the following reasons: 1) B-splines can easily calculate the slope over any range of inputs by setting a specific order, allowing the slope of the curve to be adjusted freely; 2) the knot vector and control points can be adjusted to produce a smooth and continuous curve; and 3) by keeping the knot vector non-decreasing, the normalized input preserves the order of the raw input data, ensuring the stability of the normalization.

In this study, we identified the optimal B-spline-based normalization technique for each variable based on the following two assumptions.

1. If a normalization technique reduces a wide range of raw inputs to a narrow region of the normalized input, neural networks trained with that normalization technique will assign relatively low importance to the range of raw inputs.

2. If a data sample has a low training loss, then that data sample is trivial, easy to distinguish, and of low importance.

Given the above two assumptions, the parameters of the B-spline curve are learned through the following loss function.

$Loss\mathit{ }of\mathit{ }a\mathit{ }Parametrized\mathit{ }Curve\mathit{ }\mathit{=}\mathit{ }\frac{Loss of a Data Sample}{Slope\mathit{ }of\mathit{ }the\mathit{ }Parametrized\mathit{ }Curve}$

The parameters of the curve found using the proposed loss function will have a small slope for data samples with low learning loss, thus corresponding to a narrow region of normalized raw input around those data samples.

Image 2. B-spline Changes According to Learning of Neural Networks^[1]

Image 2 shows how the method proposed in this study learns the normalization curve corresponding to each variable. At the beginning of the learning process, it takes the form of a simple linear transformation due to the lack of information about the variable in question. As learning progresses, we can see that the normalization technique learns the importance of the data samples, giving more weight to the important raw input areas.

The method proposed in this study can be applied in conjunction with recently proposed methods for efficient learning of neural networks on tabular data. For example, embedding methods such as periodic embedding and piecewise-linear embedding, which are mainly used to represent numerical features or transformer-based effective neural network structures such as FT-transformer and T2G-former, can be applied to realize more efficient neural networks for tabular data.

Image 3. Performance Results on OpenML Datasets, Average of 26 Datasets^[1]
(The values in bold represent the performance average of the first rank, and the values in underline represent the next rank.)

Tabular data plays an important role in real-world industries. Thus, AI technologies that can process it effectively are expected to create significant added value in advanced industries such as process automation, smart factories, and data-driven decision-making.

In order to effectively process such data, LG AI Research is continuously conducting various studies specialized for tabular data. In addition to the data preprocessing method proposed in this study, we are working to improve the efficiency of neural networks on tabular data by developing data augmentation techniques and efficient learning methods that can compensate for the insufficient number of samples. We are also working on generating similar data using LLMs, which have gained attention in recent years, and foundation models, which can be quickly solved even when given a new task by pre-learning a large number of tables in consideration of situations with extremely few or no sample data, such as anomaly detection. LG AI Research plans to showcase a variety of research achievements at global conferences and academic events. Stay tuned for more exciting updates!

 

NeurIPS 2024 Series

EP. 1 [NeurIPS 2024 Series] Introduction to Mutual Information Estimation Benchmark Research
EP. 3 [NeurIPS 2024 Series] Attention-Based Data Augmentation Techniques to Enhance Performance of Tabular Data Prediction
EP. 4 [NeurIPS 2024 Series] A Study on Adaptive Information Routing Technology for Multimodal Time Series Forecasting