[NeurIPS 2024 Series] Introduction to Mutual Information Estimation Benchmark Research

A total of 4,337 papers were presented at NeurIPS 2024. As shown in Figure 1, the main keywords of the conference were Large Language Model (LLM), Reinforcement Learning, Multimodal Learning, and Diffusion. In particular, studies related to Causal Learning and Biology/Chemistry were more prominent than in previous years.

Image 1. Visualization plot of papers presented at NeurIPS 2024^[1]

In the Datasets & Benchmark track, we saw a lot of interesting studies,such as large-scale multimodal aerospace benchmarks^[2], the IKEA Video Manuals for shape assembly^[3], and multi-image reasoning benchmarks^[4].These studies show that new challenges for deep learning are constantly being explored.

LG AI Research’s Data Intelligence (DI) Lab presented a study at NeurIPS 2024 in the Datasets & Benchmark track. Our study focused on addressing issues that are often overlooked in deep learning research and proposing benchmarks to more rigorously verify model performance. Specifically, our paper addressed the study of Mutual Information (MI) estimation, a measure of the relationship between two random variables often used in Representation Learning. A total of nine MI-related papers were presented at NeurIPS 2024, four of which were on MI estimation and the remaining five on utilizing MI to improve final task performance.

LG AI Research’s A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets^[7]

Mutual Information (MI) is a key metric in the field of information theory that measures the relationship (the amount of information shared) between two random variables (X, Y) and has been used in a variety of fields for a long time. MI has the property of Reparameterization-Invariance and has the ability to measure non-linear relationships. MI is utilized in a variety of fields, including:

1. Information Bottleneck Method

2. Deriving Non-vacuous Generalization Bounds

3. Analyzing Neural Network (NN) Training Dynamics

4. Optimization in various fields such as Representation Learning, Self-supervised Learning, and Reinforcement Learning

However, there is one major problem with utilizing MI. To calculate the true value of MI, you need to know the exact joint distribution and marginal distribution of two random variables, which is often not available in the real world. Therefore, researchers base their discussions on estimated MI values, but it’s not easy to verify how close those values are to the true MI.

Most MI estimation techniques developed to date have only been evaluated on the Gaussian Multivariates benchmark. However, the data we deal with in real-world deep learning research is often unstructured data, such as images and text, which is very different from Gaussian. Therefore, the question of whether existing Gaussian benchmarks ensure the reliability of realistic MI estimation has remained unanswered. To address this issue, we sought to determine whether MI estimation techniques validated on existing Gaussian datasets consistently perform well on real-world image and text data.

1) Significance of This Study

This study has three main contributions:

1. Suggestions for how to generate data to know exactly what the true value of MI is, even for arbitrary datasets

2. Building benchmarks to evaluate the accuracy of MI estimators on Gaussian Multivariates, Images, and Sentence embeddings

3. Comparing MI estimator performance and summarizing experimental results based on seven scenarios

2) How to Create a Dataset with Accurate MI Values

To generate a dataset for which we know the true MI, this study utilized same-class sampling as positive pairing^[5]. The key idea is to allow only the class information to be shared between two random variables X and Y, such that the true MI can be proven to be the same as the entropy of class variable C, i.e., I(X; Y ) = H(C).

For image data, we generated image pairs sharing a specific number class (0, 1) from MNIST data, and for text data, we utilized BERT Fine-Tuned Sentence Embeddings from the IMDB dataset to generate sentence vectors sharing Positive/Negative class. To construct a dataset with larger MI, we concatenate multiple pairs of samples as shown in Figure 2b and 2c. For images, we can insert random samples from other datasets (e.g., CIFAR-10) as nuisance to X and Y to make the dataset more realistic without affecting true MI value, as shown in Figure 2d. Because the source images remain on top without any occlusion, and there is no fixed relationship between the background chosen for X and the background chosen for Y, the nuisance does not affect the true MI. To manipulate the true MI and construct a dataset with a non-integer MI value between 0 and H(C), we adopt the concept of binary symmetric channel.

Image 2. Example of generated image dataset^[7]

3) Experiment Results

We have developed a benchmark suite based on our method, encompassing three data domains for Gaussian multivariates, images, and sentence embeddings. To demonstrate its usefulness, we examine performance of several neural MI estimators over seven key aspects:

1. Choice of critic architecture: superiority of joint critic for unstructured datasets

2. Choice of critic capacity: larger capacity does not ensure a higher estimation accuracy

3. Choice of MI estimator: no universal winner exists across the three data domains

4. Number of information sources: unstructured datasets outperform Gaussian in handling large number of information sources

5. Representation dimension: it does not affect the estimation accuracy

6. Nuisance: MINE^[6] turns out to be relatively robust

7. Network and layer dependency: estimation holds for invertible networks and upper layers

Our findings reveal discrepancies in estimation accuracy between traditional Gaussian benchmarks and unstructured data scenarios, highlighting the limitations of Gaussian benchmarks in capturing the nuances of MI estimation in practical settings. Notably, our findings on unstructured datasets demonstrate that MI estimators can yield remarkably accurate  results, particularly in conjunction with deep representations, indicating their potential to continue driving advancements in deep learning research. While our study does not cover the entire spectrum of real-world datasets, it signifies a substantial step forward in evaluating and understanding MI estimators beyond purely statistical datasets. We hope that this benchmark suite not only offers a new standard for evaluating MI estimators but also catalyzes further research, enriching our comprehension of MI across a diverse data domains.

Future Plans

This study proposed a benchmark to verify how accurately Mutual Information estimation can be performed in a real-world data environment. LG AI Research plans to continue our research to solve the problems faced by various deep learning models. In particular, we plan to further analyze the utility of MI estimators in Representation Learning, improve the generalization performance of deep learning models by utilizing MI values, and conduct follow-up studies focused on developing deep learning algorithms that can solve problems in various industrial domains.

 

NeurIPS 2024 Series

EP. 2 [NeurIPS 2024 Series] B-splines-Based Input Value Normalization Method for Tabular Data
EP. 3 [NeurIPS 2024 Series] Attention-Based Data Augmentation Techniques to Enhance Performance of Tabular Data Prediction
EP. 4 [NeurIPS 2024 Series] A Study on Adaptive Information Routing Technology for Multimodal Time Series Forecasting