[ICML 2023] Conference Review and the Latest Research Trends

ICML 2023 (The International Conference on Machine Learning), one of the world's top artificial intelligence conferences, was held in Hawaii, USA from July 23 to July 30, 2023. Researchers from LG AI Research participated and engaged in various activities, including presenting research results at the conference and workshop.

At the conference, Data Intelligence Lab presented “Gradient Surgery for One-shot Unlearning on the Generative Model” during the ICML 2023 Workshop for Generative AI & Law. Our researchers had a chance to discuss legal regulations and issues (personal information protection, intellectual property rights, etc.) that must be considered when deploying Generative AI-based models such as GPT-3 and Stable Diffusion into actual services, as well as information on appropriate training and evaluation techniques with ML researchers from various backgrounds. Additionally, by attending several presentations, we were able to explore the latest research trends and gain useful insight.

The following information covers Unlearning issues and published research. Also introduced are topics on applied research in the fields of Causal Inference, Geometric Deep Learning, and Large Scale Foundation Model, which are areas that LG AI Research is currently focusing on and received increased attention at the conference.

1. Unlearning

At ICML 2023, LG AI Research proposed and presented a simple and effective method to unlearn generative models^[34] at the Workshop for Generative AI & Law. Unlearning is removing information of a certain instance from pre-trained models in response to changes in data privacy-related laws and regulations, or user requests for personal information protection when pre-trained models such as GPT-3 and Stable Diffusion are in use. The goal here is to sufficiently remove identification information (1) with minimal cost (2) while maintaining maximum performance (3). Therefore, the objective of many Unlearning tasks is to make a given pre-trained model equivalent to a state that have been trained from the beginning (Retrain-from-Scratch), using all training data excluding specific instances.

Gradient Surgery for One-shot Unlearning on Generative Model^[34]

First, we must define what influence on data means. If an influence of an weight parameter of certain data can be calculated, the process of removing it can satisfy the objective of Unlearning^[36-39]. For [36], which aims to eliminate data influence, in the case of a linear model assuming convex loss, the above objective can be approximated with a simple Newton’s update, which shows it is possible to derive Unlearning in that setting with Exact Hessian and Gradient calculations, both theoretically and experimentally. However, in the case of recent DNN models, and further, generative models, which were the target of this study, directly calculating the Exact Hessian is intractable, and the approximation bound is also not guaranteed.

In this study, we bypassed this and took an approach to minimize influence on generation output. As a result, it was theoretically confirmed that the Gradient Surgery^[33] technique proposed in the existing multi-task learning setting can update the given model weight parameter in a way that minimizes influence on the output^[34]. Additionally, in [35], it was theoretically confirmed that the approximation error of influence on the weight parameter that occurs when multiple data are removed (either sequential or batch) can be effectively minimized with a simple calculation to minimize gradient conflict. It has also been experimentally verified that it is effective in Unlearning when applied together in situations where the existing Newton Update method was not applied.

As a result of the experiment, it was confirmed that when applied together with the existing Newton Update-based method, a given instance can be effectively removed in a generative model^[34] and large batch removal situations^[35]. This shows the results of an Unlearning experiment using classes and features in a VAE-based generative model. The fine-grained evaluation technique of the generative model that was newly discussed at ICML 2023 will be reflected in additional experiment composition and evaluation of [34] in the future. Furthermore, in the process of empirically calculating the Loss Curvature used as the Hessian Approximate^[35], we would further advance the process to ensure a tighter bound of privacy by introducing the Differential Privacy-based weight exploration framework.

Image 1. Performance of Class/Feature Unlearning VAE on MNIST138 (left columns) and CelebA (right column)^[34]

 

2. Causal Inference

Causal Discovery aims to find Causal Relation from data. This is the basis of modern science and has established itself as an important task as it makes the reasoning of any phenomenon possible. Recently, Bayesian Inference-based algorithms are showing promise in the field of Causal Inference. At ICML 2023, a plethora of research results were announced, such as Bayesian Optimization-based^[1] and Reinforcement Learning-based^[2] algorithms, breaking away from the existing Statistical Learning method that predicted Single Point Estimate. The paragraph below briefly introduces the latest research trend that approaches Causal Discovery through Probabilistic Inference.

 

GFlowNets for Causal Discovery: an Overview^[3]

Causal Relation assumes that cycles cannot exist between nodes (as in causal relationships, cause and effect are always unidirectional). Therefore, Causal Discovery can be viewed as a problem of finding the optimal DAG from the Distribution of Directed Acyclic Graph. However, since DAG is defined in a discrete, combinatorial space, there is a challenge in that it grows exponentially depending on the number of nodes. Additionally, structural learning from data is NP-complete. Even if there is an Oracle that can efficiently find all DAG Space, in the current situation where only limited observations are secured, there is a limitation that the prediction results cannot be sufficiently trusted.

Therefore, previous studies use the Sampling method from Bayesian Posterior Distribution. Factorized Variational Approximation^[4] or expensive MCMC Simulation can also be used. Recently, a research group founded by Yoshua Bengio^[3] has demonstrated the potential of a Generative Flow Network (GFlowNet)-based method for causal discovery. This can be seen as one of the recent Supervised Learning approaches that learn Bayesian Posterior to be generalizable in an unseen, realistic dataset by giving it a Maximum-likelihood Objective in a Synthetic Dataset-Causal Graph Pair.

GFlowNet aims to perform step-by-step sampling of Compositional Objects such as DAG based on a given Energy Function (Reward Function). This framework is learned through the Self-Consistency Objective. The most important factor lies in its learning by replacing the approximation process of the intractable distribution expressed by this Energy Function with a Neural Network. Through this, the entire process can be optimized in data-driven fashion. The difference from the existing Supervised Learning method is that it requires relatively little Neural Network capacity and does not require pairs by sampling from the posterior for a given single dataset. In recent studies such as [5-7], Bayesian Learning of Causal Structure is expressed based on GFlowNet, and it is shown to be encouraging in terms of uncertainty and misspecification.

 

Image 2. The concept of GFlowNets^[40]

 

Also at ICML 2023, we were able to see a number of applied studies that improve the performance of various tasks by considering Causal Relation. In [9], the Causal Model was used to explain the Generative Factor of Motion Forecasting, and [10] explains what information is essential in the model transfer process for robust generalization in dataset shift situations from a Causal perspective through Minimax Analysis. In the following paragraph, we will touch on application research that solved Neural Algorithmic Reasoning considering Causal Relation by a research group of DeepMind.

 

Neural Algorithmic Reasoning with Causal Regulation^[11]

Neural Algorithmic Reasoning is a field of research that explores the reasoning capabilities of neural networks. Reasoning refers to logical thinking and decision-making skills, such as drawing conclusions and solving complex problems. Therefore, the Reasoning of a Neural Network means that the Neural Network can perform Algorithmic Tasks (executing algorithms, solving math problems, etc.) that require logical reasoning and decision-making. Specifically, the task is to solve a computational problem using well-defined rules and procedures, as well as to efficiently and effectively convert a given input into an output through sequential steps. For example, sorting a list of numbers or finding the shortest path between two points in a graph. In Neural Algorithmic Reasoning, the goal is for the Neural Network to effectively perform this task in Unseen Data and Out-of-distribution Examples of Training Distribution.

This study addresses OOD generalization issues when input size increases. Large-sized input increases the learning difficulty by combinatorially increasing the number of trajectories that require calculation within the algorithm. If such input is given only at test time, it will be expressed in more complex patterns and structures than the data encountered during training, which can limit accurate predictions. The authors confirmed that in Neural Algorithmic Reasoning, the following Causal Graph - the Outcome at a specific point depends only on the subset of the current Snapshot - is always established. For example, in the case of the DFS below, going to node 4 can only be done through node 2 in any case.

 

Image 3. An illustration of the key observation of our work, on the depth-first search (DFS) algorithm as implemented in CLRS-30^[11]

 

In other words, it can be seen that this Conditional Independence is met, and even if the rest of the Snapshot changes, the Conditional Probability does not change. Therefore, for these cases, the authors can reduce the overall amount of calculation by calculating only the corresponding trajectory instead of calculating all trajectories. As a result, we focused on the fact that effective learning and prediction can be made even when the input size increases. For Conditionally Invariant Prediction from X to Y, the authors expressed the above situation as a self-supervised objective and proposed a simple Contrastive Learning-based augmentation method. As a result of experiments on various algorithmic tasks of CLRS-30, we experimentally confirmed encouraging performance improvements, such as OOD micro-F1 performance improving by more than 90% compared to the baseline.

 

Image 4. Per-algorithm comparison of the Triplet-GMPNN baseline and our Hint-ReLIC^[11]

 

LG AI Research seeks to explain and analyze the prediction process of the Deep Neural Network through Causal Inference. Further, we are securing improved performance in various situations such as time-series forecasting by considering Causal Relation between events or variables. Likewise, our research group of LG AI Research are now putting efforts to build a more explanatory and improved prediction pipeline accounting for causality.

 

3. Geometric Deep Learning

Recently, in the field of Structural Probabilistic Modeling, algorithms based on diffusion^[12] are showing unprecedented activity in Molecular Downstream Tasks^[14]. The diffusion model series (EDM^[15] or GeoDiff^[16]) that satisfies the group symmetry presented in EGNN^[13], etc. shows notable performance in 3D Molecular Generation and is already establishing itself as a standard. Since then, starting with these papers, research on Graph Generative Models that operate based on diffusion is gaining momentum. Among the diffusion-based generation studies presented in various ways at ICML 2023^[19-20], we would like to introduce GeoLDM^[18], a stable diffusion study for molecules.

 

Geometric Latent Diffusion Models for 3D Molecule Generation^[18]

A Diffusion Model is a type of Generative Model that applies a diffusion process that iteratively weights perturbations to instances to learn the distribution of training data. Through the process of restoring this corruption to its original denoised state with the process of parametrizing with a Neural Network, the original distribution is effectively learned. The Diffusion Model has shown remarkable performance in non-geometric domain generation tasks such as image inpainting^[19-20]. Starting with last year's ICML 2022, we have also recorded remarkable achievements in the Geometric Domain, such as Molecule's Generation Task expressed as graph data^[14-16].

It is important that the 3D graph expression satisfies the Equivariance of the Operation Group. Specifically, the 3D Coordinate of Molecule must be Equivariant for the following three operations - Rotation, Reflection, and Translation - in Euclidean Space (SE(3)). EGNN^[13] proposes a GNN that satisfies this, SE(3)-Transformers^[17] proposes a Transformer, and GeoDiff^[16] proposes the Diffusion Model.

Until now, the Geometric Domain's Diffusion Process has operated in the Feature Domain (or in the case of Molecule, Atomic Space). However, these methods require the process of directly modeling the likelihood of a complex structure, which has limitations in terms of calculation amount and expressive power. Therefore, the authors designed and proposed the Diffusion Process to operate in Latent Space. The Diffusion Model in Latent Space can model smooth distribution, and can also gain an advantage in terms of computational volume by learning/optimizing in fewer dimensions. Additionally, like Text-guided Image Generation, Graph Generation also has the advantage of being controllable and can be expanded to create a Molecular Graph with certain characteristics.

However, obtaining Latent Representation from Geometric Domain is not an easy task. To capture Roto-translational Equivariance, which is important for 3D Molecular Geometries, in Latent Representation, the authors must replace existing Autoencoding with Geometric Autoencoding that satisfies Euclidean Groups SE(n). Specifically, they propose that the Latent Feature be composed of a Point-Structured Variable that has a 3-d equivariant latent feature and a k-d invariant Latent Feature for all nodes. This point-structured Latent Space has the advantage of being able to express output in an equivariant manner to input without additional parameterized operation.

GeoLDM showed the closest performance to GT in terms of several metrics in modeling Molecular Benchmark, such as QM9 and DRUG. In addition, it experimentally demonstrated the controllability of Molecular Generation by showing a generation result that can be controlled according to value changes in the interpolation task for Molecular characteristics such as polarizability.

Image 5. Molecules generated by conditional GEOLDM^[18]

Image 6. Results obtained by our own experiments. Other results are borrowed from recent studies^[18]

 

In addition, since Manifold's Geometrical Curvature-based algorithms have shown remarkable achievements in the field of Manifold Learning^[21-22], by reflecting the Non-Euclidean characteristics recently observed in various real-world cases, there is a noticeable movement to utilize the Riemannian Manifold for the purpose of securing latent hierarchical relationships. This movement was also seen at ICML 2023 as well. The paragraphs below cover Riemannian Manifold, especially Hyperbolic Geometry with Constant Negative Curvature, and the applied research applying Hyperbolic Geometry to Depth Completion Task.

 

Hyperbolic Geometry

Hyperbolic Geometry is one of the Non-euclidean Geometries and refers to a space with Constant Negative Curvature. Compared to Euclidean Space, Hyperbolic Space has the advantage of being able to express hierarchical data with low distortion. The Volume of Hyperbolic Space grows exponentially with the radius, and therefore, exponentially growing hierarchies or tree-like structures can be expressed with low distortion. In this respect, Hyperbolic Space can be used as a good tool to express complex data structures - for example, NLP, Social Networks, Image Analysis, etc.

 

Learning Affinity with Hyperbolic Representation for Spatial Propagation^[23]

This paper starts from the limitations of existing methods in Depth Completion Task. The Depth Completion Task is a task that recovers the actual 3D Dense Depth Map when given a Sparse Depth Map (e.g. Sparse Point Cloud) and an RGB image and derives the depth value for all pixels.

The existing SOTA methodology was Spatial Propagation Network (SPN)^[24]. SPN consists of two modules: (1) Affinity Branch, which calculates pair-wise affinity between pixels, and (2) Propagation Branch, which propagates the Initial Seed to the entire image based on affinity. SPN propagates the Initial Seed (User-defined or Initial Prediction) and derives an optimal sample group (Affinity Map) for performing downstream tasks to effectively perform Downstream Low-level Vision tasks.

The authors noted that the SPN has boundary ambiguity. In other words, the Affinity Map on the Object Boundary, where the Intensity changes smoothly on the Image, is not distinguishable between each object, and bleeding occurs where they invade each other's areas. The authors believe that the cause of this problem is Ill-defined Affinity, and focused on the essential problem that existing convolution operations can only cover grid data defined in Euclidean Space. Therefore, the authors propose to solve the issue of the Affinity module by reflecting the Geodesic Distance of Hyperbolic Space.

Image 7. Comparison results between baseline and HAM^[23]

 

This paper proposes the Hyperbolic Affinity Learning Module (HAM), a module that calculates spatial affinity. HAM goes through the process of embedding and aggregating Pixel Feature into a Hyperbolic Space with a certain Curvature to make Pixel Affinity have Hierarchical Property. Specifically, it consists of two components: (1) Beta-priority and (2) Geodesic Weight. Beta-priority calculates the Relative Importance of Pixels in Hyperbolic Space and encourages semantically close Features to have high affinity. Geodesic Weight calculates the Geodesic Distance of pixels in Hyperbolic Space and encourages nearby pixels in the Hyperbolic Space to have high affinity.

 

Image 8. Qualitative comparison on NYUv2 dataset for depth completion^[23]

 

The authors experimentally confirmed that the proposed HAM outperforms SPN defined in Euclidean Space in the Depth Completion Task and Semantic Segmentation Task. Based on the Tree-likeliness analysis of each Embedding in Hyperbolic Space and the Hyperbolicity of Pixel Feature, they have shown that Hyperbolic Geometry can serve as a good tool for solving Vision Task.

 

Image 9. Quantitative semantic segmentation results on PASCAL VOC 2012^[23]

 

Although We now reside in Euclidean space expressed in 3D coordinates, the data around us often can be expressed in non-euclidean space. Tabular data and time-series data, the main form of data in real-world application, have an inherent hierarchy between variables, while graph data, which expresses relations, does not follow the distance metric defined in Euclidean space. Expansion into non-Euclidean space will enable expressive representation learning of real-world data.

 

4. Large Scale Foundation Model

Large-scale models trained on massive amounts of data such as ChatGPT show amazing performance in many areas including text, images, and audio. ICML 2023 showed a continuous interest in these Large-Scale Foundation models. At this conference, we were able to confirm a number of foundation model studies^[25,29] that work well under various realistic constraints/situations. Among these, let us introduce some of eye-catching LLM studies, assuming the situations of Long-Tail Knowledge^[29] and Human Preference^[25], respectively.

“Large Language Models Struggle to Learn Long-Tail Knowledge^[29]” addresses the relationship between the ability to learn the Large Language Models (LLM) and Long-Tail Knowledge. Long-Tail Knowledge refers to rare and obscure facts that are not well known. The authors experimentally showed that the Language Model's ability to answer fact-based questions is proportional to the number of relevant documents encountered during the pretraining process. Additionally, they present simple experimental results to improve the problem that recent models are vulnerable to Long-tail Data-related QA. As an example, it tests whether Retrieval-Augmentation, which provides external knowledge, improves Rare Fact Learning. Retrieval Module is a module that combines with LLM and returns Relevant Textual Context^[26-28]. As a result of evaluating the 2-Shot Accuracy of GPT-Neo in the Oracle setting that grants Gold Paragraphs from LM and Wikipedia, we were able to confirm that Oracle Retrieval-Augmentation radically improves accuracy compared to Closed-book LLM and alleviates the dependency of Pretraining Data on Relevant Knowledge.

“Pretraining Language Models with Human Preference^[25]” proposes a pretraining method that guides the language model to generate text aligned to human preference. The existing Language Model generates content that violates human preferences (false information, offensive comments, personal identification information, low-quality code, etc.). Accordingly, the authors propose the following five Objective-based Pretraining with Feedback that can align with human preferences while maintaining existing capabilities (performance as Downstream Tasks): Conditional Training, Dataset Filtering, Unlikelihood Loss, Reward-Weighted Regression, and Advantage-Weighted Regression. Each Objective has different Alignment-Capability trade-offs for each task, but among these, they experimentally confirmed that Conditional Training shows Pareto Frontier performance in all tasks. In Conditional Training, LM aims to create text that maximizes the Reward Function that measures human preference. In both situations - one where no prompt is given and one where an adversarially chosen prompt is given - as the Language Model samples tokens from the Learned Distribution of tokens (which maximizes expected reward) conditioned on the Human Preference Score, it can now create text that is well aligned with human preferences. The authors experimentally confirmed that PHF can effectively alleviate the finetuning contrast phenomenon in tasks such as measuring toxicity.

Image 10. (Left) Language models struggle to capture the long-tail of information on the web^[29],
(Right) Toxicity score (lower is better) of LMs^[25]

 

What was interesting is that there has been an attempt to leverage the knowledge of the large-scaled foundation models had been trained on image/text data for downstream tasks of other modalities or completely different problems. These initiatives are very encouraging. This is because the real-world situations or general modalities (Tabular, Graph, CO, etc.) are quite limited for the data collection process compared to image or text benchmark data or low on the number of samples. If it is possible to leverage knowledge of modality which can easily secure data such as images or text, then it is also possible to achieve superior performance than the hand-designed SOTA model in various tasks. Of course, these attempts have been proactively reviewed in some recent studies. In [31], GPT-2 was applied to Image Classification, and [32] proposes a method of Tabular generation by fine-tuning LLM with a simple prompt that verbally describes Tabular data. However, research to date has had limitations, such as being limited to ad-hoc or specific modality in each situation. The following paragraph examines Cross-Modal Finetuning: Align then Refine^[30], a study on Cross-modal Finetuning regardless of the type of Modality.

 

Cross-Modal Fine-Tuning: Align then Refine^[30]

The ORCA method proposed in this study aims to extend a single Large-scale Pre-trained Model to Diverse Modality. Theoretically, it operates without limitation in both In-Modality and Out-Modality. ORCA follows the Align-then-Refine Workflow in the process of adapting to the Target Modality. In other words, when a target input is given, the Embedded Feature Distribution of this input is first aligned with the Pretraining Modality. Afterwards, the entire model (Embedding Network, Model Body, Prediction Head) is fine-tuned. In this study, the Model Body is assumed to be a Transformer structure. If the In-Modality is an image, it experimented with Swin Transformer, a ViT series, and for text, with RoBERTa.

An important factor is that the process of aligning the distribution of the source and target is not to align the semantics between two Modalities but to align the Structure. This process forces the Target Embedder to ensure that the Joint Probability between the feature and label of the Embedded Target Feature is similar to that of the source. By proceeding with this process prior to overall fine-tuning, after the fine-tuning process is complete, both the source and the target of the Feature-Label Joint Probability that the Model Body meets become similar, and learning can occur smoothly during the fine-tuning process. By utilizing Optimal Transport Dataset Distance as the Distance Metric used in the alignment process, the authors expressed Joint Probability Alignment as the sum of Feature Space Alignment and Label Space Alignment (Discrete Label Space is expressed as Distribution of In-class Feature and expanded to Continuous). However, the authors claim that any Distance Metric can be used theoretically and report experimental results of various Distance Metrics.

 

Image 11. ORCA’s three-stage fine-tuning workflow^[30]

 

The authors experimented with 10 different tasks (Image, Fluid, Protein, RNAseq-related, etc.) of NAS Benchmark. When looking at the performance of SOTA (which was hand-designed to be task-specific for each task), the performance of SOTA among the Auto-ML series, and the performance of General Purpose Architecture as a baseline, ORCA was confirmed to have outperformed. In addition, for target tasks that are very far from source tasks such as PDE Solving, it was confirmed to show similar or better performance compared to the FNO series or the PINN series that are specifically designed for the physical system.

 

Image 12. Prediction errors on 10 diverse tasks^[30]

Image 13. (Left) Normalized Root Mean Squared Errors (nRMSEs) for ORCA vs. baselines on 8 PDEBench tasks with varying dimensions (1D/2D), (Right) ORCA is trained on resolution 256 and directly evaluated on resolution 512^[30]

 

Real-world situations have many constraints, and data scarcity is also a large part of them. By utilizing the knowledge of data-rich modality to predict the data-scarce modality of the general real world, the above study experimentally showed that it can break the upper bound of performance of existing hand-crafted models. Beginning with this study, we eagerly await a paradigm shift and performance leap in modalities that have relatively not benefited from the scalability of recent AI research.

ICML 2023 is an ML conference that is not limited to a specific field, and papers with a wider variety of scopes were presented at the event. Among them, LG AI Research looked into the research presented at this conference, as well as major papers in the fields of Causal Inference, Geometric Deep Learning, and Large Scale Foundation Model, which are both receiving particular attention in the academic world and currently receiving our focus.

LG AI Research has been working on interesting research topics and reported advanced results at various venues of AI research. Whileso, we have had an active communication with students and those in the field by participating in various social meetups, including ML in Korea. Throughout numerous presentations in the field, we realized that areas in which machine learning is applied are having expanded, while at the same time, the technical difficulty is increasing. LG AI Research's Data Intelligence Lab will continue to develop applied research technologies to redefine and effectively solve real-world problems.

 

▶ Gradient Surgery for One-shot Unlearning on Generative Model (Link)