[AAAI-25] Study on Denoising Distillation Model Using 3D Coordinates

Recently, AI research has been flourishing in various fields, especially in chemistry, biology, and pharmaceuticals. Every day, the news is flooded with reports on AI-driven breakthroughs, from predicting structures to designing molecules that meet specific properties. In many cases, this convergence research is utilized to develop new materials or reagents, enabling us to perform relatively accurate and fast predictions that would be difficult to predict or simulate using traditional methods. This accelerates the pace of research, and the results of AI models can also provide new inspiration.

LG AI Research has been working on various research projects on material development in collaboration with various LG affiliates that are responsible for the development and production of numerous products and materials. As one of the achievements, a joint study with LG Household & Health Care has successfully developed a cosmetic efficacy material with improved solubility and safety. LG AI Research’s AI model specialized in new material discovery, EXAONE Discovery, played a crucial role in this research. Traditionally, selecting candidate materials based on researchers’ experience and academic papers takes nearly two years, but EXAONE Discovery completed this process in just one day.

The research that served as the foundation for this cosmetic efficacy material development, “3D Denoisers are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation^[1],” was selected as an Oral paper at AAAI-25 (The Association for the Advancement of Artificial Intelligence), garnering significant attention. We are now unveiling the research process and its outcomes.

AI Meets Basic Science

When conducting research projects on material development, we realize that despite being different projects, they share common overarching goals.

 

1. Can we ensure predictive power with new distributions of data, i.e., out-of-distribution (OOD)?

2. Can we develop models for new properties with less data?

 

In fact, these are two fundamental questions that are always considered when developing AI models, even when science and AI are not converging. The LLM field is a field of study that overcomes these problems very effectively. LLM research utilizes large amounts of data to create a large model, or foundation model, that can be used to execute a given command. Due to the large amount of data, the likelihood of encountering OOD is lower. Also, since there is a pre-trained model using a large amount of data, it is possible to implement a model that performs new functions with relatively little data by fine-tuning it.

The remarkable advances made in the field of LLM have much to learn from research in chemistry and biology, particularly in terms of the overall strategy behind LLM research rather than the specifics. However, applying AI to chemistry and biology is not as straightforward as it might seem. Fundamental research philosophy is often at odds with AI, as these fields of science have been developed by humans for millennia and are driven by the desire to explain the laws of nature using extremely simple formulas. 

Typically, AI models are fundamentally based on an “inductive” approach. It is about getting a large amount of data, analyzing it, and finding a way to explain it. Automating this process is what we call AI model training. However, research in the basic sciences often does not follow an inductive approach, especially in fields such as chemistry and physics. Research begins by finding or deriving equations that can explain natural phenomena, and “deductive” methods of interpreting these equations to discover new properties and features prevail. Therefore, the convergence of these two fields is in a different realm than Vision or LLM and has not yet had the same impact as the previous two fields.

The convergence with the basic sciences is about blending the best of both AI research methodologies and scientific methodologies. The inductive methodology used by AI reads regularities in large amounts of data, making it easier to expand to a large number of functions, while traditional scientific research results can be utilized to ensure model accuracy. Based on this philosophy, LG AI Research's Materials Intelligence (MI) Lab published “3D Denoisers are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation^[1]”,  a study on denoising distillation using three-dimensional coordinates, which was orally accepted at AAAI-25.

LG AI Research’s 3D Denoisers are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation^[1]

The recent development of AI-assisted physical property prediction models is reminiscent of the Warring States period. As much as it utilizes molecules, the data it uses is based on such molecules, but even the form of input is not unified. Broadly speaking, there are two types of models: those that start with two-dimensional graphs and those that utilize three-dimensional information. In general, models that utilize three-dimensional information are superior in most property prediction applications due to the abundance of the information contained in the data. However, three-dimensional information of molecules is physically impossible to obtain at a level of accuracy and number that can effectively train an AI model, so in many cases, two-dimensional graphs are preferred.

The two-dimensional graph form utilizes the atomic structure of a molecule by embedding it in the form of a graph, which has the advantage that it can be created as long as the composition of the molecule is known. However, there is a problem that molecular structure information that cannot be expressed in two dimensions is excluded, and there are different ways to convert this two-dimensional graph into a latent vector.

These characteristics have led to two main directions of research in this field. The first is research that utilizes large amounts of publicly available three-dimensional benchmark data, such as QM9 or PCQM4Mv2, to create large models. The second is research that uses relatively small amounts of two-dimensional data to predict specific properties with specific downstream decisions. However, as mentioned earlier, both methods have clear advantages, and this study aims to combine the best of both worlds.

Typically, when working with affiliates, both the properties that need to be predicted and the structure of the molecule are rarely publicly available. Furthermore, it is often not easy to find similar shapes or even similar properties. Therefore, even if you want to fine-tune a model based on one developed in another project, it is sometimes fundamentally impossible. To address these challenges, our primary goal was to develop a model capable of generating latent that captures the general properties of molecules.

To achieve this, we utilized publicly available datasets that provide three-dimensional coordinates. The model structure is inspired by the widely known teacher-student framework. The strategy involves first training a teacher model using a large number of publicly available three-dimensional datasets and then training a student model to replicate the teacher model. At this stage, the student model is trained using two-dimensional graphs, allowing it to extract two-dimensional features while effectively incorporating three-dimensional information. Another advantage is that predictions can be made instantly, as long as the atomic composition of the molecule required for the task is known, enhancing its usefulness. The student model trained in this way can later be fine-tuned on the desired small-scale property data to serve as the final prediction model.

Image 1. Comparison between D&D and existing molecular pretraining frameworks^[1]

You can already see that the teacher and student models are different from the traditional distillation model in that they utilize different input types. However, if we take a closer look at the pre-training algorithm that trains the teacher model, the differences become even more pronounced. The algorithm that trains the teacher model aims to denoise the three-dimensional coordinates. To put it simply, it is given the three-dimensional coordinates of the constituents of a given molecule, it directly noises this coordinate information and sets the goal of having the model predict this noise directly. This sequence of processes may seem very simple, but it is scientifically quite significant.

Image 2. Illustration of our D&D framework. First we pretrain a 3D conformer encoding module by denoising perturbed conformers^[1]

It is usually assumed that the distribution of the constituents of a molecule follows the Maxwell-Boltzmann distribution. This Maxwell-Boltzmann distribution is similar to the form of Gaussian noise that is commonly considered because its shape is proportional to the square of the negative exponential. In other words, if the displacements of the constituent elements in the reference coordinates are assumed to be Gaussian, noise is generated. By developing a model to predict this noise, the loss can be interpreted as a physical quantity representing the forces acting between the constituent elements. In this case, the Gaussian noise is a simplified representation of the forces between the elements as simple harmonic motors, which is a well-established methodology used in computational chemistry to calculate the initial coordinates of the constituent elements. By including this algorithm in learning, we were able to successfully integrate knowledge from basic science into the AI model, and it has been validated in various experiments.

Image 3. Average performance and standard deviations for OGB and Curated datasets:
12 tasks on the left are from OGB, and 4 tasks on the right are from manually curated data.
Best results are in bold.^[1]

The value of this research is that it is an algorithm that maximizes synergy by effectively mixing the methodology of multi-modal, distillation, and foundation model learning using large data with the knowledge of force-field optimization in basic science, which is currently being studied in the mainstream. In addition, our research team is expanding and conducting research in several directions, including a transfer-learning model using differential derivatives based on multi-faceted expertise, and graph-pooling research that automatically sets the number of groups for classification. In particular, a lot of research is being planned and carried out, such as research on embedding quantum information beyond the classical method of force fields and extending the algorithm by applying the theory of differential geometry to teacher-student distillation learning.

The convergence of basic science and AI is expected to open new horizons in the future as it can directly contribute to the efficiency of research and manufacturing. LG AI Research's MI Lab, which conducted this study, is an organization comprised of researchers with expertise in various basic science fields, including AI majors, and is leveraging these capabilities to effectively converge scientific knowledge with traditional AI methodologies. We will continue to play a leading and pioneering role in the convergence of AI and science and take the lead in creating various research achievements as well as cases that can be applied to real-world business.