[IJCAI 2022] Review of IJCAI 2022 and Deep Graph Learning Research Trends

The International Joint Conference on Artificial Intelligence (IJCAI) was held in Vienna, Austria from July 23 to July 29, 2022. This conference on artificial intelligence boasts a long history, having been held for the first time in 1969. Since its inception, the conference has been held every two years, but since 2016 it has been held annually. The IJCAI, which is celebrating its 31st anniversary this year, has maintained a very high level, with an acceptance rate of less than 20%, while still covering a wide range of AI research topics. 

At the conference, AI researchers from around the world shared academic and cultural exchanges through research-related sessions such as invited talks, workshops, tutorials, presentations, and poster sessions, as well as through events such as a banquet, reception, and opening ceremony.

Figure 1. IJCAI 2022 Location

Although IJCAI 2022 was held in a hybrid format, the event was filled with the enthusiasm of researchers that bridged the gap between the online and offline sessions. Several research outcomes, including deep learning methodology and AI framework research, application/system research, and a robotic demonstration, were introduced. In this blog, I will introduce recent trends in AI research from IJCAI 2022.

Research Trends

IJCAI is a conference that places great emphasis on deep learning frameworks and methodologies that solve real-world problems. Through the main track research and workshops presented at IJCAI 2022, the associated research trends were identified. 

Regarding the main track papers, there were a lot of studies on reinforcement learning in general. A large audience was present for the numerous presentations on papers dealing with agent/multi-agent systems, planning, and constraint satisfaction. In addition, a number of studies on robotics and deep reinforcement learning have been presented, giving an abundance of relevant information. 

In addition, the number of federated learning studies has increased in comparison to previous years. Research related to natural language processing (NLP), such as knowledge base/graph-based reasoning, particularly stood out. There were numerous machine learning application studies, and it was intriguing to see that many of them used graphs to solve problems. Reinforcement learning and federated learning were particularly notable research fields, but there were many interesting topics among the consistently researched representation learning papers, resulting in lively conference discussions. 
    
Moreover, I was able to identify research trends using keywords from a variety of workshops held during the IJCAI 2022. 10 out of 30 workshops at the conference were devoted to reinforcement learning, among which topics such as agent systems and safe RL that dealt directly with reinforcement learning were highlighted. There were also workshops in the field of applications employing agents or reinforcement learning. 

Roughly six workshops related to reasoning were held. In addition to reasoning in NLP, workshops on diverse topics such as temporal reasoning and qualitative reasoning were held, meaning that the reasoning capabilities of the deep learning model are becoming increasingly emphasized. In addition, three workshops on time series or spatio-temporal data were held. There are an increasing number of studies emphasizing the significance of temporal or dynamic information learning, which has been a topic of significant interest in the field for some time. Additionally, two workshops on scarce data were held. 

In particular, I discovered that the term “Big Data,” which was formerly used frequently in the workshops, is now rarely mentioned at IJCAI. This means that excluding methods that can be utilized to efficiently process large amounts of data, technology for finding big insights from insufficient data is becoming more important. 

Invited Talk

The benefit of an offline conference, as opposed to an online event, is that participants can meet distinguished researchers and discuss research with authors in the field. In addition, during the Invited Talk session, renowned researchers were able to deliver talks on major research topics and the future of each research field. During the Invited Talk session at IJCAI 2022, there were a number of interesting lectures covering a wide range of topics, including deep learning methodology, AI in medicine/pharmacy, AI to solve negative social changes brought about by AI, and AI-assisted programming. Professor Judea Pearl of UCLA and Dr. Sumit Gulwani of Microsoft Research both spoke at the Invited Talk, which was attended by a large number of people. 

Figure 2: Lecture by Dr. Sumit Gulwani from Microsoft Research 
As part of a lecture on AI-assisted programming, code generation using the transformer model and reinforcement learning were discussed. A large audience was present as code generation is a practically applicable field. 

Professor Judea Pearl's lecture was one of the many Invited Talk sessions that I attended as it would be beneficial to the forecasting research that the Applied AI Research Lab focuses on. Professor Pearl's online lecture focused on the necessity and application of causal inference in machine learning. 

As the number of related studies increases, numerous researchers note that the “black-box” model is not interpreted in machine learning and that the model has some flaws, and causal inference is being utilized to improve knowledge extraction from data. According to Professor Pearl, the knowledge that is sought to be obtained through training the model is the same as deep understanding. He defines deep understanding as “knowledge that feels controllable or such a state,” and suggests that deep understanding can solve the prediction problem, the interpretation problem, and the counterfactual problem. Causal inference is the means to achieve such deep understanding. 

Causal inference encodes causal information visually without using the black-box model. As such, it is also a method for predicting the effects of action and policy. Causal inference can also be used to compute counterfactuals and identify causal relationships. This can be utilized to solve the explanation problem. In addition, causal inference can identify both direct and indirect effects, which can help in solving discrimination and fairness problems. Consequently, as causal inference is advantageous for causal prediction or calculation, it can facilitate the integration or fusion of data obtained from diverse sources or conditions. Combining data from different domains or with information about different conditions is not an easy task; however, causal inference can be used to estimate effect data in a new domain or condition accurately, resulting in a high level of data integration. This can have an impact on the issue of reducing selection bias, etc.

Figure 3: Online Invited Talk by Professor Judea Pearl 

[Graph Representation Learning: From Simple Message Passing To Utilization of Topology and Geometry]

Graph representation learning, which had been studied in the direction of updating by aggregating the messages (features) of neighboring nodes using the message passing scheme. Furthermore, recent research on graph representation learning, including IJCAI 2022, focuses on learning graph structures utilizing the characteristics of topology or geometry. Although previous research was based on a fixed static graph and a homogeneous graph in which hierarchical properties are not reflected in nodes/edges, at this year's IJCAI 2022, a dynamic graph that changes over time and a heterogeneous graph study in which hierarchical properties are reflected in nodes/edges stood out.

The Applied AI Research Lab of LG AI Research is researching session-based and graph-based recommendation systems. Universal user-item data can be viewed as a heterogeneous graph with two types of nodes —User and Item — whereas session-based recommendation data expressed as a graph becomes a dynamic/temporal graph.

Figure 4: Example of Expressing Recommendation Data as a Graph^[1] 
While being a heterogeneous graph with User and Item (Food) nodes, it is a dynamic graph containing the session information of t=1,2,…,5. Recommendations on graph data are identical to link prediction tasks. In this example, the location of t₆ between each item node and the target user is predicted.

The representation embedding methodology of nodes comprising a heterogeneous graph requires a meta-path designed to discover the relationship between two nodes and a meta-graph, which is a generalized version of this. However, meta-path and meta-graph have the drawback of being empirically defined. To overcome these limitations, a study was presented at this year’s IJCAI, proposing a method for learning a heterogeneous graph using a simplicial complex without meta-path or meta-graph^[2]. 

Simplex is a generalization of the triangle and tetrahedron concepts to arbitrary k dimensions. Also known as a polyhedron, it is capable of expressing higher-order structures such as triangles and regular tetrahedra. A space composed of these simplexes is referred to as a simplicial complex. It can be thought of as a space where a higher-order structure is gathered and formed. In other words, to view the heterogeneous graph as a simplex, the heterogeneous information must be transformed into a homogeneous simplex. It is difficult to account for heterogeneity in which information has distinct hierarchies, such as multiple node types and edge types. By converting this to k-simplex, information can be expressed by restricting it to the same k-dimensional space, and learning can occur without considering heterogeneity. 

The following describes the specific application of simplex to a heterogeneous graph. Below is an example of a movie data graph containing the categories of Actor, Movie, and Director. Based on the Movie type Node, 0-Simplex, which is the vertex, is composed of m1, m2, and m3. In the 1-hop neighboring node between m1 and m2, there are a1, d1, and d2; between m1 and m3, there are a3 and d2; and between m2 and m3, there are a2 and d2. Edges can also be expressed as 1-Simplex nodes that connect two Movie Node pairs. In other words, 1-Simplex Edge can be lifted by aggregating the features of each different type of node. 2-Simplex representing a triangle is a neighboring node of another type that is connected to the three Movie Nodes — m1, m2, and m3. Because node d2 is connected to nodes m1, m2, and m3, the triangular 2-Simplex reflects the properties of node d2. When there are more than four Movie Nodes, the 3-Simplex, or a tetrahedron, can be determined by the presence or absence of neighboring nodes of other types connected to the four Movie Nodes.

Figure 5: Example of Applying Simplex to Heterogeneous Graph Data^[2] 
This figure is an overview of finding the embedding of a node with a node type of m for a 1-hop neighbor node where there are three node types. A more detailed explanation can be found in the preceding paragraph.

By collecting the K-Simplex found in this manner, a simplicial adjacency matrix for each k (k=0,1,...,K-1) can be constructed. Using the graph attention mechanism to learn the simplicial adjacency matrix from 0 to K-1, the embedding of the node whose target type corresponds to 0-simplex is obtained. By obtaining the node embedding in this manner, it is possible to find the embedding of each node by considering the node type information of various layers without defining a meta-path or meta-graph. This was intriguing and significant because it eliminated the empirical meta-path selection process and omitted higher-order information by not utilizing meta-path or meta-graph. 

At IJCAI 2022, I presented research^[3] on learning large-scale dynamic graphs through incremental learning. When session-based recommendation data is expressed as a graph, the resulting representation is a dynamic (temporal) graph. A dynamic graph is one whose structure changes over time, for example by adding or removing nodes and edges. Graph data in the real world frequently have a dynamic structure as opposed to a static one. In other words, the graph's node embedding should be updated to reflect the altered graph structure. In contrast, the resultant resource is not only substantial, but also increasing. 

Figure 6: Structure with Blue Nodes and Thick Edges Added^[3] 
According to the added structure, the receptive field of purple node i has changed significantly since the previous time step. This can be viewed as a characteristic of the dynamic graph, but the current incremental learning method cannot account for this information about the shifting receptive field. This is due to the fact that incremental learning in the image domain assumes independence between data instances (e.g., image instances, nodes).

Existing graph nodes are affected by the newly modified structure. Since there is a dependency called edge, the propagation matrix is used to determine the level of impact. The embedding is updated by incorporating the altered structure into the embedding of the node that is significantly impacted by the new structure. In contrast, a propagation matrix can be used to identify existing nodes that are completely unaffected. The expected value of the change in the loss for the modified graph structure is obtained, and the node is sampled with the largest expected value of change in loss among the nodes unaffected by the modified graph structure. It is desirable that the sampled node does not change, but it can be interpreted as a node with a high risk of change due to the effect of parameter update based on the newly modified graph structure. In addition, I have proposed a method to prevent Catastrophic Forgetting. This node's data is used to penalize the model with knowledge distillation. A framework that addresses the problems of changing receptive fields and catastrophic forgetting problems enables more efficient learning of dynamic graphs. However, since incremental learning is employed, temporal information is not considered during embedding learning. In the future, I'd like to discover methods by which temporal patterns can be recognized. 

Figure 7: Presentation by Seoyoon Kim, Applied AI Research Lab of LG AI Research

Conclusion

Due to the pandemic, many conferences were held virtually for a long time. IJCAI 2022 that was held offline allowed me to absorb a lot of information about AI research. In particular, I was able to interact face-to-face and for an extended period with researchers from various countries who conduct research in a wide range of fields and have diverse backgrounds. The Q&A sessions, poster sessions, etc., were more informative, and there was a great deal of information I was able to obtain as a researcher. 

Through this conference, I was also able to gather that graph research will expand beyond its current framework. Until today, numerous studies on spatial GNN and graph attention have been published, with the majority of research concentrating on the node classification problem. It is anticipated that numerous studies will be conducted in the future to solve various graph-related problems. In addition, the current GNN has relied heavily on feature information, and it is predicted that the study will gradually shift its emphasis to the graph structure.