[NeurIPS 2024 Series] A Study on Adaptive Information Routing Technology for Multimodal Time Series Forecasting

Neural Information Processing Systems (NeurIPS), the world’s leading conference on artificial intelligence, was held in Vancouver, Canada, from December 10 to 15, 2024. This year’s NeurIPS featured more time series and time series forecasting-related research than any previous conference. At this year’s NeurIPS, research didn’t just focus on developing forecasting models that perform well on benchmark data. Instead, the results of time series forecasting were presented in various ways, including qualitative forecasting using LLMs^[1], multimodal forecasting^[2][3], forecasting using generative models^[4][5], analytical studies on forecasting models^[6][7], and time series benchmarks^[3].

With the recent growing interest in large models, there is a growing need to utilize large models in time series forecasting. To kickstart this push, the Time Series in the Age of Large Models (TSALM) Workshop was held at NeurIPS this year, showcasing various research and in-depth discussions on the use of large-scale models in time series forecasting. Topics included the use of large time series models, such as time series foundation models and pre-trained models that use large language models (LLMs) in time series forecasting. LG AI Research’s Data Intelligence (DI) Lab presented a paper on multimodal time series forecasting at the TSALM workshop entitled “Adaptive Information Routing for Multimodal Time Series Forecasting^[8].” It also shared LG AI Research’s ongoing research and obstacles in time series forecasting.

LG AI Research’s Adaptive Information Routing for Multimodal Time Series Forecasting^[8]

Time series forecasting is an important AI technology that can aid in the decision-making process for businesses by providing an outlook into the future. Time series forecasting is generally done by learning the patterns in a time series and forecasting the future based on past values. However, in real-world decision-making problems, such as exchange rates, stock prices, and raw materials prices, there are many factors other than the past values of the time series. Therefore, to improve accuracy, the forecasting model must be able to consider a variety of factors. Additionally, a large portion of real-world information is not in the form of a structured time series, but in the form of unstructured data such as text. LG AI Research has conducted research on multimodal time series forecasting technology to effectively utilize various types of data to improve accuracy for various real-world forecasting problems.

Multimodal time series forecasting generally aims to utilize text information along with time series to build a forecast. The currently proposed multimodal time series forecasting techniques can be broadly categorized into two main groups. The first is the study of utilizing the language model itself as a forecasting model. This equates to the methodology of finetuning a Pretrained Language Model (PLM) into a multimodal forecasting model^[9][10][11]. This methodology aims to utilize the language skills acquired during the pre-training process of PLMs and improves performance by utilizing simple text information such as domain descriptions for forecasting. There are also methodologies that utilize text information through a separate embedding model^[3][8]. These methodologies focus on developing forecasting models that can better utilize embedding information that compresses text data, and have the advantage of efficiently using large amounts of text compared to the first methodology.

About The Research 

“Adaptive Information Routing for Multimodal Time Series Forecasting,” published by LG AI Research, is a study on multimodal forecasting technology that reflects the different characteristics of time series data and text data. In general, time series data contains more local, specific, and structured information compared to text data. Text data is full of global, ambiguous, and unstructured information compared to time series data. Therefore, a method that only uses time series and text data together for model input purposes, all without considering the different characteristics of both data, has a fundamental limitation in the ability to utilize the data. In this study, we propose a new multimodal time series forecasting technique “Adaptive Information Routing (AIR)” that utilizes text data as a controller for time series models by considering the different characteristics of time series and text. 

Time series forecasting models typically operate by mixing information from different points in time, as well as between variables in time series data, to extract features and obtain forecasts based on those features. Therefore, the behavior of time series forecasting models can be controlled by which information is mixed and at what intensity. The Adaptive Information Routing (AIR) technique proposed in this study is a new multimodal time series forecasting technique that controls the behavior of time series forecasting models by regulating the information pathways through which time series information flows and is mixed within the time series model with text information. 
 
In this study, we focus on the most basic component of a neural network, the Fully-Connected (FC) Layer. The FC Layer works by multiplying a given input by a learned weight matrix, which can be divided into two FC Layers with latents to perform the same operation. Each latent between the two FC Layers is connected to every node of the input and output, so it corresponds to a specific connection between the input and output or a specific information pathway. Therefore, we can adjust the information pathway within the FC Layer by dividing the FC Layer into two and applying weights to the latents between the two FC Layers. Based on this structure, the proposed AIR uses an information routing module that obtains the weights of latents from text embeddings to adjust the behavior of time series forecasting models based on text information.

Image 1. Structure of the Information Routing Module^[8]

The image above shows the structure of the information routing model proposed in our study. It embeds text data into a fixed vector through a pre-trained embedding model and integrates multiple text data information into a single vector using an attention layer and a learnable integration token. Finally, a weight generator in the form of MLP is used to generate weights to be applied to latents.

Image 2. Example of Applying Adaptive Information Routing Techniques to the Base Model (TSMixer)^[8]

In our paper, we use TSMixer^[12], which is AIR’s base time series forecasting model and a time series forecasting model consisting of only FC Layers. Each mixer block in TSMixer consists of two FC Layers for temporal mixing and feature mixing, respectively. Each FC Layer is divided into two FC Layers and weights generated by an information routing model are applied to the latents between the divided FC Layers. The information routing model generates and utilizes different weights for every block and every FC Layer. 

To verify the performance of our proposed AIR technique, we propose and use two types of data in this paper. The first is synthetic data to verify that the AIR method can indeed adjust the mix of time series information based on text information. First, we generated target data by randomly generating 30 time series data, then randomly selecting two of the 30 and averaging them. During this process, we changed the configuration of the target at certain intervals and generated text data with information about it. From this data, a model that cannot utilize the text data will not be able to make good forecasts. However, a model that can adjust the behavior of the forecasting model based on the text data will be able to make near-perfect forecasts. The second data set is stock price data, which consists of a time series of closing prices of IT-related stocks in the US stock market and synthetic text data generated by ChatGPT. Stock price data is used to verify whether the proposed AIR technique can actually help improve forecasting performance by forecasting the stock prices of M7 stocks, which are the most representative stocks in the US stock market.

Image 3. Visualization of Synthetic Data Forecasting Results from TSMixer and TSMixer+_AIR^[8]

Image 4. Comparison Table of Synthetic Data Forecasting Errors^[8]

Image 5. Comparison Table of Stock Price Data Forecasting Errors^[8]

In experiments with synthetic data, the base model TSMixer, which utilizes only time series data, failed to make proper forecasts. On the other hand, the AIR-enabled TSMixer, which utilizes text information as a controller, made near-perfect forecasts with very small errors. In experiments with stock price data, the AIR-enabled model successfully improved performance compared to the base model, with an average error reduction of 16.93%, a high of 23.68%, and a low of 12.71%.

Future Plans

We discussed various attempts that have been made in the field of multimodal time series forecasting with the aim of using text information for time series forecasting. We also introduced a new multimodal time series forecasting technique proposed by LG AI Research. 

Accurate forecasting of the future is an important technology that can contribute to the decision-making process in the real world. To advance this technology, it is important to achieve high forecasting performance and secure explanatory power and reliability of the forecast results. At LG AI Research's DI Lab, our research goal is to make accurate and explainable forecasts by effectively utilizing various types of data. Currently, we are conducting research on various forecasting technologies in this regard. Multimodal time series forecasting is an important technology that can contribute to both aspects of improving forecasting performance and reliability. The study presented at the NeurIPS TSALM workshop is our first step into the field of multimodal forecasting and we plan to continue to research more accurate and reliable forecasting techniques based on produced results.

NeurIPS 2024 Series

EP. 1 [NeurIPS 2024 Series] Introduction to Mutual Information Estimation Benchmark Research
EP. 2 [NeurIPS 2024 Series] B-splines-Based Input Value Normalization Method for Tabular Data
EP. 3 [NeurIPS 2024 Series] Attention-Based Data Augmentation Techniques to Enhance Performance of Tabular Data Prediction