[AAAI-25] Study on Efficient Time Series Forecasting Model Learning Strategies That Alleviate Variable Redundancy

The AAAI Conference on Artificial Intelligence (AAAI), one of the top academic conferences in the field of artificial intelligence (AI), was held from February 25 to March 4 at the Pennsylvania Convention Center in Philadelphia, USA. More than 50 time series-related papers were presented at AAAI-25 this year, and the topics of the papers were very diverse. Some included leveraging Large Language Models (LLMs) to improve forecasting performance by exploiting diverse domain knowledge in the form of text^{[2, 3]}, suggesting that traditional artificial neural networks such as MLPs and LSTMs still have a lot of potential in forecasting^{[4, 5]}, and improving the efficiency of time series forecasting models^{[1, 6]}. 

Time series data is a data format that is actively processed in various industries such as finance, manufacturing, and healthcare, and LG AI Research is conducting decision support research for experts in various fields through forecasting of time series data.

LG AI Research's Data Intelligence (DI) Lab presented the paper “VarDrop:  Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting^[1]” at the AAAI-25 Main Track, presenting a new approach for efficient training of multivariate time series forecasting models. This article will introduce the motivation and methodology of this research, and review other notable related research at AAAI-25 and cover noteworthy research in the field of time series forecasting presented at AAAI-25.

 

LG AI Research’s “VarDrop: Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting^[1]“

Many recent transformer models proposed in the field of time series forecasting use the strategy of variate tokenization. Variate tokenization processes each variable in multivariate time series data as a separate token, with the meaning and number of variables determined by the scale of the service processing the time series data. For example, a household consuming electricity or a sensor in a process might be defined as a variable in multivariate time series data. While variate tokenization has the advantage of allowing relevant variables to be selectively referenced by attention operations, it suffers from a quadratic increase in computational cost with the number of variables. Unfortunately, variable tokenization strategies are permutation-invariant, which makes them inapplicable to existing efficient learning strategies that leverage tokens' location information, such as Big Bird^[7] and Pyraformer^[8]. For this reason, applying variate tokenization strategies to large-scale services requires a significant computational cost for model training.

But are all variables essential for learning? If there are variables in the data that have redundant patterns, you may be able to effectively reduce computational costs by excluding some of them from learning. When we analyzed a representative dataset used in real-world time series forecasting problems, we found that most of the variables had similar cyclical patterns to each other. This high variate redundancy means that the model is utilizing unnecessarily redundant information during the learning process. By excluding them, you can significantly reduce the number of variables you need to consider while minimizing information loss during learning.

Image 1. Covariate Shift in Traffic Dataset^[1]

To define redundant variables, you can first group variables with similar patterns and extract representative variables from each group. However, due to distribution shifts, a characteristic of time series data, the similarity between variables changes over time (Image 1). This covariate shift means that groups of variables with similar patterns need to be defined differently for each batch. If traditional clustering methodologies are utilized to define groups of similar variables, excessive computational cost for grouping is incurred during the learning process, which greatly hinders learning efficiency. Therefore, a more efficient grouping strategy than existing clustering algorithms is needed.

VarDrop, presented at AAAI-25, is an algorithm I developed prior to joining the company. It adaptively groups redundant variables in time series data on a per-batch basis and selectively leverages representative variables for training by sampling them (Image 2). The VarDrop algorithm is divided into two main steps. First, the k-Dominant Frequency Hashing (k-DFH) applies a Fourier transform to the time series data and utilizes the top k frequencies with large amplitude values as a hash value. Unlike traditional clustering algorithms, k-DFH does not compute the similarity between variables pair-wise, making it a cost-effective method. Then, representative variables are sampled from variables with the same hash value using stratified sampling. Finally, a small number of selected variables are tokenized and used to train the transformer. This token reduction method is possible due to the permutation-invariant nature of variable tokenization.

Image 2. Overall Procedure of VarDrop^[1]

VarDrop has two hyperparameters: k and gs (Group Size). First, k, the hyperparameter of k-DFH, determines the resolution of the groups. Larger values of k increase the similarity within a group, but the number of groups can be large, reducing the cost savings effect. Conversely, a smaller value of k results in lower similarity within the group, but fewer groups, which benefits from utilizing fewer tokens. Another hyperparameter, gs, determines the number of variables sampled within a group. A large gs value can minimize information loss by increasing the number of variables leveraged in each group, but the number of tokens leveraged will be larger, resulting in less cost savings. There is a trade-off between preserving performance and reducing cost, and VarDrop's hyperparameters can be chosen appropriately based on the needs of the service. 

The analysis of VarDrop's two hyper-parameters shows that there are cases where higher forecasting performance can be achieved by reducing the number of tokens. As shown in Image 3, the forecasting error and token reduction ratio for k=4 and gs=1 outperforms that for k=3 and gs=5. This indicates that a higher token reduction ratio does not necessarily lead to lower forecasting performance, and proves that VarDrop is a more effective algorithm than simply applying random sampling.

Image 3. Hyperparameter Sensitivity Analysis^[1]

VarDrop is an algorithm that adaptively groups redundant variables in time series data from batch to batch and samples representative variables for learning, which can contribute to significant savings in computation time and memory usage. LG AI Research expects VarDrop to be utilized to improve the learning efficiency of time series forecasting models in various industries such as manufacturing and finance.

Latest Research Trends in Time Series Forecasting

1) TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting Via Cross-Modality Alignment^[2]

Next are the main research trends in time series forecasting presented at AAAI-25. In the field of time series forecasting using LLM, there have been a number of recent studies that utilize time series in text form. The study “TimeCMA:  Towards LLM-Empowered Multivariate Time Series Forecasting Via Cross-Modality Alignment^[2]” significantly improved the performance of multivariate time series forecasting by aligning two different modalities, numbers and text.

Image 4. Overall framework of TimeCMA^[2]

TimeCMA is a methodology that converts time series data into appropriate prompted text and utilizes it for forecasting. Existing prompt-based methodologies have fused two different modalities, time series and text, for forecasting. The authors of TimeCMA emphasize that there is a lot of noise in text data, which requires a strategy to disentangle it. Therefore, to solve the disentangled data issue, TimeCMA proposed a cross-modality alignment with a retrieval method to obtain disentangled and robust time series embeddings. In addition, only the last token of LLM was stored and utilized to improve the inference speed of the model.

Image 5. Performance comparison according to prompt design for time series data^[2]

A variety of prompt designs exist when representing time series in text. This study reports the results of an experiment comparing the time series prediction performance of five different prompt designs (Image 5). Among the different prompt designs, prompts 3, 4, and 5, whose last value ends in a number, performed better, and in particular, prompt 5, which explicitly expressed the trend of the time series, achieved the best forecasting performance. These results indicate that time series forecasting performance can be improved by the design of the prompts and the type of LLM, suggesting that optimization of prompt-based methodologies is essential for accurate forecasting.

2) Unlocking the Power of Patch: Patch-Based MLP for Long-Term Time Series Forecasting^[4]

Here is another example of recent research. While the majority of time series forecasting models are transformer-based, there have been a number of recent studies showing that simple neural network structures such as linear and MLPs can outperform transformer-based models. The study “Unlocking the Power of Patch:  Patch-Based MLP for Long-Term Time Series Forecasting^[4],” presented at AAAI-25, poses new questions to existing forecasting models.

“Does a transformer's success in time series forecasting stem from its structural features?”
“Is existing seasonal-trend decomposition the most effective decomposition method?”

This research suggests that the answers to these questions may not be what we expect.

Image 6. Overall Structure of PatchMLP^[4]

Image 7. Overall Structure of MLP  Layer^[4]

This study proposes a new model called PatchMLP, which applies patching to MLP in the problem of long-term time series forecasting (Image 6). PatchMLP extracts Multi-Scale Patch Embeddings for the input time series and performs Latent Vector Decomposition. The authors argue that the overly complex temporal relationships inherent in the original time series prevent simple decomposition techniques from achieving effective results and that Latent Vector Decomposition can overcome this problem. Finally, the MLP layer is used to learn Intra-/Inter-Variable Dependency (Image 7). In forecasting experiments on a variety of real-world datasets, PatchMLP outperformed other state-of-the-art models, including existing Transformer-based models. These results shed new light on the question of whether the commonly adopted Transformer and Seasonal-Trend Decomposition are always a good choice.

Future Plans

In addition to improving model learning and inference efficiency, the field of time series forecasting has recently focused on improving forecasting performance using LLM and developing models utilizing various neural network structures. In the future, the field of time series forecasting is expected to expand to multi-modal forecasting utilizing text and image information, and research is expected to expand in various directions, including the development of foundation models trained on time series data from various industries such as manufacturing, finance, and energy. Multi-modal forecasting will make more realistic and accurate forecasts by comprehensively considering information other than time series, and foundation models that have learned time series data from various industries will increase the efficiency of developing new forecasting models in various applications. LG AI Research will also continue to conduct various research, including the “VarDrop” paper presented at AAAI-25, which deals with improving the learning efficiency of multivariate time series forecasting models.

AAAI-25 series

EP.1 [AAAI-25] Study on Denoising Distillation Model Using 3D Coordinates
EP.2 [AAAI-25] A Novel Augmentation Technique in Truncated SVD-Based Representation Space for Improving Tabular Data Prediction Performance