[ICLR 2024] Advancements in Time Series Forecasting

Time series forecasting, which uses artificial intelligence to predict the future based on past data, is an important area of research that can assist the decision-making process by providing information about the future in various areas.  LG AI Research’s Data Intelligence Lab is actively conducting a variety of research projects in the field of time series forecasting AI as a significant technology that can solve problems and concerns during the decision-making process, such as demand forecasting and commodity price forecasting. Innovation in time series forecasting has accelerated in recent years, and the upcoming International Conference on Learning Representations (ICLR) in May 2024 will feature more than 30 papers presenting different methodologies to improve the accuracy and efficiency of time series forecasting.

The time series forecasting AI papers to be presented at ICLR 2024 cover a wide range of research topics, including time series forecasting model architectures, time series forecasting using language models, time series forecasting using diffusion models, and time series representation learning. This article will introduce some of the most noteworthy studies of time series forecasting AI research presented at ICLR and share LG AI Research’s vision and concerns in the field of time series forecasting.

---

FITS: Modeling Time Series with 10k Parameters^[1]

The Frequency Interpolation Time Series Analysis Baseline (FITS) proposes an innovative methodology that utilizes the frequency domain for efficient time series forecasting. By leveraging the equivalence between forecasting/extrapolation in the time domain and interpolation in the frequency domain, FITS achieves excellent forecasting performance with as few as 10,000 parameters. FITS utilizes a single complex-valued linear layer for interpolation and has shown that applying a low-frequency pass filter based on harmonic frequencies can significantly reduce the model's size while preserving forecasting performance. The results of FITS, which utilizes the frequency domain to achieve good forecasting performance with a small model, show the potential of utilizing the frequency domain to develop models with high forecasting accuracy and computational efficiency.

Experiments with long-term time series forecasting benchmark datasets show that FITS outperforms existing SOTA(state-of-the-art) models on most benchmarks. In addition to the performance benefits, FITS sets itself apart from existing models in terms of computational efficiency. With its ability to achieve high forecasting performance with a small number of parameters and a small amount of computation, FITS is likely to be useful in environments where it is challenging to configure high-performance computing equipment or where multiple forecasting models need to be trained on different types of time series, such as demand forecasting for different products.

Figure 1. Methodological overview of Frequency Interpolation Time Series Analysis Baseline (FITS)^[1]

Figure 2. Benchmark forecasting performance comparison table of FITS^[1] and baseline 

Figure 3. Comparison table of computation amount between FITS^[1] and baseline methodologies

ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis^[2]

ModernTCN proposes an improved Temporal Convolution Network (TCN) model for general time series analysis, based on an innovative Convolutional Network architecture inspired by the structure of the Transformer model. The proposed methodology splits the time series data into multiple patches and embeds them as features to effectively identify temporal patterns. By using new convolution structures of depthwise convolution and convolution feed-forward networks, ModernTCN outperforms previous state-of-the-art models on various time series problems. The high-performance results of ModernTCN, which integrates the convolution architecture with the transformer design, demonstrate the potential for synergy between different neural network paradigm combinations in time series forecasting.

ModernTCN performs well on a variety of time series problems beyond time series forecasting, including anomaly detection, classification, and missing value recovery. Moreover, ModernTCN is a convolutional-based model with a faster learning speed. It uses less memory than transformer-based forecasting models, giving it a good balance of performance and computation amount. Given these characteristics, ModernTCN has the potential to be used as the underlying structure of a pre-trained time series model that can be shared and utilized for various time series tasks as a single model.

Figure 4. Temporal convolution block design proposed by ModernTCN^[2]

Figure 5. Comparison of performance (left) and training time compared to performance (right) of ModernTCN^[2]

TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting^[3]

TimeMixer proposes a new approach to leverage multi-scale time series sampled at different scales to capture different patterns in the time series. This methodology decomposes time series data into trend and seasonality components at various scales. It performs a feature mixing process that blends information from multi-scale time series by reflecting the characteristics of each component. By adopting a coarse-to-fine (top-down) feature mixing strategy for the trend component and a fine-to-coarse (bottom-up) feature mixing strategy for the seasonality component, TimeMixer outperforms existing methodologies in time series forecasting performance. TimeMixer’s results show that utilizing time series data at various scales can be a useful tool for capturing the diverse dynamics of time series data in the real world.

Figure 6. Methodology overview of TimeMixer^[3]

Figure 7. Benchmark forecasting performance comparison table of TimeMixer^[3] and baseline methodologies

Figure 8. TimeMixer^[3]’s feature mixing strategy analysis experiment result table

TimeMixer’s experimental results show that the proposed model outperforms existing models in forecasting and that the proposed bottom-up/top-down feature mixing strategies capture the relatively macroscopic nature of trends and the microscopic nature of seasonality. Therefore, it is believed that forecasting models that appropriately reflect the characteristics of trend and seasonality, such as TimeMixer, can be effectively utilized to solve forecasting problems in situations where macroscopic trendiness and microscopic periodicity are mixed.

TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting^[4]

TEMPO proposes a new methodology for learning time series models by fine-tuning a language model (GPT-2) with LoRA. This methodology decomposes time series data into trend, seasonality, and residual components using STL decomposition techniques, and then divides each component into multiple patches to be used as input to the model, allowing the model to better characterize the time series. Furthermore, TEMPO introduces a prompt pool with multiple soft prompts and allows the selection of the appropriate prompt for each component, resulting in more effective forecasts and higher performance on a variety of datasets. TEMPO also shows cross-domain learning capabilities on time series data from different domains, demonstrating the potential of utilizing language models in time series forecasting.

Figure 9. Methodology overview of TEMPO^[4]

Figure 10. Benchmark forecasting performance comparison table of TEMPO^[4] and baseline methodologies

Figure 11. Multimodal forecasting performance comparison table of TEMPO^[4] and baseline methodologies

TEMPO has cross-domain learning capabilities and outperforms existing models in zero-shot forecasting experiments based on cross-domain learning. On the other hand, TEMPO is a multimodal forecasting model that can use time series data and textual data together. It performs well in multimodal forecasting experiments with textual data and time series data used together. Multimodal forecasting models such as TEMPO are likely to be leveraged to solve the problem of forecasting commodity prices or stock prices, where the complexity of price determination limits the ability to predict using only quantitative information from structured time series data and requires the use of qualitative information from unstructured textual data for more accurate forecasting.

Generative Learning for Financial Time Series with Irregular and Scale-Invariant Patterns^[5]

Financial data is different from other time series data in that it has irregular and scale-invariant patterns. This paper introduces Financial Time Series Diffusion (FTS-Diffusion), a new generative learning framework for financial data. FTS-diffusion is a combination of three different networks: a pattern-conditioned diffusion network, a scaling autoencoder, and a pattern evolution network. By using techniques such as Dynamic Time Warping (DTW) and K-means clustering, FTS-diffusion performs well in capturing and utilizing complex patterns in financial data. The proposed methodology outperforms the existing baseline techniques in data augmentation experiments on financial time series forecasting problems and shows an excellent ability to process financial data with irregular and scale-invariant patterns.

Figure 12. Methodological overview of FTS-diffusion^[5] 

Figure 13. Comparison of data augmentation experiment performance using FTS-diffusion^[5]

FTS-diffusion is a generative model that reflects the characteristics of financial data, which are different from general time series data, and has outperformed existing generative models in data augmentation experiments using generative models. The FTS-diffusion model proposed in this paper is a generative model used for data augmentation, but if the research on time series models that reflect the characteristics of financial data can be expanded to forecasting models instead of generative models, it is expected to effectively improve the time series forecasting performance in the financial field, such as stock prices and exchange rates.

Soft Contrastive Learning for Time Series^[6]

SoftCLT proposes a new contrastive learning approach specialized for time series data that effectively trains the information and correlations inherent in time series data. By combining two contrastive learning techniques, instance-wise contrastive learning, and temporal contrastive learning, SoftCLT effectively learns the correlation between time series and the correlation between points in a time series. The proposed methodology can be used in combination with existing time series representation learning techniques. By training better representations, it leads to improved performance in various downstream tasks such as time series classification, anomaly detection, and forecasting.

Figure 14. Methodology overview of SoftCLT^[6]

Figure 15. Time series classification (top left), anomaly detection (bottom left), and time series forecasting (right) SoftCLT^[6] performance table

SoftCLT’s learning techniques are combined with existing representation learning techniques to learn better representations, resulting in better performance in time series classification, anomaly detection, and forecasting. The SoftCLT technique, which learns a representation of time series data that reflects the characteristics of the time series, is expected to be useful for learning large amounts of time series data to train a pre-trained model that can be used commonly for various time series tasks.

DAM: A Foundation Model for Forecasting^[7]

DAM aims to create unified forecasting models that can handle different time series datasets and forecasting environments with a single model. The methodology utilizes predefined basis functions with 437 different frequencies covering periods from one minute to several years, enabling basis-based interpretable forecasts. DAM utilizes Coefficient to represent input time series on a predefined basis along with the input time series to make forecasts, and in doing so, utilizes a transformer-based architecture to take advantage of irregularly sampled time series data. DAM can learn using multiple time series datasets with different characteristics, perform well on a variety of domains, and shows great adaptability to different forecasting environments.

Figure 16. Methodology overview of DAM^[7]

 

To ensure that DAM performs well in a variety of forecasting environments, this paper measured the performance of a single DAM model trained across multiple domains. Experimental results showed that the DAM model performed comparably to forecasting models learned specifically for each domain across a variety of domains, demonstrating the potential of integrated models to address multiple forecasting problems with a single model. Furthermore, it outperformed existing models in zero-shot forecasting experiments, where it forecasts on new domains not used in the training process, demonstrating its adaptability to new forecasting environments. With this cross-domain forecasting performance and adaptability to new domains, a foundation forecasting model addresses a wide range of forecasting problems and will be a useful tool for solving problems such as demand forecasting, where new forecasting problems arise frequently.

Figure 17. Benchmark forecasting performance comparison table of DAM^[7] and baseline methodologies

Figure 18. Forecasting performance comparison table for new domains between DAM^[7] and baseline methodologies

 

Conclusion: Towards Future Innovations

Recently, research on time series forecasting AI has been increasingly active, and recent research results are not limited to the development of simple forecasting model structures and training techniques but also in various directions that overcome the limitations of existing forecasting techniques by utilizing various tools, from traditional time series analysis methods to the recently emerging language models and diffusion models. The papers to be presented at ICLR 2024 utilize these tools to develop models that reflect the intrinsic characteristics of time series data to improve forecasting performance and further open up new possibilities, such as cross-domain forecasting and zero-shot forecasting.

To solve the problem of time series forecasting to support decision-making, which is the focus of LG AI Research, we need to be able to make reliable forecasts with high accuracy in a variety of environments. Therefore, in addition to high forecasting performance, forecasting models must be adaptable to new forecasting environments and reliable to ensure that their forecasts are well-founded. It is expected that the research presented at ICLR 2024, such as forecasting techniques that reflect the characteristics of time series, cross-domain forecasting techniques, and multimodal forecasting techniques using text data, are in line with the direction that LG AI Research is considering and aiming for in time series forecasting. LG AI Research’s Data Intelligence Lab is conducting a variety of research, including research on forecasting models using multimodal data and research on time series foundation models, with a particular focus on forecasting technology that makes accurate and reliable forecasts by securing the basis for forecasting. Based on this, we plan to continue to work on securing more reliable forecast technology and applying them in the industry.