Jaehoon Lee
2024.05.03
The Latest Research Trends in Time Series Foundation Models
When people talk about time series data in AI, they are referring to tabular data with a time axis, such as stock prices or sensor data. Time series data can be found everywhere around us, from weather data to factory sensor data, financial data, and more. With time series data so intertwined in our lives, solving time series problems holds significant importance for our everyday lives. If we can predict the weather, breakdowns in factory machinery, or changes in finance in advance, we can better respond to uncertainties in the future. Currently, many companies are working hard to solve time series problems through the use of AI. In the following sections, I will outline some of the current research, and shed light on time series foundation models, an area that is gaining particular interest as of late.
1. The Rise of Time Series Foundation Models
A foundation model is an AI model that has been pre-trained with large-scale data and has the ability to solve a wide range of problems. It’s important to note that rather than being specific to a particular problem, a foundation model is a generalized model. Several foundation models currently being studied in areas such as natural language and vision show huge potential, with the best-performing models excelling at various tasks. As a result, a lot of time and resources are devoted to creating foundation models for time series.
2. Categorization of Time Series Foundation Models
Current research on time series foundation models can be categorized as shown in Figure 1 below. Methodologies can be divided into those that utilize large language models(LLM) and those that build its own large models for time series. Methodologies based on large language models can further be split into three categories—1) Without Adaptation, 2) Adapt LLM, 3) Adapt to LLM—depending on how they utilize the models. Additionally, research into building specific language models includes collecting large time series data and creating one unified model for training from diverse time series data.
Figure 1. Classification of Time Series Foundation Models
2-1. Time Series Foundation Models Based on Large Language Models
This methodology essentially seeks to build a time series foundation model by gaining generalized capabilities from a large language model. It can be divided into the following subcategories depending on how the large language model is utilized.
1) Without Adaptation. This methodology assumes that a large language model has acquired some knowledge of the time series domain from a variety of natural language data. Thus, the process of adapting large language models to the time series domain is minimal or doesn’t exist. The main focus is how to write good prompts for time series problems.
For example, PrompCast has designed its input-output text to solve time series problems as shown in Figure 2. However, one paper suggests that designing prompts in this way can have major problems, with irregular tokenization of numbers cited as a main example. For example, when tokenizing 480, it’s created with one token, which is 480, while 481 is created with two tokens, 48 and 1. This begs the question of whether a large language model can understand numbers well.
LLM-Time tried to solve this problem by representing each digit as a single token so that the large language model could easily comprehend the time series problem. Interestingly enough, this research showed that the large language model performed well in predicting time series without any additional training processes such as fine-tuning. According to the research, this is because large language models have the ability to identify simple patterns, and when solving time series problems, focusing on simple patterns usually leads to better performance than complex ones.
Figure 2. PromptCast Structure[2]
2) Adapt LLM. Unlike the Without Adaptation methodology, this methodology emphasizes that by default, the large language model is unfamiliar with the time series domain and needs to be adapted. The model used for this methodology typically comes in the form shown in Figure 3. The large language model is the central model, and there’s an adaptation layer that connects the time series data to the large language model from front to back. Fine-tuning all or part of the large language model and its surrounding adaptation layers to time series data allows the large language model to become more familiar with time series data.
The most important question of this method is how to design the adaptation layer and how to fine-tune the large language model. OFA used patching to tokenize the time series data and designed the model with linear layers both in front of and behind the large language model, as shown in Figure 4. It also improved performance by training only some of the parameters of the large language model rather than all of them. This research showed that the Adapt LLM methodology can be expanded to different time series problems by having different output layers for different time series problems. For example, if the output layer is a simple linear layer, it becomes a model for solving regression problems. Conversely, if the output layer is a combination of linear and softmax, it becomes a model for classification problems.
Based on this research, TEMPO utilized time series decomposition-based patching and low-rank adaptation (LoRA) to further improve performance.[6] And, the LLM4TS research also proposed a way to further refine the fine-tuning process.[7] It proposed first training the model to familiarize itself with the overall time series data, then focusing on a specific time series problem, such as prediction.
Figure 3. Differences in Model Structure between Adapt LLM and Adapt to LLM Methodologies
Figure 4. OFA Structure[5]
3) Adapt to LLM. You can think of this methodology as being somewhere in between the previous two. Without Adaptation assumes that the large language models have some knowledge of the time series domain, while Adapt LLM assumes that the large language models lack that knowledge. Unlike the other two, the Adapt to LLM methodology is based on a conjecture that lage language models have the knowledge of the time series domain that is represented by text. This means that while large language models can understand the time series domain, their abilities become limited when the domain is converted to text.
This leads to the most important question on this methodology: “How can time series data be converted well into text?” Figure 3 shows how a model is typically designed using this methodology. It’s quite similar to Adapt LLM, but the difference is that it doesn’t train a large language model at all. Figure 5 also shows how this methodology works intuitively. The fine-tuned adaptation layers on either side convert time series data into text and text into time series data. The large language model in the middle then performs specific tasks based on the text-converted time-series data. For instance, Time-LLM research utilizes the Cross Attention mechanism to clarify the process of converting time series to text. It maps time series data to text by querying the input time series data, using different words in the text as keys and values.
Figure 5. Intuitive Visualization of the Adapt to LLM Methodology
2-2. Time-Series Own Foundation Models
Unlike the previous methodologies which relied primarily on large language models to work as generalized foundation models, this methodology aims to build specific large models for a time series. Researchers who use this methodology are interested in collecting enough data so the model can be generalized. There is also an interest in building a single generalized model that can accommodate a large and heterogeneous set of data. There is various heterogeneity between the various time series data collected, such as differences in observation frequency or time series domain.
Examples of research in this methodology include TimeGPT[9], TimesFM[10], and Moirai[11]. Figure 6 summarizes the characteristics of these models. All of these methodologies sought to build a foundation model by collecting a massive amount of data totaling over 10 billion pieces and using it all to train the model.
Among them, TimesFM created and utilized Google Trend data to show trends in search frequency, Wiki Pageview data to show the number of views of Wikimedia pages, and synthetic data with simple time series patterns. Unlike other research, this methodology uses more readily available data to build a large dataset to train the model, which has a more meaningful impact.
When it comes to constructing generalized models, most studies utilizing this methodology adopt transformer family models, which have more than 100 million parameters. Some use encoder-decoder series models, such as TimeGPT, while others use decoder-only series models, such as TimesFM and Moirai. To build generalized models from time series, Moirai uses different patch sizes for different observation frequencies and solves multivariate time series problems by randomly sampling variables.
Figure 6. Characteristics of TimeGPT[9], TimesFM[10], Moirai[11]
3. The Future Plans of LG AI Research
As mentioned above, foundation models for time series are being actively researched by a handful of companies. However, given the age of such research, there are many areas that need further study and improvement. At LG AI Research, we are working to discover and solve fundamental problems with existing research.
At LG AI Research’s Advanced Machine Learning(AML) Lab, we’ve discovered that most time series foundation models are univariate models that cannot account for multivariates. We are currently exploring ways to solve this limitation. Although the degree of connection may vary from case to case, every single time series in the world is connected to each other in some way. For example, rising temperatures increase the demand for air conditioners, which affects the stock price of a particular company. This correlation is where foundation model research, which seeks to build general time series models, first took shape.
Due to this, most existing models have developed univariate-based methodologies, focusing on one time series at a time. Building a multivariate foundation model is nearly impossible with current technology, as it would necessitate considering an almost infinite number of time series simultaneously.
However, in AML lab, we have identified the limitations of univariate models in terms of their model capacity compared to multivariate models. Furthermore, we have proposed a random sampling method to transition towards multivariate models, particularly in situations where the number of time series approaches infinity. Rather than considering all of the near-infinite time series at once, we saw the possibility that learning from a random sample of just a few of them could have the same effect as training from the whole.
Additionally, LG AI Research has discovered other important fundamental issues that need to be addressed in time series foundation models and will continue to research them to create more advanced time series foundation models.
- 참고
- [1] R. Bommasani et al., “On the Opportunities and Risks of Foundation Models.” arXiv, Jul. 12, 2022. doi: 10.48550/arXiv.2108.07258.
[2] H. Xue and F. D. Salim, “PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting,” arXiv.org. Accessed: Jan. 08, 2024. [Online]. Available: https://arxiv.org/abs/2210.08964v5
[3] D. Spathis and F. Kawsar, “The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models,” arXiv.org. Accessed: Jan. 08, 2024. [Online]. Available: https://arxiv.org/abs/2309.06236v1
[4] N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson, “Large Language Models Are Zero-Shot Time Series Forecasters,” arXiv.org. Accessed: Jan. 08, 2024. [Online]. Available: https://arxiv.org/abs/2310.07820v1
[5] T. Zhou, P. Niu, X. Wang, L. Sun, and R. Jin, “One Fits All:Power General Time Series Analysis by Pretrained LM,” arXiv.org. Accessed: Jan. 08, 2024. [Online]. Available: https://arxiv.org/abs/2302.11939v6
[6] D. Cao et al., “TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting,” arXiv.org. Accessed: Jan. 08, 2024. [Online]. Available: https://arxiv.org/abs/2310.04948v2
[7] C. Chang, W.-Y. Wang, W.-C. Peng, and T.-F. Chen, “LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters,” arXiv.org. Accessed: Jan. 08, 2024. [Online]. Available: https://arxiv.org/abs/2308.08469v4
[8] M. Jin et al., “TIME-LLM: TIME SERIES FORECASTING BY REPRO- GRAMMING LARGE LANGUAGE MODELS”.
[9] A. Garza and M. Mergenthaler-Canseco, “TimeGPT-1.” arXiv, Oct. 05, 2023. Accessed: Feb. 14, 2024. [Online]. Available: http://arxiv.org/abs/2310.03589
[10] A. Das, W. Kong, R. Sen, and Y. Zhou, “A decoder-only foundation model for time-series forecasting.” arXiv, Feb. 04, 2024. Accessed: Feb. 13, 2024. [Online]. Available: http://arxiv.org/abs/2310.10688
[11] G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified Training of Universal Time Series Forecasting Transformers.” arXiv, Feb. 04, 2024. Accessed: Feb. 13, 2024. [Online]. Available: http://arxiv.org/abs/2402.02592