Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of
  Experts
Deep learning for time series forecasting has seen significant advancements
over the past decades. However, despite the success of large-scale pre-training
in language and vision domains, pre-trained time series models remain limited
in scale and operate at a high cost, hindering the development of larger
capable forecasting models in real-world applications. In response, we
introduce Time-MoE, a scalable and unified architecture designed to pre-train
larger, more capable forecasting foundation models while reducing inference
costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE
enhances computational efficiency by activating only a subset of networks for
each prediction, reducing computational load while maintaining high model
capacity. This allows Time-MoE to scale effectively without a corresponding
increase in inference costs. Time-MoE comprises a family of decoder-only
transformer models that operate in an auto-regressive manner and support
flexible forecasting horizons with varying input context lengths. We
pre-trained these models on our newly introduced large-scale data Time-300B,
which spans over 9 domains and encompassing over 300 billion time points. For
the first time, we scaled a time series foundation model up to 2.4 billion
parameters, achieving significantly improved forecasting precision. Our results
validate the applicability of scaling laws for training tokens and model size
in the context of time series forecasting. Compared to dense models with the
same number of activated parameters or equivalent computation budgets, our
models consistently outperform them by large margin. These advancements
position Time-MoE as a state-of-the-art solution for tackling real-world time
series forecasting challenges with superior capability, efficiency, and
flexibility.
66f39a43c99cc12bf2b1226d
Xiaoming Shi
66f39a43c99cc12bf2b1226e
Shiyu Wang
66f39a43c99cc12bf2b1226f
Yuqi Nie
66f39a43c99cc12bf2b12270
Dianqi Li
66f39a43c99cc12bf2b12271
Zhou Ye
66f39a43c99cc12bf2b12272
Qingsong Wen
66f39a43c99cc12bf2b12273
Ming Jin
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of
  Experts
Deep learning for time series forecasting has seen significant advancements
over the past decades. However, despite the success of large-scale pre-training
in language and vision domains, pre-trained time series models remain limited
in scale and operate at a high cost, hindering the development of larger
capable forecasting models in real-world applications. In response, we
introduce Time-MoE, a scalable and unified architecture designed to pre-train
larger, more capable forecasting foundation models while reducing inference
costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE
enhances computational efficiency by activating only a subset of networks for
each prediction, reducing computational load while maintaining high model
capacity. This allows Time-MoE to scale effectively without a corresponding
increase in inference costs. Time-MoE comprises a family of decoder-only
transformer models that operate in an auto-regressive manner and support
flexible forecasting horizons with varying input context lengths. We
pre-trained these models on our newly introduced large-scale data Time-300B,
which spans over 9 domains and encompassing over 300 billion time points. For
the first time, we scaled a time series foundation model up to 2.4 billion
parameters, achieving significantly improved forecasting precision. Our results
validate the applicability of scaling laws for training tokens and model size
in the context of time series forecasting. Compared to dense models with the
same number of activated parameters or equivalent computation budgets, our
models consistently outperform them by large margin. These advancements
position Time-MoE as a state-of-the-art solution for tackling real-world time
series forecasting challenges with superior capability, efficiency, and
flexibility.
Time-MoE, a scalable and efficient architecture using a sparse mixture-of-experts design, achieves state-of-the-art time series forecasting by pre-training large models on a massive dataset without increasing inference costs.
Time-MoE mixture-of-experts MoE computational efficiency decoder-only transformer auto-regressive forecasting horizons input context lengths Time-300B forecasting precision scaling laws token training