title: Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of
    Experts
summary: Deep learning for time series forecasting has seen significant advancements
  over the past decades. However, despite the success of large-scale pre-training
  in language and vision domains, pre-trained time series models remain limited
  in scale and operate at a high cost, hindering the development of larger
  capable forecasting models in real-world applications. In response, we
  introduce Time-MoE, a scalable and unified architecture designed to pre-train
  larger, more capable forecasting foundation models while reducing inference
  costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE
  enhances computational efficiency by activating only a subset of networks for
  each prediction, reducing computational load while maintaining high model
  capacity. This allows Time-MoE to scale effectively without a corresponding
  increase in inference costs. Time-MoE comprises a family of decoder-only
  transformer models that operate in an auto-regressive manner and support
  flexible forecasting horizons with varying input context lengths. We
  pre-trained these models on our newly introduced large-scale data Time-300B,
  which spans over 9 domains and encompassing over 300 billion time points. For
  the first time, we scaled a time series foundation model up to 2.4 billion
  parameters, achieving significantly improved forecasting precision. Our results
  validate the applicability of scaling laws for training tokens and model size
  in the context of time series forecasting. Compared to dense models with the
  same number of activated parameters or equivalent computation budgets, our
  models consistently outperform them by large margin. These advancements
  position Time-MoE as a state-of-the-art solution for tackling real-world time
  series forecasting challenges with superior capability, efficiency, and
  flexibility.
paper_authors._id: 66f39a43c99cc12bf2b1226d
paper_authors.name: Xiaoming Shi
paper_authors._id: 66f39a43c99cc12bf2b1226e
paper_authors.name: Shiyu Wang
paper_authors._id: 66f39a43c99cc12bf2b1226f
paper_authors.name: Yuqi Nie
paper_authors._id: 66f39a43c99cc12bf2b12270
paper_authors.name: Dianqi Li
paper_authors._id: 66f39a43c99cc12bf2b12271
paper_authors.name: Zhou Ye
paper_authors._id: 66f39a43c99cc12bf2b12272
paper_authors.name: Qingsong Wen
paper_authors._id: 66f39a43c99cc12bf2b12273
paper_authors.name: Ming Jin
paper_title: Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of
    Experts
paper_summary: Deep learning for time series forecasting has seen significant advancements
  over the past decades. However, despite the success of large-scale pre-training
  in language and vision domains, pre-trained time series models remain limited
  in scale and operate at a high cost, hindering the development of larger
  capable forecasting models in real-world applications. In response, we
  introduce Time-MoE, a scalable and unified architecture designed to pre-train
  larger, more capable forecasting foundation models while reducing inference
  costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE
  enhances computational efficiency by activating only a subset of networks for
  each prediction, reducing computational load while maintaining high model
  capacity. This allows Time-MoE to scale effectively without a corresponding
  increase in inference costs. Time-MoE comprises a family of decoder-only
  transformer models that operate in an auto-regressive manner and support
  flexible forecasting horizons with varying input context lengths. We
  pre-trained these models on our newly introduced large-scale data Time-300B,
  which spans over 9 domains and encompassing over 300 billion time points. For
  the first time, we scaled a time series foundation model up to 2.4 billion
  parameters, achieving significantly improved forecasting precision. Our results
  validate the applicability of scaling laws for training tokens and model size
  in the context of time series forecasting. Compared to dense models with the
  same number of activated parameters or equivalent computation budgets, our
  models consistently outperform them by large margin. These advancements
  position Time-MoE as a state-of-the-art solution for tackling real-world time
  series forecasting challenges with superior capability, efficiency, and
  flexibility.
paper_ai_summary: Time-MoE, a scalable and efficient architecture using a sparse mixture-of-experts design, achieves state-of-the-art time series forecasting by pre-training large models on a massive dataset without increasing inference costs.
paper_ai_keywords: Time-MoE mixture-of-experts MoE computational efficiency decoder-only transformer auto-regressive forecasting horizons input context lengths Time-300B forecasting precision scaling laws token training