Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning
<|reference_start|>Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning: In this study, we present aLLM4TS, an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning. Central to our approach is that we reconceive time-series forecasting as a self-supervised, multi-patch prediction task, which, compared to traditional contrastive learning or mask-and-reconstruction methods, captures temporal dynamics in patch representations more effectively. Our strategy encompasses two-stage training: (i). a causal continual pre-training phase on various time-series datasets, anchored on next patch prediction, effectively syncing LLM capabilities with the intricacies of time-series data; (ii). fine-tuning for multi-patch prediction in the targeted time-series context. A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding. Such a design directly transposes individual patches into temporal sequences, thereby significantly bolstering the model's proficiency in mastering temporal patch-based representations. aLLM4TS demonstrates superior performance in several downstream tasks, proving its effectiveness in deriving temporal representations with enhanced transferability and marking a pivotal advancement in the adaptation of LLMs for time-series analysis.<|reference_end|>
@article{bian2024multi-patch,
    title={Multi-Patch Prediction: Adapting LLMs for Time Series Representation
  Learning},
    author={Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, Qiang Xu},
    journal={arXiv preprint arXiv:2402.04852},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.04852},
    primaryClass={cs.LG}
}