A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Code

• github.com/yuqinie98/patchtst
  OfficialIn paperpytorch★ 2,489
• github.com/timeseriesAI/tsai
  pytorch★ 6,019
• github.com/thuml/iTransformer
  pytorch★ 2,050
• github.com/WenjieDu/PyPOTS
  pytorch★ 1,970
• github.com/etna-team/etna
  pytorch★ 193
• github.com/romilbert/samformer
  tf★ 191
• github.com/arclab-mit/sw-driver-forecaster
  pytorch★ 3
• github.com/MindCode-4/code-2/tree/main/patchtst
  mindspore★ 0

Abstract

We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.

Tasks

Benchmark Results

Reproductions