Code release for "Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting" (NeurIPS 2022), https://arxiv.org/abs/2205.14415.
Conceptual Captions is a dataset containing (image-URL, caption) pairs designed for the training and evaluation of machine learned image captioning systems.
Fast SHAP value computation for interpreting tree-based models.
Key parameters and specifications
Key parameters and specifications
Parameter
Value
Language
Python
Category
Model Development
Subcategory
Architecture
Keywords
Timeseries
Stars
567
Forks
102
Contributors
1
Created date
2022-09-30
Updated date
2026-10-01
Stars in 1 day
0
Stars in 7 days
0
Rank
16,894
Previous rank
16,687
Rank change
-207
7-day stars rank
15,791
Previous 7-day stars rank
15,797
7-day stars rank change
6
Total stars rank
17,568
Previous total stars rank
17,540
Total stars rank change
-28
Country
China
README summary
Non-stationary Transformers This is the codebase for the paper: Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting, NeurIPS 2022. [[Slides]]( [[Poster]]( :triangular_flag_on_post: **News** (2023. 02) Non-stationary Transformer has been included in [[Time-Series-Library]]( which covers long- and short-term forecasting, imputation, anomaly detection, and classification. Discussions There are already several discussions about our paper, we appreciate a lot for their valuable comments and efforts: [[Official]]( [[OpenReview]]( [[Zhihu]]( Architecture Series Stationarization Series Stationarization
unifies the statistics of each input and converts the output with restored statistics for better predictability. De-stationary Attention De-stationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from unstationarized series. All the six benchmark datasets can be obtained from Google Drive or Tsinghua Cloud. Training scripts Non-stationary Transformer We provide the Non-stationary Transformer experiment scripts and hyperparameters of all benchmark dataset under the folder . Non-stationary
framework to promote other Attention-based models We also provide the scripts for other Attention-based models (Informer, Autoformer), for example: Experiment Results Main Results For multivariate forecasting results, the vanilla Transformer equipped with our framework consistently achieves state-of-the-art performance in all six benchmarks and prediction lengths. Model Promotion By applying our framework to six mainstream Attention-based models. Our method consistently improves the forecasting ability. 51%** on FEDformer, making each of them surpass previous state-of
-the-art. Future Work We will keep equip the following models with our proposed Non-stationary Transformers framework: [x] Transformer [x] iTransformer [x] Informer [x] Autoformer [x] FEDformer [x] Crossformer [x] Reformer [x] . Note: Series Stationarization as an architecture-free module has been widely applied for addressing non-stationarity in time series. Please refer to time-series-library for the implementationdetails. Citation If you find this repo useful, please cite our paper. Contact If you have any questions or want
to use the code, please contact liuyong21@mails. Acknowledgement This repo is built on the Autoformer repo, we appreciate the authors a lot for their valuable code and efforts.
Frequently asked questions
How many stars and forks are recorded?
thuml/Nonstationary_Transformers has 567 stars and 102 forks.
What is the repository rank?
thuml/Nonstationary_Transformers is ranked 16,894.
What language and category are recorded?
thuml/Nonstationary_Transformers has 567 stars, uses Python and belongs to Model Development.
Which repository is closest in rank?
thuml/Nonstationary_Transformers is compared with suriyadeepan/practical_seq2seq, ranked 16,893 with 567 stars.
:triangular_flag_on_post: News (2023.02) Non-stationary Transformer has been included in [Time-Series-Library], which covers long- and short-term forecasting, imputation, anomaly detection, and classification.
Discussions
There are already several discussions about our paper, we appreciate a lot for their valuable comments and efforts: [Official], [OpenReview], [Zhihu].
Architecture
Series Stationarization
Series Stationarization unifies the statistics of each input and converts the output with restored statistics for better predictability.
De-stationary Attention
De-stationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from unstationarized series.
For multivariate forecasting results, the vanilla Transformer equipped with our framework consistently achieves state-of-the-art performance in all six benchmarks and prediction lengths.
Model Promotion
By applying our framework to six mainstream Attention-based models. Our method consistently improves the forecasting ability. Overall, it achieves averaged 49.43% promotion on Transformer, 47.34% on Informer, 46.89% on Reformer, 10.57% on Autoformer, 5.17% on ETSformer and 4.51% on FEDformer, making each of them surpass previous state-of-the-art.
Future Work
We will keep equip the following models with our proposed Non-stationary Transformers framework:
[x] Transformer
[x] iTransformer
[x] Informer
[x] Autoformer
[x] FEDformer
[x] Crossformer
[x] Reformer
[x] ......
Note: Series Stationarization as an architecture-free module has been widely applied for addressing non-stationarity in time series. Please refer to time-series-library for the implementationdetails.
Citation
If you find this repo useful, please cite our paper.
@article{liu2022non,
title={Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting},
author={Liu, Yong and Wu, Haixu and Wang, Jianmin and Long, Mingsheng},
booktitle={Advances in Neural Information Processing Systems},
year={2022}
}
