Improving Transformers using Faithful Positional Encoding
<|reference_start|>Improving Transformers using Faithful Positional Encoding: We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.<|reference_end|>
@article{idé2024improving,
    title={Improving Transformers using Faithful Positional Encoding},
    author={Tsuyoshi Id'e, Jokin Labaien, and Pin-Yu Chen},
    journal={arXiv preprint arXiv:2405.09061},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.09061},
    primaryClass={cs.LG}
}