Yeah, to leverage attention, you need to be predicting a finite number of embeddings that hopefully you see lots of in training.
So to use this for time series prediction, you want a transformer to operate on higher level, discrete features than the sample space. Applying it directly to samples is like a classification problem with 2^16 classes (for 16 bit audio, say), which is probably too many and this problem formulation ignores the inherent correlation between classes.
Transformers should be used to predict things like beats, words, high level recurring patterns.
An architecture might be Time series → Conv blocks → quantization → Transformer → Deconv → Fully connected → Time series. Check out Facebook’s Wav2Vec paper for such an example.