Well, frankly, implementation details vary, some being pretty complex. I’d suggest to start from some existing design for a similar problem.
Your attention description sounds right, but it is pretty generic… For encoder, it can be a self-attention layer, to support unequal importance of time points. For some tasks, you’d want to encode positions too (and/or time-based features), for same reason as in NLP - to make element order matter.
As for decoders, I’m not too familiar with RNN-less context unrolling for convolutional decoders. I believe another attention module is one way to do it. Again, it may be better to check what they did in relevant research papers.