You’d create data windows for sampling purposes, but Conv & RNN inputs have no “window” dimension - it is just “time”.
With your toy problem, different batching/sampling schemes would work, but general approach would be:

sample batch_size sequences of length t
use input shape [batch, channels_in=1, t]
in general, you can predict multiple values in parallel (one step into the future), and this is the natural approach with convolutions. but it is possible to use many-to-one mode too (keywords: dilated causal convolutions).