If the data inputs are sequentially dependent on each other(such as in a time series), perhaps what you need is a recurrent neural network(i.e. LSTM, GRU, etc.) or a transformer. RNNs have a gated memory in the layer, good for the model to learn sequential information. Transformers, on the other hand, keep positional information of the sequence, and allow for more asynchronuous processing of sequential information. I.e. you might put in the last frame as inputs in an RNN, while in a Transformer, you’d put in 60 frames as one input(batches would then be windows of 60 frames while just randomizing which window is used).
What you’re asking might not be possible as autograd stores the operations in a DAG during forward propagation. You can read more about autograd here:
https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#computational-graph
But I think if you become more familiar with the math behind the scenes, you’ll find there are more elegant solutions to your problem.