@smth thanks. So, just re-read the docs, looks like i can basically do this as well?

simulate a batch of 32 by running each seq at a time and calling .step() only once
for ex_i in range(0, 32):
    # gradients accumulate every call to backward
    # by calling step after 32 backward calls we can simulate a batch of 32
    loss = someLoss(f(x), y)
    loss.backward()

optimizer.step()