I guess the problem is, that indexing makes the tensors non-contiguous (meaning they are not in neighboring memory cells). You could give it a try with
for j in range(sequence['input'].size(2) - 1):
                inputs = sequence['input'][:, :, j:j+2, :, :].contiguous().cuda(args.gpu, non_blocking=True)
                t = sequence['target'][:, :, j+1, :, :].contiguous().cuda(args.gpu, non_blocking=True)


This snippet may realllocate some memory. Usually contiguous tensors become non-contiguous by some view/reshaping-operations (and maybe indexing)