You are right. The outer loop could use a single sample (and expand it to a matching batch size), while the inner loop could use batches.
This would calculate (left is outer loop sample, right inner loop samples):
[0] - [0]
[0] - [1]
[0] - [3]
# next batch
[0] - [4]
[0] - [5]
[0] - [6]
# next outer loop iteration
[1] - [0]
[1] - [1]
[1] - [2]
...