Update:
If I stick
torch.cuda.memory_summary(device=None, abbreviated=True)

in a loop that I iterate over 25 times I can push up to a batch size of 16. Sleeping and waiting in a busy loop do not reproduce the same behaviour.
Without looping over this function invocation multiple times I see periods where I will get many CUDA allocation failures in a row. Looping results in occasional CUDA allocation failures which is what I would expect due to sometimes encountering input sequences of high length.