Just in case anyone ever finds this - the code above works fine. The issue was when the network detects objects, sometimes their coordinates happen to be in the padded area. As a result, when you’re trying to rescale the boxes, they go beyond edges of the original frames.
Does anyone know why this might be happening? I had to write a simple condition like this to catch it. Works fine now.