Are you sure you’re measuring that right? I’m asking because most people don’t.
For integral scaling, it can be faster (and with deterministic backward and all) to use array operations. You might split this into several lines, but to give you an idea, this would upscale a BCHW inp by 2x:
out = inp[:,:, :,None, :, None].expand(-1, -1, -1, 2, -1, 2).reshape(inp.size(0), inp.size(1), 2 * inp.size(2), 2* inp.size(3))

Best regards
Thomas