I guess you can modify the original pad_sequence that you can find here 21 for your needs in the 2D case. By replacing things like out_tensor[i, :length, ...] = tensor by out_tensor[i, :height, :width ...] = tensor.