It depends on your use case what the valid workaround would be.
I.e. why are Inf values in the grid and what would they mean?
If you think these grid values should use the padding, your workaround might work, on the other hand you might want to investigate why these Inf values are created and avoid them.