Maybe the simplest thing for you would be to set these to a small epsilon (like 1e-5).
They will have a large gradient, but it will be zeroed out by the relu/threshold as 0 * large_value = 0. Contrary to 0 * +inf = nan.