Could you check the inputs, output, as well as the parameters of this layer and their stats in particular?
Since this layer is now running in float32, the values are either huge in their magnitude, or another layer is causing the invalid values.