As this layer creates the NaN output, I would try to narrow down, when and if its parameters are getting NaN values.
Since the input to this layer is apparently finite, the weight or bias has to become NaN or Inf eventually to create a NaN output.