Are you also running a max-pooling layer? I can see how a max-pooling implementation somewhere might start with
max = -inf

and then try to find the max of a window, but comparing anything to nan is futile, and maybe the max remains to be -inf