Thanks for your quick response. Yes And I don’t know how this is possible. I assumed that anomaly detection find the first occurrence of NaN and reports it, doesn’t it?
Any better way to find the first nan in network? Here is the warning from the second error
Warning: Error detected in MulBackward0. Traceback of forward call that caused the error:
  File "train.py", line 339, in <module>
    main(args)
  File "train.py", line 133, in main
    train_joint
  File "train.py", line 202, in train
    p, planlogits = graph_model(batch)
  File "~/anaconda3/envs/torch-sci/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "~/torch-sci/GraphWriter/models/newmodel.py", line 158, in forward
    z = (1-s)*z

I can keep tracking it back that way, It’s kinda time consuming because I have to run my model for couple of iterations until I get the next error