@ptrblck
First i explain how my architecture looks like
Net_1(inp1,inp2):
out_1=series of convolution layers for (inp1)
out_2=series of convolution layers for (inp2)
return out_1,out_2

Net_2(inp1)
out_3 = Series of fcc (inp1)
return out_3

Net Ensemble(inp1,inp2)
out_1,out_2=Net_1(inp1,inp2)
z= Do some operation with (out_1,out_2)
out_f=Net_2(z)
return out_f

Backprop using net ensemble.

After digging through, i found this
There are two scenarios that turns out when i load the saved model.
a)First Case: When i loads the saved model , it loads the same parameters (weights and biases) of all the layers present in my architecture during training, but in this case i get nan value for only out_2 from Net_1, but i dont get nan’s for out_1.
b)Second Case: . When i loads the saved model , it returns different values [dump] for all the parameters of all the layers present in my architecture, in this case i don’t get nan from both out_1 and out_2
All this happens while loading the same saved file , so both the cases occurs inconsistently.
I ran the architecture for 120 epochs , and the model seems to perform as expected while training and i don’t get nan’s for both out_1,out_2 while training the model , this issue only arises after i save my model.
For debugging purposes i used torch.ones as input for inp_1,inp_2
I don’t understand how to solve this issue.
The virutal enviroment is same for training and loading purposes, except i am using GPU while training and CPU while loading, although when i train on GPU and also load on GPU the error persist.