Hello man, I ones had the same problem,  I was getting Nans all the time. Lastly I found out that my data contains nans. Run the below command to check if your data contains Nans
outputs = model(images)
check = int((outputs != outputs).sum())
if(check>0):
    print("your data contains Nan")
else:
    print("Your data does not contain Nan, it might be other problem")