I figured out the error. I had a variable as a dictionary and I was using some elements of the dictionary as inputs to another neural network. I worked around it by having the this new input to require grad as False using .detach() method.