Did you remove your custom modules too? I guess you must have repacked the data into a new Variables somewhere, thus breaking the history graph. You might want to check if list(model.parameters())[0].grad is not None. If it is, you must be breaking the graph somewhere (you should never need to take the .data do some op, and wrap it in a Variable).