Following up my own post, I found that my problem seems to go away if I loop over params and simple set p.grad = None and empty the cache afterwards.
Do any pytorch devs have an opinion on the right way to do this?  @smth @apaszke ?