FYI, excellent debugging from @mrshenli and @ezyang deep in the guts of autograd led to https://github.com/pytorch/pytorch/pull/22983 80 and this was merged yesterday. Please give the latest nightly builds a try to see if fixed the issue.