It seems like I am facing a similar issue (segfault at exactly the same line), though it seems to only happen when I am using multiple GPUs. In my case, it started happening after I added some code for accumulating gradients instead of updating at every batch.
I am also using pytorch from master.
@rotabulo, may I ask if you also get this when using only one GPU, or no GPUs at all?