Great! I think that if you can keep them as Tensors (i.e. don’t strictly need autograd for that part), then it’s best to do this. The error was caused because the history was never freed, I don’t think it has to do anything with masked_copy_. If you look into how assignment is implemented, it actually also uses an in-place function (but it’s hidden behind that syntactic sugar).
Cool! Once you put up your PyTorch implementation I can take a look to make sure it’s running at full speed!