This looks a lot like the memory corruption issue I spent an entire week chasing and fixing. The fix is in https://github.com/pytorch/pytorch/pull/60470 2, hence it’s only available in PyTorch 1.10 for now, sorry.
If my memory serves me right, this problem occurs when an object contained in an RRef is mutated in-place, in particular when one of its tensors is removed. (Like, imagine an RRef holding a dict of tensors, and one of those items being popped from the dict).
My fix ensures that in those cases we at least don’t crash, however it’s not a “full” fix for these kind of scenarios, because the desired behavior for mutating RRefs is, in my view, poorly specified. In general, I’d strongly recommend to use RRefs as immutable. If you need to modify an RRef you should always be able to extract its value, modify it as you want, and then re-wrap the new version of that value in a new RRef (and stop using the old one). This would be fully safe.