I think I found out, what the problem was. However, it’s incredible that this could happen without even a warning being printed.
It seems, pytorch has a problem when multiplying two slices of tensors with different datatype. If I call .contiguous() on the tensors before multiplying, everything works fine. Minimal example: https://gist.github.com/koelscha/4dc9255a83e39b36b1cf20afa3c6fa74 2
I would expect to at least see a warning that this causes problems, since it works fine in numpy