I don’t agree with you on this, because it would mean, that pytorch has to know the expected behavior. This would violate pytorch’s concept of allowing the user to create highly flexible networks. One could enforce the input of a layer to be pushed to the correct device, but that would hide other (potentially more critical) errors, since usually wrong devices are an indicator for some wrong graph definition.