It depends on your use case, if you need to register these tensors as buffers, i.e. if they should be part of the state_dict and be pushed to the specified devices via .to().
Regarding the detaching: yes, you would either repackage/recreate the hidden states, detach them, or keep them attached, as it allows you to implement different use cases and workflows (e.g. automatic detaching of tensors would limit the usability and would disallow certain use cases).