Yeah, basically just tensors registered with the module, so that they will be moved to the host or device and saved in the state_dict in case you would like to serialize your model.
The running estimates of nn.BatchNorm layers are a good example. While they don’t need gradients to be updated, they should still be moved with the layer and saved to disc.