You can use self.register_buffer(param_name, initial_value) to add these tensors in the model’s state dict.
This snippet shows how to do it:
def __init__(self):
    super().__init__()
    ...
    self.running_means = {}
        for i in range(100):
            value = torch.randn(5)
            self.running_means[f'running_mean{i}'] = value
            self.register_buffer(f'running_mean{i}', value)
    ...