Thanks for your reply. I tried now, but the result is the same.
Here is the modified part:
self.foo = torch.tensor(2018, device=self.weight.device)

Because A is a layer, I use self.weight.
It seems to me that inside the method of the module class, the buffer variable can be assigned correctly, but after that, it is reinitialized, or maybe not modified at all in the method of the class.
I also print the buffer variable outside the code file which contains the definition of class is as following:
for m in model.modules():
    if isinstance(m, A):
        print('c: ', m.foo.item)

and will output the random large number, instead of 2018.
With single GPU, I do not need to use the buffer, and there is no such problem.