pytorcher:

It sounded like doing register_buffer allows you to define “self” variables that ensure that they work properly on multiple GPUs. Did I misread his post about that?


It does, but depends what you want to do.
Having it being a buffer means that the replication code will properly replicate it and move it to the right GPU for this particular replica. So you will be able to use it properly with the inputs from this Module.
But that does not mean that it will be shared between the replicas!