ypxie:

Will this code restrain the shard_model grad only being bounded with one local_model?
Cause share_model.grad will not be None after running this function for once.  And other threads of local_model won’t be able to change _grad anymore. Or _grad will not be accesible to other threads?


It seems like share_memory does not share the gradients, so shared_model.grad will still be None for other processes even it has been run once on one process.
@apaszke, I wonder if I understand this correctly?