Thank you, single underscores do indeed seem to fix it.
But with double underscores the variable would still be accessible through my_module._MyModule__my_parameter.grad. However, in the example above, printing my_module._MyModule__my_parameter.grad leads to exactly the same result as just calling my_module.my_parameter.grad.
Does the name mangling impact autograd?