Hi,
This is not what detach() is for.
You can see detach as being a breakpoint so that no gradient will flow above this point.
In your case, if you detach tmp and tmp1, no gradients will be propagated to embeds1 and embeds2.
In pytorch, the gradients are only computed for the Variables that are created by the user with the parameter requires_grad=True. So no gradient will be computed for the temporary Variables by default.
When using nn, keep in mind that creating an nn.Parameter is the same as a Variable with requires_grad=True.