This is closer, but did not work.
Reason is that dout_dp should be detached from the computation graph of out when connected to a and vice versa. (Mathematically its possible, we have all the information).
Let me illustrate further with a code example:
import torch
m = 2
n = 3
batch = 4

a = torch.ones(batch, m)
a.retain_grad()
layer = torch.nn.Linear(m, n, bias=False)
layer.weight.data.copy_(torch.ones(n, m))

dout_dp = torch.ones(n, m)*5  # A **given** dout_dp, detached from `out` and `a`.
layer.weight.grad = dout_dp
v2 = layer(a)   # reconstruct a computation graph a->p

# Problem: `p.grad` is detached from the reconstructed computation graph of `a`
# so the backward below not going to work
torch.autograd.backward(list(layer.parameters()), 
                        [p.grad for p in layer.parameters()])
print(a.grad)  # None

I think that doing p.backward(p.grad) is an advancement and probably part of the future solution, but we are lacking a connection to the gradient, which must to be added artificially somehow:
How do we tell pytorch that a has connection to p.grad (and figure that connection automatically from the new computation graph a->p)?
I see you did there p.retain_grad(), I guess this saves the “connection” from a->p->out which is what we want to avoid,
as we want to start “clean” from the gradient and restored computation graph “a->p”.
From what I understand, it seems like autograd does not “use” parameter gradients (like p.grad) directly in its computation, but only AccumulateGradients to them.