So I am trying to use pytorch for a non-neural net application for taking local pose estimation to global pose estimation. I am trying to use a custom loss function to compute the difference between a global 3d xyz coordinates and 2d uv image pixel coordinates. My question is do gradients persist when you use retain_grad but pass the tensor into a new tensor initialization?
I’m using a kinematic hand model for pose estimation so I am trying to find the gradients of the yaw/pitch/roll of the rotation matrix of the model wrist, translation matrix of the wrist, and flexion/abduction angles of the fingers after forward kinematics on the hand model. I randomly initialized the R, p, theta values and am using ground truth values of the image coordinates to perform gradient descent
yaw_w   = torch.tensor([0]).float()
pitch_w = torch.tensor([0]).float()
roll_w  = torch.tensor([0]).float()

p_wrist = torch.tensor([0., 50., 350.])
theta = torch.tensor([[0., 0., 0., 0.],
                      [0., 0., 0., 0.],
                      [0., 0., 0., 0.],
                      [0., 0., 0., 0.],
                      [0., 0., 0., 0.]])

def R_from_angs(yaw, pitch, roll):
    Rx = torch.tensor([[1, 0, 0],
                       [0, torch.cos(roll), -torch.sin(roll)],
                       [0, torch.sin(roll), torch.cos(roll)]]).float()

    Ry = torch.tensor([[torch.cos(pitch), 0, torch.sin(pitch)],
                       [0, 1, 0],
                       [-torch.sin(pitch), 0, torch.cos(pitch)]]).float()

    Rz = torch.tensor([[torch.cos(yaw), -torch.sin(yaw), 0],
                       [torch.sin(yaw), torch.cos(yaw), 0],
                       [0, 0, 1]]).float()
    R = torch.mm(torch.mm(Rz, Ry), Rx)
    return R

def loss_2d(xyz, uv_pred, c_scores):
    # projecting into image plane using camera intrinsics
    uv_proj = perspective_proj(xyz, torch.as_tensor(ci).float()) 
    uv_diff = uv_proj - torch.as_tensor(uv_pred).float()
    uv_norm = torch.norm(uv_diff, p=2, dim=1).view(21,1) ** 2
    loss = torch.sum(c_scores*uv_norm)
    return loss

yaw = torch.tensor(yaw_w, requires_grad=True)
pitch = torch.tensor(pitch_w, requires_grad=True)
roll = torch.tensor(roll_w, requires_grad=True)
p = torch.tensor(p_wrist, requires_grad=True)
theta = torch.tensor(theta, requires_grad=True)

R_wrist = R_from_angs(yaw, pitch, roll)

uv_pred, conf_score, xyz_pred, viz_crop = caffe_predictions(img)
# forward kinematics from hand Rotation, translation, and finger theta angles
xyz_fk_init = hand_fk(R_wrist, p, torch.from_numpy(avg_bone_length).float(), parent_unit_vec,theta)

loss = loss2d(xyz_fk_init, uv_pred, torch.as_tensor(conf_score).float())

When I perform backpropagation on the custom loss function above I lose the gradients for yaw/pitch/roll. I think it has to do with the fact that I am creating new tensors to create the transformation matrix. Any idea?
yaw.retain_grad()
pitch.retain_grad()
roll.retain_grad()
p.retain_grad()
theta.retain_grad()

loss.backward()
print(p.grad) #does return a value
print(pitch.grad) #gives me nothing

I know this is quite a wall of text so I really appreciate you going through and reading it!