for now I have attempted at solving my problem by making the un-necessary weights as 0 after every step of gradient descent using a mask matrix with 1’s at places with variables we want to compute the gradients for and 0’s in the places which we want to ignore gradient computation by element wise multiplying the weight matrix with it . it seems to have solved my problem but yet I don’t know if its accurate to do it this way .