Hello Nio,
I am not sure if I understand the problem, but you can try to create two tensors, one with the variables that you need to optimize and the other with the other set of variables, and when using the vector concatenate those two tensors.
I don’t know if that applicable to your problem, but maybe it could help.