Hello!
Is there a way to optimiize only a certain part of a tensor?
I am truing to use Pythorch as a backend for ptychographycal engine, which requires to optimize  large [2^12,2^12,2] tensor O. But I can formulate loss function only for set of the overlaping slices of this big tensor O[up:down,left:right,:] so I need to calculate updates for each of these slices consistently. Currently, I just iterate through list of these slices and use optimizer with respect to the whole big tensor which  require to calclate gradient for the whole O each step which takes most of the time, despite I know that only small part of it O[up:down,left:right,:]  influence on the loss function and should be updated this step.
So if there any way to turn on gradient calculation only  for some part of O  each step, or are there any other ways to solve this problem?