I see. You could try to check the source code of fillpoly and see, if you could either port it directly to PyTorch, so that Autograd would create the backward pass automatically for you, implement the backward function manually (if possible at all), or alternatively let your model output the mask directly (might be the easiest approach).