I think you would need to be more specific in your problem setting, maybe you can enhance your code snippet to be self-contained (i.e. runnable), maybe even with just 1 leaf.
Also, if we think of the backward as a functional that maps dloss/doutput to dloss/dinput, you would need to say what output has a gradient and what input should have one, and maybe something about your loss. Indexing itself is discrete and thus cannot produce meaningful gradients, but obviously, people have found ways to formulate losses on coordinates (e.g. Intersection over Union) to be able to deal with the need to predict shapes. Very likely the solution involves some way of talking more about coordinates and less about pixels.
Best regards
Thomas