We would likely want to think about what the derivative should be mathematically before we look at the implementation. What should happen to y_pred relative to loss? You write that no interpolation is possible, but so if y_pred had some gradient that causes the rounding to decrement idx by one from one step to the other, then you end up with y_true[old_idx[:, 0] - 1, old_idx[:, 1]. This would suggest that if that is smaller than that y_pred[:, 0] should have a positive gradient. (Leaving aside the batch thing…)
One variant where you try to avoid needing to differentiating y_pred is the REINFORCE algorithm in RL. Essentially, when y_pred is a sample from a probability distribution, you can still say the probability of y_pred should go up if the loss is small (“success”) or down if it is large (“failure”).