That is, some floating-point input is rounded and cast as an integer in order to be used as an array index

I think the resulting gradient is 0 in that case. I think even if you would do something less “excotic” like predicting some output rounding the logits, and then use it to compute the MSE against a target label, it would give you a gradient of 0.

From what I’ve learned about automatic differentiation in class, it is possible to compute the gradient of functions written with code even when doing overwrites

I am by no means an expert in how autodiff is implemented, but I think I heard somewhere that everything in a computer should be differentiable because you can decompose everything into atomic operations (basically additions). But I am not sure that it’s implemented like that in the autodiff submodule. I have actually no idea how it’s implemented in PyTorch’s submodule, but if I had to speculate, based on what I’ve seen, it’s really manually defined for certain operations.