I would recommend to start with creating the right loss function first even if it’s slow in the first version.
I.e. you shouldn’t care too much about nested loop in the first iteration and make sure the outputs and gradients are expected.
Once this is done you can use this slow method as the baseline and try to use vectorized operations to speed it up.