tf.convert_to_tensor(onehot_encoded[label.numpy()])

The problem is actually in this part of the return statement. If I simply keep the labels as categorical numbers, it works, but then I will be having problems with softmax