Thanks for the info!
Interesting idea to use the square root.
Just to dig a bit deeper: are you calculating the OA by summing the right predictions (diag of confusion matrix) and dividing it by the sum of all samples?
This might be problematic in an imbalanced setup.