Based on the screenshot you shared, your training accuracy and confusion matrix looks good.
What do you mean by “tried to do inference on my training set and it gave me bad results”?
Did you switch to eval and the results were also bad on the training set?
If your batch size isn’t small, it’s not problematic to use BatchNorm layers. The opposite could be problematic.