I guess the problem of always predicting the same class is caused by the MobilenetV3. By making Preprocessing False, I normalized the images myself to a range of [-1,1] and the model is working fine.
I tried ModelMaker and downloaded the model from TF Hub,  it makes the process easier a lot. I will probably use it in my next projects. Thank you to everyone who helped.