@theairbend3r
Thanks for the tips and the recommendations,  my training error is also not increasing. Basically, around 20 epochs, i have the following performances and it will never go beyond:

Epoch 0 | Train loss 1.587 | Train acc 0.263 |  Valid loss 1.575 | Valid acc 0.280
Epoch 27 | Train loss 1.073 | Train acc 0.553 |  Valid loss 1.596 | Valid acc 0.357

I tried to change the HP but nothing help.
@vdw thanks for the solutions, I have done the changes but performances stay the same. I’ll dig into that, maybe my data are in the wrong shape.