I would try to use a simpler decoder still.
Also you can try different regularization techniques (batchnorm, weight decay, etc).
Another thing to check is that your training and validation sets are actually properly correlated. Make you split them with samples from each class in each, etc.