TheOraware:

There is only slight improvement from fold 0.80588 to 0.80626


In case this value shows the losses, I would consider the improvement rather noise than a real change.
I think your approach looks valid, but the model is clearly not training well, as the loss increases.
This would also mean that the accuracy using the logits output from self.decoder_classifier will most likely be also random, but you could check it nevertheless.