ah i see. so if i get this correct that the training accuracy seems to be improving while the validation accuracy is getting worse. so it would seem like a standard overfitting problem. so is the training set or the validation set too small. if training is too small, then it directly implies not enough samples to learn from, if validation is too small, then you might end up with a val set of difficult samples that do not reflect how good or bad the network is.
however i’ll assume that when you say validation and train sets are fine, that this might not be the case. if so then it might be a coding issue, like dissimilar transforms on train and val etc. to rule this out i would suggest using the train set as the train as well as the val set. if the code is fine then you would see what is expected, improved accuracy from the train loss and the validation part of your code, else you will see a mismatch.
thats all i could think of at the moment