I see your point. One last question, Are you shuffling data in validation and test sets? If the answer is yes, this can be the problem. As I said previously without more information about your network, and training code it is not possible to me to guess what is the problem.