It is hard to debug a training session with only the architecture of the network. Could you share the training code?
How does the loss behave during the training session? Is it decreasing properly? Do you normalize your inputs? Just throwing some ideas…