Well the only way to actually help u now is to reimplement ur network architecture
And see things for myself coz currently I’m not able to spot any thing that looks wrong here.
I’ll probably get back to u tomorrow.