can you add an additional layer to your fully connected layer? like going from 200 -> 2 seems like a big drop. May go from 200 -> 25 (1/8) -> 2 ( ~ 1/12). See if that works for you.
Edit 1: did you use seed in order to replicate your result?
Edit 2: I suggest also using a dropout layer somewhere between your convolution layers in order to regularize