Since your input has temporal information (i.e, “before” and “after” matters), perhaps CNNs are not the best networks to learn their structure? Have you tried using Recurrent Neural Networks on your data?