Solved by Joshua_Clancy in post #2 
              
              
                Solved it. For people who end up having a similar issue here was the problem: I went over board with my use of convolutions with a (1,1) kernel. While these are useful in the right circumstances, if you layer too many of them on top each other, the network becomes overly parallel and spatial informa…