Solved it. For people who end up having a similar issue here was the problem: I went over board with my use of convolutions with a (1,1) kernel. While these are useful in the right circumstances, if you layer too many of them on top each other, the network becomes overly parallel and spatial information is lost. Most people shouldn’t have this problem unless they are making their own weird networks.