Solved by agt in post #10 
              
              
                I cleaned up and modularized my model and solved the case with multiple samples all having the same output. It had to do with me passing the original inputs into a linear layer instead of the convolved inputs. 
Since all the inputs contain similar sets of elements, except the elements are related in…