Solved by tom in post #3 
              
              
                There are two parts: 

As Avinash points out, the default mode 'fan_in' is probably a good choice.
For some intuition of why this is: Each output is a weighted sum of fan_in inputs. For linear, this is one row of the matrix multiplication, for convolutions it is number of in-channels  * kernel size.
…