Hello,
I am looking into the superresolution example and printed out the weights of the second convolution layer. It turns out these are ‘kinda weird’. So I looked into them and found that the orthogonal weight initialization does not initialize a large section of the weights of a 4 dimensional matrix. Yes, I know that the documentation states that ‘dimensions beyond 2’ are flattened. Does not mean though that the values of a large portion of the matrix should be empty.
Picture at http://werner.yellowcouch.org/log/orthogonal-weight-initialisation-in-pytorch-seems-kinda-weird/ 30