I was also thinking about this, and found this issue:

  
      github.com/pytorch/pytorch
  
  
    



Issue: Efficient forward pass in nn.Linear 142


	
	opened by yunjey
	on 2017-07-20
	
	
	closed by soumith
	on 2017-07-20
	


Hi,
Maybe we can improve the operation speed in nn.Linear.
nn.Linear creates a weight matrix of shape (out_features, in_features) as you can see...




  
  
    
    
  
  


From what i understand, transposing in forward pass has no overhead. But backward pass will be less efficient if

input.matmul(weight)