Solved by snakers41 in post #3 
              
              
                Yeah, I thought about this. But in our particular case this would make model distribution much more complicated (or it would negate the effects of factorization). 
But I fould an elegant solution. I forgot that bias=True is not mandatory in PyTorch. You can just do this and it solves the problem. Yo…