Solved by smth in post #2 
              
              
                this is exactly what you are supposed to be doing. shoehorning it into nn.Sequential is not a great idea, Sequential is meant to be a very simple container that’s dumb.