Solved by albanD in post #6 
              
              
                You should keep in mind that the forward method in pytorch is actually executing this code every time you do a forward with a given input. 
So setting the fc models to identity can be done only once in the __init__ as well. 
The cat operation takes Tensors as input, not layers. 
I think you meant so…