Solved by SimonW in post #2 
              
              
                Your new network has a new set of parameters. You need to re-create the optimizer too because the old optimizer isn’t tracking your new parameters.