Solved by Zeeyuu in post #2 
              
              
                Okay it seems what I saved in the list is memory location instead of the weights values. If I clone the initial weights they will no longer be the same.