title: Using PyTorch nn.Sequential() to define a network in a flexible way but with results beyond expectation
body: <p>I tried to define a network in a more flexible way using <code>nn.Sequential</code> so that I can define its number of layers according to <code>layernum</code>:</p>
  <pre class="lang-py prettyprint-override"><code>seed = 0
  torch.manual_seed(seed)
  # ====== net_a =====
  layers = [ nn.Linear(7, 64), nn.Tanh()]
  for i in range(layernum-1): # layernum = 3
      layers.append(nn.Linear(64, 64))
      layers.append(nn.Tanh())
  layers.append(nn.Linear(64, 8))
  net_x = nn.Sequential(*layers)
  net_y = nn.Sequential(*layers)
  net_z = nn.Sequential(*layers)
  
  # ====== net_b =====
  net_x = nn.Sequential(
      nn.Linear(7, 64),
      nn.Tanh(),
      nn.Linear(64, 64),
      nn.Tanh(),
      nn.Linear(64, 64),
      nn.Tanh(),
      nn.Linear(64, 8),
  )
  net_y = nn.Sequential(
      #... same as net_x
  )
  net_z = nn.Sequential(
      #... same as net_x
  )
  
  # print(net_x)
  # print(net_x[0].weight)
  </code></pre>
  <p>I use both of them <strong>individually</strong>, i.e. they are in the same .py file, but I use either of them and make the other as comments. Both of them consist of 3 networks with respect to 3 dimentions (x,y, and z).</p>
  <p>I expected them to be the same network with same training performance.</p>
  <p>The structure seems to be the same according to <code>print(net_x)</code>:</p>
  <pre class="lang-py prettyprint-override"><code>    # Sequential(
      #   (0): Linear(in_features=7, out_features=64, bias=True)
      #   (1): Tanh()
      #   (2): Linear(in_features=64, out_features=64, bias=True)
      #   (3): Tanh()
      #   (4): Linear(in_features=64, out_features=64, bias=True)
      #   (5): Tanh()
      #   (6): Linear(in_features=64, out_features=8, bias=True)
      # )
  </code></pre>
  <p>But their initial weights are different according to <code>print(net_x[0].weight)</code>:</p>
  <pre class="lang-py prettyprint-override"><code>print(net_x[0].weight) # net_a
      # tensor([[-0.0028,  0.2028, -0.3111, -0.2782, -0.1456,  0.1014, -0.0075],
      #         [ 0.2997, -0.0335,  0.1000, -0.1142, -0.0743, -0.3611, -0.2503],
      #         ......
  
  print(net_x[0].weight) # net_b
      # tensor([[ 0.2813,  0.2968,  0.0078,  0.1518,  0.3776, -0.3247,  0.0071],
      #         [ 0.3448, -0.0988, -0.2798,  0.3347,  0.3581,  0.2229,  0.2841],
      #         ......
  </code></pre>
  <p>======<strong>ADDED</strong>=====</p>
  <p>I trained the network like this:</p>
  <pre class="lang-py prettyprint-override"><code>def train_on_batch(x, y, net, stepsize=innerstepsize):
      x = totorch(x)
      y = totorch(y)
      if(use_cuda):
          x,y = x.cuda(),y.cuda()
      net.zero_grad()
      ypred = net(x)
      loss = (ypred - y).pow(2).mean()
      loss.backward()
      for param in net.parameters():
          param.data -= stepsize * param.grad.data
  
  iteration = 100
  for iter in range(iteration):
  
      # TRAIN
      PrepareSample() # get in_support
      for i in range(tnum_support):
          out_x = trajectory_support_x[i,1:9]
          out_y = trajectory_support_y[i,1:9]
          out_z = trajectory_support_z[i,1:9]
          # Do SGD on this task
          for _ in range(innerepochs): # SGD 1 times
              train_on_batch(in_support[i], out_x, net_x)
              train_on_batch(in_support[i], out_y, net_y)
              train_on_batch(in_support[i], out_z, net_z)
  
      # TEST
      if iter==0 or (iter+1) % 10 == 0:
          ind = [0,1,2,3,4,5,6,7,8,9]
          loss = [0,0,0,0,0,0]
          for i in range(tnum_test):
              inputs = in_test[i]
              outputs_x = trajectory_test_x[i].tolist()
              x_test = trajectory_test_x[i,[0,9]]
              y_test = trajectory_test_x[i,1:9]
              pred_x = np.hstack((x_test[0],predict(inputs, net_x),x_test[1]))
              loss[i] = np.square(predict(inputs, net_x) - y_test).mean() # mse
  
              inputs = in_test[i]
              outputs_y = trajectory_test_y[i].tolist()
              x_test = trajectory_test_y[i,[0,9]]
              y_test = trajectory_test_y[i,1:9]
              pred_y = np.hstack((x_test[0],predict(inputs, net_y),x_test[1]))
              loss[i+2] = np.square(predict(inputs, net_y) - y_test).mean() # mse
  
              inputs = in_test[i]
              outputs_z = trajectory_test_z[i].tolist()
              x_test = trajectory_test_z[i,[0,9]]
              y_test = trajectory_test_z[i,1:9]
              pred_z = np.hstack((x_test[0],predict(inputs, net_z),x_test[1]))
              loss[i+4] = np.square(predict(inputs, net_z) - y_test).mean() # mse
  
          iterNum.append(iter+1)
          avgloss.append(np.mean(loss))
  
  </code></pre>
  <p>both of them are trained with <strong>exactly the same data</strong> (they are in the same .py file and of course use the same data).</p>
  <p>=====<strong>This is avgloss of net_a</strong>:
  <a href="https://i.stack.imgur.com/45GVo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/45GVo.png" alt="enter image description here" /></a></p>
  <p>=====<strong>This is avgloss of net_a with <code>torch.manual_seed(seed)</code> before every network definition</strong>:
  <a href="https://i.stack.imgur.com/C19uT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/C19uT.png" alt="enter image description here" /></a></p>
  <p>=====<strong>This is avgloss of net_b</strong>:
  <a href="https://i.stack.imgur.com/StEmp.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/StEmp.png" alt="enter image description here" /></a></p>
  <p>=====<strong>This is avgloss of net_b with <code>torch.manual_seed(seed)</code> before every network definition</strong>:
  <a href="https://i.stack.imgur.com/IArEn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IArEn.png" alt="enter image description here" /></a>
  The training of <code>net_a</code> is weird, the MSE is high at initial time, and didn't reduce. On the contrary, the training of <code>net_b</code> seems common, the MSE is relatively low at first, and reduce to a smaller value after 100 iterations.</p>
  <p>Is anyone know how to fix this?  I would like to go through different layer number, layer size, activation functions of the network. I don't want to write every network of specified hyper-parameters.</p>