Answers.Body: <ol>
  <li><p>The seeds themselves were randomly sampled. So each time I run the script, I get a different value of the average F1 score.
  Now I am not sampling the seeds each time. I have fixed those values.</p>
  </li>
  <li><p>The high variance is because of the model's sensitivity to the training data. In PU-Learning, we sparsely label(~1%) the positive data for training. Especially in the case of graph datasets, the model performance is sensitive to which nodes have been labelled.</p>
  </li>
  </ol>
  <p>So, variance is at least fixed for the given set of seeds. Now I get identical values each time I run the script.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-25T16:48:45.653Z
Answers.LastActivityDate: 2023-01-26T03:56:09.253Z
Answers.LastEditDate: 2023-01-26T03:56:09.253Z
Answers.UserDisplayName: willtryagain
Body: <p>I am training GAT using a custom loss function(<a href="https://arxiv.org/abs/1703.00593" rel="nofollow noreferrer">PU Loss</a>) on the Cora and Citeseer dataset. My training file looks like</p>
  <pre><code>f1_scores = []  
  N_ITER = 10
  seeds = np.random.randint(1000, size=N_ITER) 
  
  for i in range(N_ITER):
      seed_value = seeds[i]
      np.random.seed(seed_value)
      random.seed(None)
      torch.manual_seed(seed_value)
      model = GAT().to(device)
      # train it 
      # find f1 score
      f1_scores.append(f1)
  
  print(np.mean(f1_scores))
  </code></pre>
  <p>When I run this file multiple times by doing</p>
  <pre><code> for i in `seq 1 10`; do python train.py; done
  </code></pre>
  <p>I am getting high variance in the values (for e.g 0.43 and 0.76). I don't understand why this is happening even after taking the mean.</p>
  <ol>
  <li>Is this the right way to take the mean of the model's F1 scores?</li>
  <li>How to reduce this variance?</li>
  </ol>
  <p>I have followed the steps mentioned <a href="https://stackoverflow.com/a/64544325/10503059">here</a>. I must use a NN. I increased the weight decay (L2) values without any success.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-22T16:31:46.030Z
Comments.Text: I don't know about PU-learning, but from glimpsing into the introduction of the paper you provided it sounds like this loss is designed for binary classification. Cora/Citeseer contain 6/7 classes and are not binary classification tasks. Did you account for this in your script?
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-22T17:13:25.087Z
Comments.Text: Yes, I chose one class as positive and the rest as negative, as described [here](https://arxiv.org/abs/2103.04683).
Comments.UserDisplayName: willtryagain
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T16:47:53.723Z
Comments.Text: Hmm, okay then I have no clue but I also don't know about the PU training. Sadly, I currently have no time to read the paper you provide. Maybe you could give a brief overview on that method to make it less effort for the community to provide ideas.
Comments.UserDisplayName: Chillston
Tags: <graph-neural-networks><variance-reduction>
Title: How to reduce variance in F1 scores of GAT across multiple runs while using PU Loss?
UserDisplayName: willtryagain
ThreadText: # How to reduce variance in F1 scores of GAT across multiple runs while using PU Loss?
  
  (Asked by: username_0 on 2023-01-22)
  
  I am training GAT using a custom loss function([PU Loss](https://arxiv.org/abs/1703.00593)) on the Cora and Citeseer dataset. My training file looks like
  
  ```
  f1_scores = []  
  N_ITER = 10
  seeds = np.random.randint(1000, size=N_ITER) 
  
  for i in range(N_ITER):
      seed_value = seeds[i]
      np.random.seed(seed_value)
      random.seed(None)
      torch.manual_seed(seed_value)
      model = GAT().to(device)
      # train it 
      # find f1 score
      f1_scores.append(f1)
  
  print(np.mean(f1_scores))
  
  ```
  
  When I run this file multiple times by doing
  
  ```
   for i in `seq 1 10`; do python train.py; done
  
  ```
  
  I am getting high variance in the values (for e.g 0.43 and 0.76). I don't understand why this is happening even after taking the mean.
  
  1. Is this the right way to take the mean of the model's F1 scores?
  2. How to reduce this variance?
  
  I have followed the steps mentioned [here](https://stackoverflow.com/a/64544325/10503059). I must use a NN. I increased the weight decay (L2) values without any success.
  
  **Tags:** graph-neural-networks, variance-reduction
  
  > **Comments:**
  > **username_1**: I don't know about PU-learning, but from glimpsing into the introduction of the paper you provided it sounds like this loss is designed for binary classification. Cora/Citeseer contain 6/7 classes and are not binary classification tasks. Did you account for this in your script?  
  > **username_0**: Yes, I chose one class as positive and the rest as negative, as described [here](https://arxiv.org/abs/2103.04683).  
  > **username_1**: Hmm, okay then I have no clue but I also don't know about the PU training. Sadly, I currently have no time to read the paper you provide. Maybe you could give a brief overview on that method to make it less effort for the community to provide ideas.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-01-25. Score: 0 | Accepted answer
  
  1. The seeds themselves were randomly sampled. So each time I run the script, I get a different value of the average F1 score.
     Now I am not sampling the seeds each time. I have fixed those values.
  2. The high variance is because of the model's sensitivity to the training data. In PU-Learning, we sparsely label(~1%) the positive data for training. Especially in the case of graph datasets, the model performance is sensitive to which nodes have been labelled.
  
  So, variance is at least fixed for the given set of seeds. Now I get identical values each time I run the script.
  
  
  ---
  
  