Answers.Body: <p>The concern with your approach is called 'leakage' because you are parameterising the same data set that you are training. This can easily lead to overtraining is in ANN (artificial neural networks) and is not cool. Overtraining is where the ANN very tightly wraps around the training data set that it performs poorly on any other data set. In other words it is overtightened.</p>
  <p>The way they get around it is to split the data set usually 60:20:20 into a training split, parameterisation (validation) split and test (testing) split. If the parameterisation split gets mixed with the training set - leakage happens and overtraining results. Normally the data is trained, and the trained model parameters are assessed/optimised in the parameter split. The trained data and optimised parameters are then tested in the test split.</p>
  <p>As long as you are aware of the approach you are using, that I guess that is ok. If you separate your training and parameterisation (validation) data that would be cool.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-23T01:09:06.760Z
Answers.LastActivityDate: 2020-06-23T19:08:05.120Z
Answers.LastEditDate: 2020-06-23T19:08:05.120Z
Answers.UserDisplayName: M__
Body: <p>I have a script in R/RStudio which creates random datasets of binomial variables, feeds them through a neural network, and calculates their likelihood ratio statistic and deviance. I'd like the script to loop with the <code>seed</code> incremented by 1, and collect the value from each run into a data frame, from which the values get averaged. For example, if the script returns <code>5, 10, and 15</code> on the first, second, and third iterations of the loop, I'd like the script to create a data frame with those values and then calculate their mean.</p>
  <p>Here is my script, annotated:</p>
  <pre><code>library(neuralnet)
  
  x=1021
  
  #change nrow to number of times code is looped
  result &lt;- data.frame(matrix(nrow = 1000, ncol = 2)) 
  colnames(result) &lt;- c(&quot;LogL&quot;, &quot;D&quot;)
  
  repeat{
  
  set.seed(x)
  
  #2 inputs, max number of terms 3
  x1=rbinom(2000,1,0.5)
  x2=rbinom(2000,1,0.5)
  y=rbinom(2000,1,0.5)
  
  data=data.frame(x1,x2,y)
  
  head(data)
  
  #hidden units = 2
  nnet=neuralnet(y~x1+x2,data,hidden=2,err.fct = &quot;ce&quot;,linear.output = F,likelihood = T)
  
  predictions=compute(nnet,data[,1:2])
  
  predictions$net.result
  
  #likelihood ratio stat
  argument=y*log(predictions<span class="math-container">$net.result)+(1-y)*log(1-predictions$</span>net.result)
  
  head(argument)
  
  #sum of likelihood ratio stat
  LogL=sum(argument)
  
  LogL
  
  D=-2*LogL
  
  D
  
  result[x-1022, 1] &lt;- LogL
  result[x-1022, 2] &lt;- D
  
  result
  
  #results &lt;- data.frame(LogL, D)
  
  x=x+1
  if (x==2021) {
    break
  }
  
  print(x)
  
  
  }
  
  result
  </code></pre>
  <p>My repeat loop works, but a thousand iterations, for example, takes multiple minutes. Is there a more efficient way to code such a loop?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-22T23:01:23.177Z
Comments.Text: Your `repeat` termination conditions will never be satisfied. Every round, x will be set to 1021, incremented by 1 and compared to 1023, which will never be true. How will the repeat ever stop?
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-22T23:26:41.513Z
Comments.Text: There are other approaches to parameterisation in ANN. @RamRS question on terminating recursion is definitely a good question.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-22T23:44:49.673Z
Comments.Text: Thanks, edited that and it works now, but it's super slow. I'll update the question!
Comments.UserDisplayName: Equinox
Tags: <r><machine-learning><loop>
Title: How do I set a neural network to loop multiple times and average the resulting values?
ThreadText: # How do I set a neural network to loop multiple times and average the resulting values?
  
  (Asked by: username_0 on 2020-06-22)
  
  I have a script in R/RStudio which creates random datasets of binomial variables, feeds them through a neural network, and calculates their likelihood ratio statistic and deviance. I'd like the script to loop with the `seed` incremented by 1, and collect the value from each run into a data frame, from which the values get averaged. For example, if the script returns `5, 10, and 15` on the first, second, and third iterations of the loop, I'd like the script to create a data frame with those values and then calculate their mean.
  
  Here is my script, annotated:
  
  ```
  library(neuralnet)
  
  x=1021
  
  #change nrow to number of times code is looped
  result <- data.frame(matrix(nrow = 1000, ncol = 2)) 
  colnames(result) <- c("LogL", "D")
  
  repeat{
  
  set.seed(x)
  
  #2 inputs, max number of terms 3
  x1=rbinom(2000,1,0.5)
  x2=rbinom(2000,1,0.5)
  y=rbinom(2000,1,0.5)
  
  data=data.frame(x1,x2,y)
  
  head(data)
  
  #hidden units = 2
  nnet=neuralnet(y~x1+x2,data,hidden=2,err.fct = "ce",linear.output = F,likelihood = T)
  
  predictions=compute(nnet,data[,1:2])
  
  predictions$net.result
  
  #likelihood ratio stat
  argument=y*log(predictions$net.result)+(1-y)*log(1-predictions$net.result)
  
  head(argument)
  
  #sum of likelihood ratio stat
  LogL=sum(argument)
  
  LogL
  
  D=-2*LogL
  
  D
  
  result[x-1022, 1] <- LogL
  result[x-1022, 2] <- D
  
  result
  
  #results <- data.frame(LogL, D)
  
  x=x+1
  if (x==2021) {
    break
  }
  
  print(x)
  
  }
  
  result
  
  ```
  
  My repeat loop works, but a thousand iterations, for example, takes multiple minutes. Is there a more efficient way to code such a loop?
  
  **Tags:** r, machine-learning, loop
  
  > **Comments:**
  > **username_1**: Your `repeat` termination conditions will never be satisfied. Every round, x will be set to 1021, incremented by 1 and compared to 1023, which will never be true. How will the repeat ever stop?  
  > **username_2**: There are other approaches to parameterisation in ANN. @username_1 question on terminating recursion is definitely a good question.  
  > **username_0**: Thanks, edited that and it works now, but it's super slow. I'll update the question!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-23. Score: 1
  
  The concern with your approach is called 'leakage' because you are parameterising the same data set that you are training. This can easily lead to overtraining is in ANN (artificial neural networks) and is not cool. Overtraining is where the ANN very tightly wraps around the training data set that it performs poorly on any other data set. In other words it is overtightened.
  
  The way they get around it is to split the data set usually 60:20:20 into a training split, parameterisation (validation) split and test (testing) split. If the parameterisation split gets mixed with the training set - leakage happens and overtraining results. Normally the data is trained, and the trained model parameters are assessed/optimised in the parameter split. The trained data and optimised parameters are then tested in the test split.
  
  As long as you are aware of the approach you are using, that I guess that is ok. If you separate your training and parameterisation (validation) data that would be cool.
  
  
  ---
  
  