Answers.Body: <blockquote>
  <p>I added &quot;longitudinal variables&quot; that take into account the number of times the students took the test and their most recent average cumulative score:<br>
  My Question:<br>
  a. Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (better than nothing)<br>
  b. are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)</p>
  </blockquote>
  <p>You actually have a pretty good grasp of what is going on here.
  With regards to your problem there are 2 interactions I see you should keep in mind:</p>
  <p><strong>Predictability and Explain-ability</strong></p>
  <p>you have some set of n factors [x1,x2,...,xn] and one binary label you are trying to predict Pass/Fail, with your positive label being those likely to fail.</p>
  <p>So you are trying to figure out <strong>P(Fail | {x1,x2,...xn})</strong></p>
  <p>You want be able to catch as many students as possible that are likely to fail so that you can help them, but with a model that is too complex, you lose the ability to explain <em>why</em> they needed the help. This can prevent you from helping future students by being proactive and addressing root causes.</p>
  <p>To address your first question:</p>
  <blockquote>
  <p><em>Does the (LV) approach ... sound reasonable (better than nothing)</em></p>
  </blockquote>
  <p>This approach makes a reasonable (common sense) assumption that when it comes to fitness, past performance acts as a good <a href="https://medium.com/analytics-vidhya/introduction-to-bayesian-statistics-for-data-science-and-analytics-part-1-93e38d67fab5" rel="nofollow noreferrer">Bayesian prior</a>.</p>
  <p>To address your second question:</p>
  <blockquote>
  <p>are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)</p>
  </blockquote>
  <p>Decision trees are <a href="https://arxiv.org/pdf/2111.02513.pdf" rel="nofollow noreferrer">not affected by collinearity</a> so they are a great model to address multicollinearity and variance inflation.</p>
  <p>Decision trees also offer the added benefit of <a href="https://aigents.co/data-science-blog/coding-tutorial/machine-learning-explainability-a-hands-on-introduction" rel="nofollow noreferrer"><strong>explainablity</strong></a>, which when it comes to dealing with students helps to mitigate ethical-social issues that might crop up. (why did you help <em>that</em> kid and not <em>my</em> kid)</p>
  <p>Random forests allow you to increase the predictability of the model, but lose some explainability. I would recommend starting with a single decision tree first. However, with random forest you can plot feature importance. <a href="https://www.r-bloggers.com/2021/07/feature-importance-in-random-forest/" rel="nofollow noreferrer">R-doc: feature importance</a></p>
  <p>Be aware that with decision trees it is very easy to <a href="http://saedsayad.com/decision_tree_overfitting.htm" rel="nofollow noreferrer">overfit</a>, one of the main ways this happens is by making a tree that is too deep and allowing too few samples at each split <a href="https://www.rdocumentation.org/packages/parsnip/versions/0.1.6/topics/decision_tree" rel="nofollow noreferrer">R-doc: decision_tree</a>.</p>
  <p>You can <em>start</em> by trying to keep the depth small (3-4), and the min samples not too small (10+) and seeing how far that takes you.</p>
  <p>You can increase the depth from there. Remember that you only have 200 samples (which is small for this kind of problem). So if the minimum number of samples to split is 10 that gives you only 20 nodes before they become a leaf.
  To reach 20 nodes, you only need a depth of 5.</p>
  <p>Hope this helps :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-08T06:13:45.137Z
Answers.Text: @ J.Kent : Thank you for your answer! I was interested in knowing if the general idea behind adding "longitudinal indicator variables" is a mathematically valid approach - if this approach is mathematically valid, then the choice of model is secondary (e.g. I now have the liberty of choosing whichever supervised classification model I want based on my interpretability requirements)! BTW - what do you mean by "LV Approach"? Thank you so much!
Answers.UserDisplayName: stats_noob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-08T06:49:57.223Z
Answers.Text: LV is just short hand for "Longitudinal Variable".  
  
  Even though I know what you *mean*, I wouldn't think of this in terms of being *mathematically* valid. You probably mean it more in the sense of being *statistically* valid, it's probably better to think of this as being *valid modeling approach*. Since machine learning in general is about Bayesian inference **P( y | x)** I would say you are safe.  More specifically, what you are doing is called [**Feature Engineering**](https://medium.com/mindorks/what-is-feature-engineering-for-machine-learning-d8ba3158d97a)
Answers.UserDisplayName: J.Kent
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-08T05:53:25.853Z
Answers.LastActivityDate: 2022-06-08T05:53:25.853Z
Answers.UserDisplayName: J.Kent
Answers.Body: <p>I think there are some things you can do to get it work better.</p>
  <p>Suggestions:</p>
  <ul>
  <li>add a column to the input giving number of previous tries at the test.  If there was a score to go with it, include the average and standard deviation of those, if you can.</li>
  <li>start with 5-fold CV to get a sense of how well it generalizes, and to get a sense of the spread of your losses/fit-performance</li>
  <li>look at the h2o.ai random forest tool because you can get several
  good things out of it that aren't available in all the 'old stuff'.
  The &quot;Flow&quot; interface through the browser gives you good plots and
  fit-analysis.  (They have some very nice speed-ups for random forests in there.)</li>
  </ul>
  <p>I have been &quot;that kid&quot; who took physicals and struggled, and took it again.
  Having an estimated probability of membership can be more useful than a pass-fail because a 51% +/- 1% chance of passing and a 99%  +/- 1% chance in passing can be very different creatures.  If you have tried several times and failed, you might be more likely to fail again.</p>
  <p>The h2o.ai &quot;Flow&quot; interface makes &quot;pretty&quot; tables and graphs with very little effort, and that can be nice for write-ups to give to a boss, or instructor.</p>
  <p>I would not start with a single tree.  You can learn all the wrong things.  It is a Faustian bargain to pay the price of being much more  incorrect to buy the ability to explain those incorrect actions clearly.  There are tools for explaining RF, and in particular, any single outcome of a forest is a weighted average of single branches, so you can get the bounds on the axes that drove the decision and say &quot;because it is in this window, the answer is that&quot;.</p>
  <p>References:</p>
  <ul>
  <li><a href="https://www.usu.edu/math/adele/forests/ENAR.htm#slide0227.htm" rel="nofollow noreferrer">https://www.usu.edu/math/adele/forests/ENAR.htm#slide0227.htm</a></li>
  <li><a href="https://julienbeaulieu.github.io/2019/10/16/model-interpretation-with-random-forests-and-going-beyond-simple-predictions/" rel="nofollow noreferrer">https://julienbeaulieu.github.io/2019/10/16/model-interpretation-with-random-forests-and-going-beyond-simple-predictions/</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T18:40:47.633Z
Answers.Text: @stats_noob - https://www.youtube.com/watch?v=06-AZXmwHjo&t=1990s
Answers.UserDisplayName: EngrStudent
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T14:43:28.537Z
Answers.LastActivityDate: 2022-06-11T14:26:53.963Z
Answers.LastEditDate: 2022-06-11T14:26:53.963Z
Answers.UserDisplayName: EngrStudent
Body: <p>Recently, I had the following question about supervised classification models (e.g. random forest) for longitudinal data.</p>
  <p>Suppose I have the following data about students passing a fitness test - the students (each student has an &quot;id&quot;) who enroll in a school take a fitness test each year and record their height and weight (at the start of each school year, before the fitness test). They can either pass (1) or fail (0) the fitness test each year. The school is interested in knowing which students are likely to fail the fitness test, so they can focus more attention on these students. Naturally, some students might have taken the fitness test more times than other students.</p>
  <p>I simulated some data (using the R programming language) to show how the historical data might look like:</p>
  <pre><code>score &lt;- c(&quot;1&quot;,&quot;0&quot;)
  score &lt;- as.numeric(sample(score, 1000, replace=TRUE, prob=c(0.3, 0.7)))
  id_sample &lt;- 1:140
  id &lt;- sample(id_sample, replace = TRUE, 1000)
  height &lt;- abs(rnorm(1000, 150,5))
  weight &lt;- abs(rnorm(1000, 75,5))
  
  data = data.frame(id, height, weight, score)
  data &lt;- data[order(data$id),]
  </code></pre>
  <p>I then added two variables to this data - one to show how many times the fitness test was taken, the another to show the (cumulative) average number of times the test was passed:</p>
  <pre><code>library(dplyr)
  data =  data.frame(data %&gt;% group_by(id) %&gt;% mutate(counter = row_number(id)))
  data<span class="math-container">$csum &lt;- ave(data$</span>score, data<span class="math-container">$id, FUN=cumsum)
  data$</span>average &lt;- data<span class="math-container">$csum/data$</span>counter
  </code></pre>
  <p>Now, suppose some of the students are about to take this test again and we would like to predict what their score will be - some of these students are existing students, but some of these students are new and have never taken the test before (i.e. they have no historical data):</p>
  <pre><code>id_sample &lt;- 1:140
  id &lt;- sample(id_sample, replace = FALSE, 23)
  height &lt;- abs(rnorm(23, 150,5))
  weight &lt;- abs(rnorm(23, 75,5))
  
  new_data = data.frame(id, height, weight)
  new_data &lt;- new_data[order(new_data$id),]
  
  id_sample &lt;- 141:200
  id &lt;- sample(id_sample, replace = FALSE, 5)
  height &lt;- abs(rnorm(5, 150,5))
  weight &lt;- abs(rnorm(5, 75,5))
  
  #simulating data for students who never took the test before
  n_data = data.frame(id, height, weight)
  n_data &lt;- n_data[order(n_data$id),]
  
  test_data = rbind(new_data, n_data)
  </code></pre>
  <p>Now, to this test data, (where applicable) I added &quot;longitudinal variables&quot; that take into account the number of times the students took the test and their most recent average cumulative score:</p>
  <pre><code>#counter
  max = data.frame(data %&gt;% 
               group_by(id) %&gt;%
               filter(counter == max(counter)))
  
  colnames(max)[5] &lt;- &quot;max_counter&quot;
  
  max<span class="math-container">$max_counter = max$</span>max_counter + 1
  
  test_with_counter =  merge(x = test_data, y = max, by = &quot;id&quot;, all.x = TRUE)
  
  test  = test_with_counter[, c(1,2,3,7,9)]
  
   test<span class="math-container">$max_counter[is.na(test$</span>max_counter)] &lt;- 1
  
   test<span class="math-container">$average[is.na(test$</span>average)] &lt;- 0
  
  #formatting
  colnames(test)[2] &lt;- &quot;height&quot;
  colnames(test)[3] &lt;- &quot;weight&quot;
  colnames(test)[4] &lt;- &quot;counter&quot;
  data<span class="math-container">$csum = NULL
  data$</span>score = as.factor(data$score)
  </code></pre>
  <p>At this point, there is nothing stopping me from training a supervised classification model (e.g. random forest) to predict the &quot;score&quot; variable for the test data:</p>
  <pre><code>#skip cross validation for brevity of question
  library(randomForest)
  rf &lt;- randomForest(score~., data=data)
  pred = predict(rf, newdata = test)
  
  print(rf)
  
  Call:
   randomForest(formula = score ~ ., data = data) 
                 Type of random forest: classification
                       Number of trees: 500
  No. of variables tried at each split: 2
  
          OOB estimate of  error rate: 23.4%
  Confusion matrix:
      0   1 class.error
  0 636  79   0.1104895
  </code></pre>
  <p><strong>My Question:</strong> Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (e.g. better than &quot;nothing&quot;) - or are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.) ? Or is it better to use some supervised classification model/software implementation that has been specifically designed for longitudinal data (e.g. <a href="https://cran.r-project.org/web/packages/LongituRF/LongituRF.pdf" rel="nofollow noreferrer">https://cran.r-project.org/web/packages/LongituRF/LongituRF.pdf</a>)? Thanks!</p>
  <p>Note:</p>
  <ul>
  <li><p>This is a rough sketch of the situation I am dealing with - I am also planning to include variables such as &quot;number of days that elapsed since last fitness test&quot;.</p>
  </li>
  <li><p>The sample data in this stackoveflow question is randomly simulated and obviously wont show any longitudinal trends.</p>
  </li>
  <li><p>I have heard that models such as Random Forest have the ability to recover/model around complex interactions and correlations within the data that otherwise need to be explicitly specified in standard supervised models (<a href="https://ishwaran.org/papers/IKBL.AOAS.pdf" rel="nofollow noreferrer">https://ishwaran.org/papers/IKBL.AOAS.pdf</a>).</p>
  </li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-08T11:50:14.293Z
Comments.Text: Can you please put your **specific question** in the title? "Machine Learning Models for Longitudinal Data" is not a question, but the title of a list.
Comments.UserDisplayName: nbro
Tags: <classification><supervised-learning><random-forests>
Title: Machine Learning Models for Longitudinal Data
UserDisplayName: stats_noob
ThreadText: # Machine Learning Models for Longitudinal Data
  
  (Asked by: username_0 on 2022-06-06)
  
  Recently, I had the following question about supervised classification models (e.g. random forest) for longitudinal data.
  Suppose I have the following data about students passing a fitness test - the students (each student has an "id") who enroll in a school take a fitness test each year and record their height and weight (at the start of each school year, before the fitness test). They can either pass (1) or fail (0) the fitness test each year. The school is interested in knowing which students are likely to fail the fitness test, so they can focus more attention on these students. Naturally, some students might have taken the fitness test more times than other students.
  I simulated some data (using the R programming language) to show how the historical data might look like:
  score <- c("1","0")
  score <- as.numeric(sample(score, 1000, replace=TRUE, prob=c(0.3, 0.7)))
  id_sample <- 1:140
  id <- sample(id_sample, replace = TRUE, 1000)
  height <- abs(rnorm(1000, 150,5))
  weight <- abs(rnorm(1000, 75,5))
  
  data = data.frame(id, height, weight, score)
  data <- data[order(data$id),]
  
  I then added two variables to this data - one to show how many times the fitness test was taken, the another to show the (cumulative) average number of times the test was passed:
  library(dplyr)
  data =  data.frame(data %>% group_by(id) %>% mutate(counter = row_number(id)))
  data$csum <- ave(data$score, data$id, FUN=cumsum)
  data$average <- data$csum/data$counter
  
  Now, suppose some of the students are about to take this test again and we would like to predict what their score will be - some of these students are existing students, but some of these students are new and have never taken the test before (i.e. they have no historical data):
  id_sample <- 1:140
  id <- sample(id_sample, replace = FALSE, 23)
  height <- abs(rnorm(23, 150,5))
  weight <- abs(rnorm(23, 75,5))
  
  new_data = data.frame(id, height, weight)
  new_data <- new_data[order(new_data$id),]
  
  id_sample <- 141:200
  id <- sample(id_sample, replace = FALSE, 5)
  height <- abs(rnorm(5, 150,5))
  weight <- abs(rnorm(5, 75,5))
  
  #simulating data for students who never took the test before
  n_data = data.frame(id, height, weight)
  n_data <- n_data[order(n_data$id),]
  
  test_data = rbind(new_data, n_data)
  
  Now, to this test data, (where applicable) I added "longitudinal variables" that take into account the number of times the students took the test and their most recent average cumulative score:
  #counter
  max = data.frame(data %>% 
               group_by(id) %>%
               filter(counter == max(counter)))
  
  colnames(max)[5] <- "max_counter"
  
  max$max_counter = max$max_counter + 1
  
  test_with_counter =  merge(x = test_data, y = max, by = "id", all.x = TRUE)
  
  test  = test_with_counter[, c(1,2,3,7,9)]
  
   test$max_counter[is.na(test$max_counter)] <- 1
  
   test$average[is.na(test$average)] <- 0
  
  #formatting
  colnames(test)[2] <- "height"
  colnames(test)[3] <- "weight"
  colnames(test)[4] <- "counter"
  data$csum = NULL
  data$score = as.factor(data$score)
  
  At this point, there is nothing stopping me from training a supervised classification model (e.g. random forest) to predict the "score" variable for the test data:
  #skip cross validation for brevity of question
  library(randomForest)
  rf <- randomForest(score~., data=data)
  pred = predict(rf, newdata = test)
  
  print(rf)
  
  Call:
   randomForest(formula = score ~ ., data = data) 
                 Type of random forest: classification
                       Number of trees: 500
  No. of variables tried at each split: 2
  
          OOB estimate of  error rate: 23.4%
  Confusion matrix:
      0   1 class.error
  0 636  79   0.1104895
  
  My Question: Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (e.g. better than "nothing") - or are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.) ? Or is it better to use some supervised classification model/software implementation that has been specifically designed for longitudinal data (e.g. https://cran.r-project.org/web/packages/LongituRF/LongituRF.pdf)? Thanks!
  Note:
  
  This is a rough sketch of the situation I am dealing with - I am also planning to include variables such as "number of days that elapsed since last fitness test".
  
  The sample data in this stackoveflow question is randomly simulated and obviously wont show any longitudinal trends.
  
  I have heard that models such as Random Forest have the ability to recover/model around complex interactions and correlations within the data that otherwise need to be explicitly specified in standard supervised models (https://ishwaran.org/papers/IKBL.AOAS.pdf).
  
  **Tags:** classification, supervised-learning, random-forests
  > **Comments:**
  > **username_1**: Can you please put your **specific question** in the title? "Machine Learning Models for Longitudinal Data" is not a question, but the title of a list.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-06-08. Score: 2
  
  > I added "longitudinal variables" that take into account the number of times the students took the test and their most recent average cumulative score:  
  > My Question:  
  > a. Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (better than nothing)  
  > b. are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)
  
  You actually have a pretty good grasp of what is going on here.
  With regards to your problem there are 2 interactions I see you should keep in mind:
  
  **Predictability and Explain-ability**
  
  you have some set of n factors [x1,x2,...,xn] and one binary label you are trying to predict Pass/Fail, with your positive label being those likely to fail.
  
  So you are trying to figure out **P(Fail | {x1,x2,...xn})**
  
  You want be able to catch as many students as possible that are likely to fail so that you can help them, but with a model that is too complex, you lose the ability to explain *why* they needed the help. This can prevent you from helping future students by being proactive and addressing root causes.
  
  To address your first question:
  
  > *Does the (LV) approach ... sound reasonable (better than nothing)*
  
  This approach makes a reasonable (common sense) assumption that when it comes to fitness, past performance acts as a good [Bayesian prior](https://medium.com/analytics-vidhya/introduction-to-bayesian-statistics-for-data-science-and-analytics-part-1-93e38d67fab5).
  
  To address your second question:
  
  > are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)
  
  Decision trees are [not affected by collinearity](https://arxiv.org/pdf/2111.02513.pdf) so they are a great model to address multicollinearity and variance inflation.
  
  Decision trees also offer the added benefit of [**explainablity**](https://aigents.co/data-science-blog/coding-tutorial/machine-learning-explainability-a-hands-on-introduction), which when it comes to dealing with students helps to mitigate ethical-social issues that might crop up. (why did you help *that* kid and not *my* kid)
  
  Random forests allow you to increase the predictability of the model, but lose some explainability. I would recommend starting with a single decision tree first. However, with random forest you can plot feature importance. [R-doc: feature importance](https://www.r-bloggers.com/2021/07/feature-importance-in-random-forest/)
  
  Be aware that with decision trees it is very easy to [overfit](http://saedsayad.com/decision_tree_overfitting.htm), one of the main ways this happens is by making a tree that is too deep and allowing too few samples at each split [R-doc: decision\_tree](https://www.rdocumentation.org/packages/parsnip/versions/0.1.6/topics/decision_tree).
  
  You can *start* by trying to keep the depth small (3-4), and the min samples not too small (10+) and seeing how far that takes you.
  
  You can increase the depth from there. Remember that you only have 200 samples (which is small for this kind of problem). So if the minimum number of samples to split is 10 that gives you only 20 nodes before they become a leaf.
  To reach 20 nodes, you only need a depth of 5.
  
  Hope this helps :)
  
  > **Comments:**
  > **username_0**: @ J.Kent : Thank you for your answer! I was interested in knowing if the general idea behind adding "longitudinal indicator variables" is a mathematically valid approach - if this approach is mathematically valid, then the choice of model is secondary (e.g. I now have the liberty of choosing whichever supervised classification model I want based on my interpretability requirements)! BTW - what do you mean by "LV Approach"? Thank you so much!  
  > **username_2**: LV is just short hand for "Longitudinal Variable".    Even though I know what you *mean*, I wouldn't think of this in terms of being *mathematically* valid. You probably mean it more in the sense of being *statistically* valid, it's probably better to think of this as being *valid modeling approach*. Since machine learning in general is about Bayesian inference **P( y | x)** I would say you are safe.  More specifically, what you are doing is called [**Feature Engineering**](https://medium.com/mindorks/what-is-feature-engineering-for-machine-learning-d8ba3158d97a)  
  
  ---
  
  ### Answer by username_3 on 2022-06-10. Score: 1
  
  I think there are some things you can do to get it work better.
  
  Suggestions:
  
  * add a column to the input giving number of previous tries at the test. If there was a score to go with it, include the average and standard deviation of those, if you can.
  * start with 5-fold CV to get a sense of how well it generalizes, and to get a sense of the spread of your losses/fit-performance
  * look at the h2o.ai random forest tool because you can get several
    good things out of it that aren't available in all the 'old stuff'.
    The "Flow" interface through the browser gives you good plots and
    fit-analysis. (They have some very nice speed-ups for random forests in there.)
  
  I have been "that kid" who took physicals and struggled, and took it again.
  Having an estimated probability of membership can be more useful than a pass-fail because a 51% +/- 1% chance of passing and a 99% +/- 1% chance in passing can be very different creatures. If you have tried several times and failed, you might be more likely to fail again.
  
  The h2o.ai "Flow" interface makes "pretty" tables and graphs with very little effort, and that can be nice for write-ups to give to a boss, or instructor.
  
  I would not start with a single tree. You can learn all the wrong things. It is a Faustian bargain to pay the price of being much more incorrect to buy the ability to explain those incorrect actions clearly. There are tools for explaining RF, and in particular, any single outcome of a forest is a weighted average of single branches, so you can get the bounds on the axes that drove the decision and say "because it is in this window, the answer is that".
  
  References:
  
  * <https://www.usu.edu/math/adele/forests/ENAR.htm#slide0227.htm>
  * <https://julienbeaulieu.github.io/2019/10/16/model-interpretation-with-random-forests-and-going-beyond-simple-predictions/>
  
  > **Comments:**
  > **username_3**: @stats_noob - https://www.youtube.com/watch?v=06-AZXmwHjo&t=1990s  
  
  ---
  
  