Body: <p>I have encountered this problem on how to predict the probability of a periodically happening event occurring at a given time.</p>
  <p>For example, we have an event called <em><strong>being_an_undergrad</strong></em>. There are many data points: bob is an undergrad from (1999 - 2003), Bill is an undergrad from (1900 - 1903), Alice is an undergrad from (1900 - 1905), and there are many other data points such as (2010 - 2015), (2011 - 2013) ....</p>
  <p>There are many events(data points) of <em><strong>being_an_undergrad</strong></em>. The lasting interval varies, it might be 1 year, 2 years, 3 years, .... or even 10 years. But the majority is around 4 years.</p>
  <p>However, I am wondering given all the data points above. If I now know that Jason starts college in 2021, and how can I calculate/predict the probability that he will still be an undergrad in 2022? and 2023? and 2024 .... 2028, etc.</p>
  <p>My current dataset consists of 10000 tuples representing events of different relations. The relations are all continuous relations similar to the example above. There are about 10 continuous relations in total in this dataset, such as <code>isMarriedTo</code>, <code>beingUndergrad</code>, <code>livesIn</code>, etc. For each relation, there are about 1000 data points(1000 durations) about this relation, for example,</p>
  <pre><code>&lt;Leo, isUndergrad, Harvard, 2010 - 2011&gt;, &lt;Leo, isUndergrad, Stanford, 2013 - 2016&gt;.....
  
  &lt;Jason, livesIn, US, 1990 - 2021&gt;, &lt;Richard, livesIn, UK, 1899- 1995&gt; ...
  </code></pre>
  <p>My problem now is that I want to get a confidence level(probability) when I want to predict one event happening at a specific time point. For example, I want to predict the probability that event &lt;Jason, livesIn, US, 2068&gt; happens, given:</p>
  <p>1.the above datasets which includes info about the relation: <code>livesIn</code></p>
  <p>2.the starting time when Mike lives in US, say he started to live in US since 2030.</p>
  <p>I have used normal distribution to simulate, but I am wondering if there are any other better AI / ML / Stats approaches. Thanks a lot!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-27T11:06:49.250Z
Comments.Text: If the only data you have is samples from population of period lengths, and you want to predict probability of a new randomly sampled one (from the same population), then this looks like a basic question in statistics, which is not really an AI question. If you still think this is an AI question, could you explain where AI is involved? For instance, if there was a lot more contextual data about the time periods, then this *might* be a suitable problem for machine learning. If that is the case, please use [edit] to explain roughly what contextual data you have.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-27T12:05:54.547Z
Comments.Text: Hi @NeilSlater! Thanks a lot for your comment. Yup, initially I thought of using statistics approaches such as survival function or poisson distribution. But their accuracy is not that good. Thus, I am thinking of using AI/ML to do the prediction. Actually from my point of view, AI is also like a way of modelling(just like physics/math). For example, we can also use ML/DL to learn the Newton's rules right haha(by inputting lots of experiment data to train the Newton's rule's model). Thus, I was wondering if there are similar time-series approaches in ML/DL that could tackle this problem.
Comments.UserDisplayName: Leonard
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-27T13:45:36.613Z
Comments.Text: Unless there is some context that you have not explained in your data, a simple statistics approach will get the best accuracy, most ML methods should get close to it but will not be better, and can easily be worse. ML cannot improve on basic probabilities when the prediction is essentially a guessing game.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-27T13:49:21.323Z
Comments.Text: Advanced AI might even be able to use information such as the student's name, because that may be correlated with demographics that do affect length of the course and likelihood to complete it. It is still not clear from your question that this is what you are asking though.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-27T14:40:52.137Z
Comments.Text: Hi @NeilSlater! Thank you so much for your very detailed explanation : ) Actually in my context, I have a dataset of continuous relations similar to the example above. There are about 10 relations in total, such as isMarriedTo, beingUndergrad, livesIn, etc. For each relation, there are about 1000 data points about this relation(1000 durations), such as  <Leo, isUndergrad, 2010 - 2011> and <Leo, isUndergrad, 2013 - 2016> ........
Comments.UserDisplayName: Leonard
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-27T16:07:27.890Z
Comments.Text: Then your dataset could work nicely with a nonlinear ML approach, perhaps xgboost would be good. Please use [edit[ to add a little bit about the dataset, how you intend to use it, plus where you are stuck understanding or using ML. Out of interest, have you used any ML before for other problems?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-28T14:48:59.483Z
Comments.Text: @NeilSlater Hi Neil! Thanks a lot for your suggestions, I have just done the editing. : ) To be honest, this is actually my first project using ML haha. I am very new to ML haha and I have just started to learn it.
Comments.UserDisplayName: Leonard
Tags: <machine-learning><regression><time-series><probability><probability-distribution>
Title: Predicting the probability of a periodically happening event occurring at a given time
UserDisplayName: Leonard
ThreadText: # Predicting the probability of a periodically happening event occurring at a given time
  
  (Asked by: username_0 on 2021-03-27)
  
  I have encountered this problem on how to predict the probability of a periodically happening event occurring at a given time.
  For example, we have an event called being_an_undergrad. There are many data points: bob is an undergrad from (1999 - 2003), Bill is an undergrad from (1900 - 1903), Alice is an undergrad from (1900 - 1905), and there are many other data points such as (2010 - 2015), (2011 - 2013) ....
  There are many events(data points) of being_an_undergrad. The lasting interval varies, it might be 1 year, 2 years, 3 years, .... or even 10 years. But the majority is around 4 years.
  However, I am wondering given all the data points above. If I now know that Jason starts college in 2021, and how can I calculate/predict the probability that he will still be an undergrad in 2022? and 2023? and 2024 .... 2028, etc.
  My current dataset consists of 10000 tuples representing events of different relations. The relations are all continuous relations similar to the example above. There are about 10 continuous relations in total in this dataset, such as isMarriedTo, beingUndergrad, livesIn, etc. For each relation, there are about 1000 data points(1000 durations) about this relation, for example,
  , .....
  
  ,  ...
  
  My problem now is that I want to get a confidence level(probability) when I want to predict one event happening at a specific time point. For example, I want to predict the probability that event  happens, given:
  1.the above datasets which includes info about the relation: livesIn
  2.the starting time when Mike lives in US, say he started to live in US since 2030.
  I have used normal distribution to simulate, but I am wondering if there are any other better AI / ML / Stats approaches. Thanks a lot!
  
  **Tags:** machine-learning, regression, time-series, probability, probability-distribution
  > **Comments:**
  > **username_1**: If the only data you have is samples from population of period lengths, and you want to predict probability of a new randomly sampled one (from the same population), then this looks like a basic question in statistics, which is not really an AI question. If you still think this is an AI question, could you explain where AI is involved? For instance, if there was a lot more contextual data about the time periods, then this *might* be a suitable problem for machine learning. If that is the case, please use [edit] to explain roughly what contextual data you have.  
  > **username_0**: Hi @NeilSlater! Thanks a lot for your comment. Yup, initially I thought of using statistics approaches such as survival function or poisson distribution. But their accuracy is not that good. Thus, I am thinking of using AI/ML to do the prediction. Actually from my point of view, AI is also like a way of modelling(just like physics/math). For example, we can also use ML/DL to learn the Newton's rules right haha(by inputting lots of experiment data to train the Newton's rule's model). Thus, I was wondering if there are similar time-series approaches in ML/DL that could tackle this problem.  
  > **username_1**: Unless there is some context that you have not explained in your data, a simple statistics approach will get the best accuracy, most ML methods should get close to it but will not be better, and can easily be worse. ML cannot improve on basic probabilities when the prediction is essentially a guessing game.  
  > **username_1**: Advanced AI might even be able to use information such as the student's name, because that may be correlated with demographics that do affect length of the course and likelihood to complete it. It is still not clear from your question that this is what you are asking though.  
  > **username_0**: Hi @NeilSlater! Thank you so much for your very detailed explanation : ) Actually in my context, I have a dataset of continuous relations similar to the example above. There are about 10 relations in total, such as isMarriedTo, beingUndergrad, livesIn, etc. For each relation, there are about 1000 data points about this relation(1000 durations), such as  <Leo, isUndergrad, 2010 - 2011> and <Leo, isUndergrad, 2013 - 2016> ........  
  > **username_1**: Then your dataset could work nicely with a nonlinear ML approach, perhaps xgboost would be good. Please use [edit[ to add a little bit about the dataset, how you intend to use it, plus where you are stuck understanding or using ML. Out of interest, have you used any ML before for other problems?  
  > **username_0**: @NeilSlater Hi Neil! Thanks a lot for your suggestions, I have just done the editing. : ) To be honest, this is actually my first project using ML haha. I am very new to ML haha and I have just started to learn it.  
  
  ---
  
  ## 0 answers
  
  