Answers.Body: <p>A <strong>sequence</strong> of <span class="math-container">$n$</span> random variables <span class="math-container">$z_{1:n} = z_1, z_2, \dots, z_n$</span> is i.i.d. if</p>
  <ol>
  <li>they are identically distributed, i.e. each random variable <span class="math-container">$z_i$</span> has the same distribution</li>
  <li>the joint distribution of all of them is just the product of the marginal distributions of each r.v.</li>
  </ol>
  <p>So, let's imagine a thought experiment in which we throw a coin <span class="math-container">$n$</span> times, so you get a <strong>sequence</strong> of results, but you still don't know what those actual results are (i.e. you don't know whether <span class="math-container">$z_i$</span> is heads or tails), because this is just a thought experiment, so we're still talking about random variables and probability distributions and not datasets.</p>
  <p>You might think: well, we're sampling <span class="math-container">$n$</span> times from the same distribution because we have only 1 coin, so we have only one r.v. In reality, you can think that each throw is associated with a different r.v. <span class="math-container">$z_i$</span>, but that all these r.v.s, <span class="math-container">$z_1, z_2, \dots, z_n$</span>, have the same probability distribution, for example, a Bernoulli with the same <span class="math-container">$p$</span> (the parameter of the Bernoulli). Now, let's say that <span class="math-container">$p = 0.5$</span>. This means that, for a single coin toss, there's a 50% chance that the coin will be tails and a 50% chance it lands on heads. This does not have to be the case. In fact, we could also have a weird coin that prefers to land on heads, so let's say that <span class="math-container">$p = 0.7$</span>, which is the probability it lands on heads. That's fine, and the sequence of random variables <span class="math-container">$z_{1:n}$</span> can still be i.i.d. How is this possible?</p>
  <p>What's the Bernoulli pmf?</p>
  <p><span class="math-container">$$f_\text{marginal}(k_i;p)=p^{k_i}(1-p)^{1-k_i},$$</span>
  where <span class="math-container">$k_i\in \{0,1\}$</span>.</p>
  <p>So, each <span class="math-container">$z_i$</span> has this Bernoulli pmf, with the same <span class="math-container">$p$</span>. This is the <em>identically distributed</em> part of iid.</p>
  <p>For simplicity, let <span class="math-container">$n = 2$</span>, so <span class="math-container">$z_{1:n} = z_{1:2} = z_1, z_2$</span>.</p>
  <p>So, if <span class="math-container">$z_1$</span> and <span class="math-container">$z_n$</span> are independent, we have that their joint is just the product of their marginals</p>
  <p><span class="math-container">\begin{align}
  f_\text{joint}(k_1, k_2;p)
  &amp;=(p^{k_1}(1-p)^{1-k_1}) (p^{k_2}(1-p)^{1-k_2}) \\
  &amp;=p^{k_1} p^{k_2} (1-p)^{1-k_1} (1-p)^{1-k_2} \\
  &amp;=p^{k_1 + k_2} (1-p)^{2-k_1 - k_2} \\
  \end{align}</span>
  where <span class="math-container">$k_1 \in \{0,1\}$</span> is the outcome for <span class="math-container">$z_1$</span> and <span class="math-container">$k_2 \in \{0,1\}$</span> is the outcome for <span class="math-container">$z_2$</span>.</p>
  <p>So, as before, let's say that <span class="math-container">$p= 0.7$</span>, then that can be written as</p>
  <p><span class="math-container">$$
  f_\text{joint}(k_1, k_2; 0.7) = 
  (0.7^{k_1}(1-0.7)^{1-k_1}) (0.7^{k_2}(1-0.7)^{1-k_2})
  $$</span></p>
  <p>If <span class="math-container">$k_1=0$</span> and <span class="math-container">$k_2 = 0$</span>, we have</p>
  <p><span class="math-container">\begin{align}
  f_\text{joint}(0, 0;0.7) 
  &amp;= (0.7^0(1-0.7)^1) (0.7^0(1-0.7)^1) \\
  &amp;= 0.3 * 0.3 = 0.09
  \end{align}</span></p>
  <p>If <span class="math-container">$k_1=1$</span> and <span class="math-container">$k_2 = 1$</span>, we have</p>
  <p><span class="math-container">\begin{align}
  f_\text{joint}(1, 1;0.7) 
  &amp;= (0.7^1(1-0.7)^0) (0.7^1(1-0.7)^0) \\
  &amp;= 0.7 * 0.7 = 0.49
  \end{align}</span></p>
  <p>If <span class="math-container">$k_1= 1$</span> and <span class="math-container">$k_2 = 0$</span>, we have</p>
  <p><span class="math-container">\begin{align}
  f_\text{joint}(1, 1;0.7) 
  &amp;= (0.7^1(1-0.7)^0) (0.7^0(1-0.7)^1) \\
  &amp;= 0.7 * 0.3 = 0.21
  \end{align}</span></p>
  <p>If <span class="math-container">$k_1= 0$</span> and <span class="math-container">$k_2 = 1$</span>, we have</p>
  <p><span class="math-container">\begin{align}
  f_\text{joint}(1, 1;0.7) 
  &amp;= (0.7^0(1-0.7)^1) (0.7^1(1-0.7)^0) \\
  &amp;= 0.3 * 0.7 = 0.21
  \end{align}</span></p>
  <p>So, the probabilities are not uniform, but we still have two independent r.v.s, because we defined their joint as the product of their marginals.</p>
  <p>Now, set <span class="math-container">$p = 0.5$</span>, you will see that we will have <span class="math-container">$1/4$</span> for all the combinations of <span class="math-container">$k_1$</span> and <span class="math-container">$k_2$</span>.</p>
  <p>Generally, the joint distribution of <span class="math-container">$n$</span> independent Bernoulli can be  compactly written as follows</p>
  <p><span class="math-container">$$
  f_\text{joint}(k_1, \dots, k_n; p) 
  =
  p^{\sum k_i}(1-p)^{n- \sum k_i}
  $$</span></p>
  <p><strong>Conclusion</strong>: you cannot determine whether a sequence of r.v.s is i.i.d. by just looking at the probabilities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T08:05:35.650Z
Answers.Text: Okay, I think I understood now. If we consider a dataset, then we should not consider features as random variables to call the dataset an iid and we should view instances of the dataset as random variables.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T08:56:41.403Z
Answers.Text: @hanugm You can consider instances of the dataset as random variables, I would say. The dataset would be like the "sequence" here.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T00:09:40.037Z
Answers.LastActivityDate: 2022-01-16T09:14:20.427Z
Answers.LastEditDate: 2022-01-16T09:14:20.427Z
Answers.UserDisplayName: nbro
Body: <p>Let <span class="math-container">$X_1, X_2$</span> be two discrete random variables. Each random variable takes two values: <span class="math-container">$1, 2$</span></p>
  <p>The probability distribution <span class="math-container">$p_1$</span> over <span class="math-container">$X_1, X_2$</span> is given by</p>
  <p><span class="math-container">$$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$</span>
  <span class="math-container">$$p_1(X_1=1, X_2 = 2) = \dfrac{1}{4}$$</span>
  <span class="math-container">$$p_1(X_1=2, X_2 = 1) = \dfrac{1}{4}$$</span>
  <span class="math-container">$$p_1(X_1=2, X_2 = 2) = \dfrac{1}{4}$$</span></p>
  <p>The probability distribution <span class="math-container">$p_2$</span> over <span class="math-container">$X_1, X_2$</span> is given by</p>
  <p><span class="math-container">$$p_2(X_1=1, X_2 = 1) = \dfrac{8}{16}$$</span>
  <span class="math-container">$$p_2(X_1=1, X_2 = 2) = \dfrac{4}{16}$$</span>
  <span class="math-container">$$p_2(X_1=2, X_2 = 1) = \dfrac{3}{16}$$</span>
  <span class="math-container">$$p_2(X_1=2, X_2 = 2) = \dfrac{1}{16}$$</span></p>
  <p>Suppose <span class="math-container">$D_1, D_2$</span> are the datasets generated by <span class="math-container">$p_1, p_2$</span> respectively.</p>
  <p><strong>Then which dataset can I call an iid?</strong> I am guessing as <span class="math-container">$D_1$</span> since we can prove the random variables are independent and are identically distributed and for <span class="math-container">$D_2$</span>, iid does not hold.</p>
  <hr />
  <p><span class="math-container">$\underline{\text{ For }D_1}$</span></p>
  <p>Identical : <span class="math-container">$p_1(X_1 = x_1) = p_1(X_2 = x_2)= \dfrac{1}{2} \text{ where } x_1 = x_2  \in \{1, 2\}$</span><br />
  Independent: <span class="math-container">$p_1(X_1 = x_1,X_2 = x_2) = \dfrac{1}{4} = p_1(X_1 = x_1) p_1(X_2 = x_2) \text{ for  } x_1, x_2 \in \{1, 2\}$</span></p>
  <hr />
  <p>We can show that random variables <span class="math-container">$X_1, X_2$</span> are not iid if we consider <span class="math-container">$p_2$</span>.</p>
  <p>Is the iid I am discussing is different from making an iid of a dataset as answered <a href="https://ai.stackexchange.com/a/31655/18758">here</a>? If not, where am I going wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-12T08:09:58.057Z
Comments.Text: The probability distributions used to describe i.i.d. vs non-i.i.d. are normally considered per example, whilst here it looks like you want to consider a pd over the whole dataset at once. Whilst that is possible, that is an unusual approach, and I am not quite sure what insight you hope to gain with it. Is that really what you want to ask? The pd examples you used in the linked question were clearly per example
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-12T08:25:09.600Z
Comments.Text: @NeilSlater It is difficult for me to understand  what you mean by *per example* related probability distributions. I can image only a single pd for a whole dataset. If the random variables obey independence and identical distribution properties as $p_1$ in $D_1$, then I can recognize it as iid else non-iid as $D_2$ because of $p_2$. If possible, please recommend some material so that I can understand in detail what you are saying.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-12T08:58:17.470Z
Comments.Text: Sorry I don't understand your comment or question. I hope someone can help answer your question.
Comments.UserDisplayName: Neil Slater
Tags: <terminology><datasets><probability-distribution><random-variable><iid>
Title: Which of the following probability distribution is generating an iid dataset?
UserDisplayName: hanugm
ThreadText: # Which of the following probability distribution is generating an iid dataset?
  
  (Asked by: username_0 on 2021-09-11)
  
  Let $X\_1, X\_2$ be two discrete random variables. Each random variable takes two values: $1, 2$
  
  The probability distribution $p\_1$ over $X\_1, X\_2$ is given by
  
  $$p\_1(X\_1=1, X\_2 = 1) = \dfrac{1}{4}$$
  $$p\_1(X\_1=1, X\_2 = 2) = \dfrac{1}{4}$$
  $$p\_1(X\_1=2, X\_2 = 1) = \dfrac{1}{4}$$
  $$p\_1(X\_1=2, X\_2 = 2) = \dfrac{1}{4}$$
  
  The probability distribution $p\_2$ over $X\_1, X\_2$ is given by
  
  $$p\_2(X\_1=1, X\_2 = 1) = \dfrac{8}{16}$$
  $$p\_2(X\_1=1, X\_2 = 2) = \dfrac{4}{16}$$
  $$p\_2(X\_1=2, X\_2 = 1) = \dfrac{3}{16}$$
  $$p\_2(X\_1=2, X\_2 = 2) = \dfrac{1}{16}$$
  
  Suppose $D\_1, D\_2$ are the datasets generated by $p\_1, p\_2$ respectively.
  
  **Then which dataset can I call an iid?** I am guessing as $D\_1$ since we can prove the random variables are independent and are identically distributed and for $D\_2$, iid does not hold.
  
  ---
  
  $\underline{\text{ For }D\_1}$
  
  Identical : $p\_1(X\_1 = x\_1) = p\_1(X\_2 = x\_2)= \dfrac{1}{2} \text{ where } x\_1 = x\_2 \in \{1, 2\}$  
  Independent: $p\_1(X\_1 = x\_1,X\_2 = x\_2) = \dfrac{1}{4} = p\_1(X\_1 = x\_1) p\_1(X\_2 = x\_2) \text{ for } x\_1, x\_2 \in \{1, 2\}$
  
  ---
  
  We can show that random variables $X\_1, X\_2$ are not iid if we consider $p\_2$.
  
  Is the iid I am discussing is different from making an iid of a dataset as answered [here](https://ai.stackexchange.com/a/31655/18758)? If not, where am I going wrong?
  
  **Tags:** terminology, datasets, probability-distribution, random-variable, iid
  > **Comments:**
  > **username_1**: The probability distributions used to describe i.i.d. vs non-i.i.d. are normally considered per example, whilst here it looks like you want to consider a pd over the whole dataset at once. Whilst that is possible, that is an unusual approach, and I am not quite sure what insight you hope to gain with it. Is that really what you want to ask? The pd examples you used in the linked question were clearly per example  
  > **username_0**: @NeilSlater It is difficult for me to understand  what you mean by *per example* related probability distributions. I can image only a single pd for a whole dataset. If the random variables obey independence and identical distribution properties as $p_1$ in $D_1$, then I can recognize it as iid else non-iid as $D_2$ because of $p_2$. If possible, please recommend some material so that I can understand in detail what you are saying.  
  > **username_1**: Sorry I don't understand your comment or question. I hope someone can help answer your question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-16. Score: 2 | Accepted answer
  
  A **sequence** of $n$ random variables $z\_{1:n} = z\_1, z\_2, \dots, z\_n$ is i.i.d. if
  
  1. they are identically distributed, i.e. each random variable $z\_i$ has the same distribution
  2. the joint distribution of all of them is just the product of the marginal distributions of each r.v.
  
  So, let's imagine a thought experiment in which we throw a coin $n$ times, so you get a **sequence** of results, but you still don't know what those actual results are (i.e. you don't know whether $z\_i$ is heads or tails), because this is just a thought experiment, so we're still talking about random variables and probability distributions and not datasets.
  
  You might think: well, we're sampling $n$ times from the same distribution because we have only 1 coin, so we have only one r.v. In reality, you can think that each throw is associated with a different r.v. $z\_i$, but that all these r.v.s, $z\_1, z\_2, \dots, z\_n$, have the same probability distribution, for example, a Bernoulli with the same $p$ (the parameter of the Bernoulli). Now, let's say that $p = 0.5$. This means that, for a single coin toss, there's a 50% chance that the coin will be tails and a 50% chance it lands on heads. This does not have to be the case. In fact, we could also have a weird coin that prefers to land on heads, so let's say that $p = 0.7$, which is the probability it lands on heads. That's fine, and the sequence of random variables $z\_{1:n}$ can still be i.i.d. How is this possible?
  
  What's the Bernoulli pmf?
  
  $$f\_\text{marginal}(k\_i;p)=p^{k\_i}(1-p)^{1-k\_i},$$
  where $k\_i\in \{0,1\}$.
  
  So, each $z\_i$ has this Bernoulli pmf, with the same $p$. This is the *identically distributed* part of iid.
  
  For simplicity, let $n = 2$, so $z\_{1:n} = z\_{1:2} = z\_1, z\_2$.
  
  So, if $z\_1$ and $z\_n$ are independent, we have that their joint is just the product of their marginals
  
  \begin{align}
  f\_\text{joint}(k\_1, k\_2;p)
  &=(p^{k\_1}(1-p)^{1-k\_1}) (p^{k\_2}(1-p)^{1-k\_2}) \\
  &=p^{k\_1} p^{k\_2} (1-p)^{1-k\_1} (1-p)^{1-k\_2} \\
  &=p^{k\_1 + k\_2} (1-p)^{2-k\_1 - k\_2} \\
  \end{align}
  where $k\_1 \in \{0,1\}$ is the outcome for $z\_1$ and $k\_2 \in \{0,1\}$ is the outcome for $z\_2$.
  
  So, as before, let's say that $p= 0.7$, then that can be written as
  
  $$
  f\_\text{joint}(k\_1, k\_2; 0.7) =
  (0.7^{k\_1}(1-0.7)^{1-k\_1}) (0.7^{k\_2}(1-0.7)^{1-k\_2})
  $$
  
  If $k\_1=0$ and $k\_2 = 0$, we have
  
  \begin{align}
  f\_\text{joint}(0, 0;0.7)
  &= (0.7^0(1-0.7)^1) (0.7^0(1-0.7)^1) \\
  &= 0.3 \* 0.3 = 0.09
  \end{align}
  
  If $k\_1=1$ and $k\_2 = 1$, we have
  
  \begin{align}
  f\_\text{joint}(1, 1;0.7)
  &= (0.7^1(1-0.7)^0) (0.7^1(1-0.7)^0) \\
  &= 0.7 \* 0.7 = 0.49
  \end{align}
  
  If $k\_1= 1$ and $k\_2 = 0$, we have
  
  \begin{align}
  f\_\text{joint}(1, 1;0.7)
  &= (0.7^1(1-0.7)^0) (0.7^0(1-0.7)^1) \\
  &= 0.7 \* 0.3 = 0.21
  \end{align}
  
  If $k\_1= 0$ and $k\_2 = 1$, we have
  
  \begin{align}
  f\_\text{joint}(1, 1;0.7)
  &= (0.7^0(1-0.7)^1) (0.7^1(1-0.7)^0) \\
  &= 0.3 \* 0.7 = 0.21
  \end{align}
  
  So, the probabilities are not uniform, but we still have two independent r.v.s, because we defined their joint as the product of their marginals.
  
  Now, set $p = 0.5$, you will see that we will have $1/4$ for all the combinations of $k\_1$ and $k\_2$.
  
  Generally, the joint distribution of $n$ independent Bernoulli can be compactly written as follows
  
  $$
  f\_\text{joint}(k\_1, \dots, k\_n; p)
  =
  p^{\sum k\_i}(1-p)^{n- \sum k\_i}
  $$
  
  **Conclusion**: you cannot determine whether a sequence of r.v.s is i.i.d. by just looking at the probabilities.
  
  > **Comments:**
  > **username_0**: Okay, I think I understood now. If we consider a dataset, then we should not consider features as random variables to call the dataset an iid and we should view instances of the dataset as random variables.  
  > **username_2**: @hanugm You can consider instances of the dataset as random variables, I would say. The dataset would be like the "sequence" here.  
  
  ---
  
  