Answers.Body: <p>As far as I know, it doesn't make sense to say that a probability distribution is i.i.d., as you're saying.</p>
  <p>The property i.i.d. is a property of a <strong>sequence of random variables</strong>.</p>
  <p>In your case, the random variables are <span class="math-container">$z_i = (x_i, y_i)$</span>, so it's not just the input <span class="math-container">$y_i$</span> or the label <span class="math-container">$y_i$</span>, but both.</p>
  <p>The rest of the explanation can be taken from <a href="https://ai.stackexchange.com/a/34166/2444">the other answer that I've just given</a>, but, in a few words, the shape of your joint doesn't determine whether your samples are i.i.d. or not.</p>
  <p>In practice, sampling/drawing i.i.d. from a distribution could mean that you will not change the probabilities of occurrence of the samples or create other dependencies. See <a href="http://www.scholarpedia.org/article/Sampling_bias" rel="nofollow noreferrer">this article</a> on the sampling bias.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T08:08:17.607Z
Answers.Text: I think it is the input $x_i$ not $y_i$. Please check.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T09:08:26.350Z
Answers.Text: $x_i$ is the input, right, but what I meant is that you sample both $x_i$ and $y_i$ at the same time from the joint $p(x, y) = p(y \mid x) p(x)$, that's why I denote the pair $(x_i, y_i)$ by $z_i$ to emphasize that the pair is a random variable too.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T09:19:05.060Z
Answers.Text: Yeah. My chronic misconception is gone now.  Till now, I view features as random variables and talk about the iid property of the dataset. Today I realized that we need to see the instances as random variables and the dataset as an iid sequence.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T10:06:31.673Z
Answers.Text: @hanugm Note that not all datasets might be iid. For example, time-series datasets, where consecutive points have dependencies. In those cases, the sample $x_t$ depends on the sample $x_{t-1}$ and probably many other previous data points. However, it's true that many ML algorithms/models assume that the data is iid, even though sometimes that might not be the case.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T00:21:25.670Z
Answers.LastActivityDate: 2022-01-16T15:14:09.633Z
Answers.LastEditDate: 2022-01-16T15:14:09.633Z
Answers.UserDisplayName: nbro
Answers.Body: <blockquote>
  <p>My doubt is that how can we draw <em>i.i.d</em> from every probability distribution if our distribution is not an <em>i.i.d</em> distribution.</p>
  </blockquote>
  <p>Probability distributions cannot be defined as <em>i.i.d.</em> or not <em>i.i.d.</em>.</p>
  <p>The term <em>i.i.d.</em> is a property of a dataset. A dataset can be created that is <em>i.i.d.</em> with respect to a particular probability distribution. It doesn't matter what that distribution is, it just has to exist and be relevant to the purpose the ML is being put to. An underlying population or distribution of data is assumed in a lot of theories in machine learning - for instance, it defines measures of how good a function is at deriving labels from inputs.</p>
  <p>If you define in terms of a population, then i.i.d. is a little bit like your example <span class="math-container">$p(x_i)$</span> if you could sample equally likely from any member of the population. But such a population rarely consists of the product of all possible traits and all possible labels, with one example of each. Some combinations of traits and labels will be rare or impossible, and you will not expect to see them in any sample from the population.</p>
  <p>Commonly, with real-world datasets, there is an assumption of an underlying but unknown probability distribution that is being drawn from, and some efforts are made to make the resulting dataset <em>i.i.d.</em> by e.g. varying free choices on the collection of data (e.g. when, where to collect), shuffling the order of samples, or by removing elements that could affect the sampling with respect to the distribution of interest (e.g. correcting for response rates by social class in surveys). This effort is worthwhile because assumptions about i.i.d. are used in practice by ML models, and they can perform worse if data is not i.i.d.</p>
  <blockquote>
  <p>To put simply, dataset with all possible <span class="math-container">$256 \times 256 \times 3$</span> images is <em>i.i.d</em> but the dataset with all dogs is not an <em>i.i.d</em>.</p>
  </blockquote>
  <p>This is incorrect. Assuming your goal is to classify dog breeds from pictures, then your first dataset is meaningless. It consists mostly of white noise images equally labeled as &quot;poodle&quot; or &quot;spaniel&quot; etc, and even when it was a recognizable picture of a dog for the <span class="math-container">$1$</span> in <span class="math-container">$10^{100}$</span> or so images where that happened, it would most likely have the wrong label. If the second dataset of dogs was all-natural photos of dogs, or all pictures of dogs found on the internet, or any other well-defined population, and it was curated properly when collected to avoid bias or correlation, then it <em>could</em> be i.i.d.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-11T11:45:04.130Z
Answers.LastActivityDate: 2022-01-15T00:32:21.110Z
Answers.LastEditDate: 2022-01-15T00:32:21.110Z
Answers.UserDisplayName: Neil Slater
Body: <p>Consider the following paragraph from <code>2 Learning in High Dimensions</code> in from of the paper titled <a href="https://arxiv.org/pdf/2104.13478.pdf#page=9" rel="nofollow noreferrer">Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges</a></p>
  <blockquote>
  <p>Supervised machine learning, in its simplest formalisation, considers
  a set of <span class="math-container">$N$</span> observations <span class="math-container">$D = \{(x_i, y_i)\}_{i=1}^{N}$</span> <strong>drawn
  <em>i.i.d</em></strong>. from an underlying data distribution <span class="math-container">$P$</span> defined over <span class="math-container">$\mathcal{X} \times \mathcal{Y}$</span>, where <span class="math-container">$\mathcal{X}$</span> and
  <span class="math-container">$\mathcal{Y}$</span> are respectively the data and the label domains. The
  defining feature in this setup is that <span class="math-container">$\mathcal{X}$</span> is a
  high-dimensional space: one typically assumes <span class="math-container">$\mathcal{X} =
   \mathbb{R}^d$</span> to be a Euclidean space of large dimension <span class="math-container">$d$</span>.</p>
  </blockquote>
  <p>Here, it is mentioned that <span class="math-container">$N$</span> observations are <strong>drawn i.i.d</strong> from probability distribution <span class="math-container">$P$</span>, which is defined over <span class="math-container">$\mathcal{X} \times \mathcal{Y}$</span>.</p>
  <p>My doubt is that how can we draw <em>i.i.d</em> from every probability distribution if our distribution is not an <em>i.i.d</em> distribution. The only <span class="math-container">$P$</span> I know to be an <em>i.i.d</em> is the following</p>
  <p><span class="math-container">$$p(x_i) = \dfrac{1}{|\mathcal{X} \times \mathcal{Y}|} \text{ for } x_i \in \mathcal{X} \times \mathcal{Y} \text{ and } 1 \le i \le |\mathcal{X} \times \mathcal{Y}|$$</span></p>
  <p>To put simply, dataset with all possible <span class="math-container">$256 \times 256 \times 3$</span> images is <em>i.i.d</em> but the dataset with all dogs is not an <em>i.i.d</em>.</p>
  <p>As per my knowledge, every possible distribution may not be an <em>i.i.d</em> distribution. Then, without knowing anything about the distribution, how can we draw <em>i.i.d</em>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-16T23:09:47.917Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/133263/discussion-on-question-by-hanugm-how-can-we-draw-i-i-d-from-any-probability-di).
Comments.UserDisplayName: nbro
Tags: <datasets><probability-distribution><iid>
Title: How can we "draw i.i.d" from any probability distribution?
UserDisplayName: hanugm
ThreadText: # How can we "draw i.i.d" from any probability distribution?
  
  (Asked by: username_0 on 2021-09-11)
  
  Consider the following paragraph from `2 Learning in High Dimensions` in from of the paper titled [Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges](https://arxiv.org/pdf/2104.13478.pdf#page=9)
  
  > Supervised machine learning, in its simplest formalisation, considers
  > a set of $N$ observations $D = \{(x\_i, y\_i)\}\_{i=1}^{N}$ **drawn
  > *i.i.d***. from an underlying data distribution $P$ defined over $\mathcal{X} \times \mathcal{Y}$, where $\mathcal{X}$ and
  > $\mathcal{Y}$ are respectively the data and the label domains. The
  > defining feature in this setup is that $\mathcal{X}$ is a
  > high-dimensional space: one typically assumes $\mathcal{X} =
  > \mathbb{R}^d$ to be a Euclidean space of large dimension $d$.
  
  Here, it is mentioned that $N$ observations are **drawn i.i.d** from probability distribution $P$, which is defined over $\mathcal{X} \times \mathcal{Y}$.
  
  My doubt is that how can we draw *i.i.d* from every probability distribution if our distribution is not an *i.i.d* distribution. The only $P$ I know to be an *i.i.d* is the following
  
  $$p(x\_i) = \dfrac{1}{|\mathcal{X} \times \mathcal{Y}|} \text{ for } x\_i \in \mathcal{X} \times \mathcal{Y} \text{ and } 1 \le i \le |\mathcal{X} \times \mathcal{Y}|$$
  
  To put simply, dataset with all possible $256 \times 256 \times 3$ images is *i.i.d* but the dataset with all dogs is not an *i.i.d*.
  
  As per my knowledge, every possible distribution may not be an *i.i.d* distribution. Then, without knowing anything about the distribution, how can we draw *i.i.d*?
  
  **Tags:** datasets, probability-distribution, iid
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/133263/discussion-on-question-by-hanugm-how-can-we-draw-i-i-d-from-any-probability-di).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-01-16. Score: 3
  
  As far as I know, it doesn't make sense to say that a probability distribution is i.i.d., as you're saying.
  
  The property i.i.d. is a property of a **sequence of random variables**.
  
  In your case, the random variables are $z\_i = (x\_i, y\_i)$, so it's not just the input $y\_i$ or the label $y\_i$, but both.
  
  The rest of the explanation can be taken from [the other answer that I've just given](https://ai.stackexchange.com/a/34166/2444), but, in a few words, the shape of your joint doesn't determine whether your samples are i.i.d. or not.
  
  In practice, sampling/drawing i.i.d. from a distribution could mean that you will not change the probabilities of occurrence of the samples or create other dependencies. See [this article](http://www.scholarpedia.org/article/Sampling_bias) on the sampling bias.
  
  > **Comments:**
  > **username_0**: I think it is the input $x_i$ not $y_i$. Please check.  
  > **username_1**: $x_i$ is the input, right, but what I meant is that you sample both $x_i$ and $y_i$ at the same time from the joint $p(x, y) = p(y \mid x) p(x)$, that's why I denote the pair $(x_i, y_i)$ by $z_i$ to emphasize that the pair is a random variable too.  
  > **username_0**: Yeah. My chronic misconception is gone now.  Till now, I view features as random variables and talk about the iid property of the dataset. Today I realized that we need to see the instances as random variables and the dataset as an iid sequence.  
  > **username_1**: @hanugm Note that not all datasets might be iid. For example, time-series datasets, where consecutive points have dependencies. In those cases, the sample $x_t$ depends on the sample $x_{t-1}$ and probably many other previous data points. However, it's true that many ML algorithms/models assume that the data is iid, even though sometimes that might not be the case.  
  
  ---
  
  ### Answer by username_2 on 2021-09-11. Score: 1
  
  > My doubt is that how can we draw *i.i.d* from every probability distribution if our distribution is not an *i.i.d* distribution.
  
  Probability distributions cannot be defined as *i.i.d.* or not *i.i.d.*.
  
  The term *i.i.d.* is a property of a dataset. A dataset can be created that is *i.i.d.* with respect to a particular probability distribution. It doesn't matter what that distribution is, it just has to exist and be relevant to the purpose the ML is being put to. An underlying population or distribution of data is assumed in a lot of theories in machine learning - for instance, it defines measures of how good a function is at deriving labels from inputs.
  
  If you define in terms of a population, then i.i.d. is a little bit like your example $p(x\_i)$ if you could sample equally likely from any member of the population. But such a population rarely consists of the product of all possible traits and all possible labels, with one example of each. Some combinations of traits and labels will be rare or impossible, and you will not expect to see them in any sample from the population.
  
  Commonly, with real-world datasets, there is an assumption of an underlying but unknown probability distribution that is being drawn from, and some efforts are made to make the resulting dataset *i.i.d.* by e.g. varying free choices on the collection of data (e.g. when, where to collect), shuffling the order of samples, or by removing elements that could affect the sampling with respect to the distribution of interest (e.g. correcting for response rates by social class in surveys). This effort is worthwhile because assumptions about i.i.d. are used in practice by ML models, and they can perform worse if data is not i.i.d.
  
  > To put simply, dataset with all possible $256 \times 256 \times 3$ images is *i.i.d* but the dataset with all dogs is not an *i.i.d*.
  
  This is incorrect. Assuming your goal is to classify dog breeds from pictures, then your first dataset is meaningless. It consists mostly of white noise images equally labeled as "poodle" or "spaniel" etc, and even when it was a recognizable picture of a dog for the $1$ in $10^{100}$ or so images where that happened, it would most likely have the wrong label. If the second dataset of dogs was all-natural photos of dogs, or all pictures of dogs found on the internet, or any other well-defined population, and it was curated properly when collected to avoid bias or correlation, then it *could* be i.i.d.
  
  
  ---
  
  