Answers.Body: <p>You will lose a lot of information mapping the 3-dimensional probability distributions onto scalars. The best you can do without loss of information is 2-dimensions per distribution. To get the 2-dimensional encoding, you can just drop any of the three dimensions, as its value is fully determined by the other two dimensions (because they have to sum to one). Here is the mathematical reasoning:</p>
  <p>Your input distributions <span class="math-container">$X$</span> consists of <span class="math-container">$N$</span> elements with <span class="math-container">$3$</span> dimensions each, so <span class="math-container">$X \in \mathbb{R}^{N \times 3}$</span>. Also, each of these distributions <span class="math-container">$x_i \in X$</span> sums to one (<span class="math-container">$\sum_j x_{ij} = 1$</span>), and every entry is bigger than zero <span class="math-container">$x_{ij} &gt; 0$</span>.
  Geometrically, this means that every vector <span class="math-container">$x_i$</span> lies on the surface of a triangle in <span class="math-container">$\mathbb{R}^3$</span> that spans between the basis vectors of <span class="math-container">$\mathbb{R}^3$</span>.</p>
  <p>A function that maps each of the distributions <span class="math-container">$x_i$</span> onto a scalar would therefore be a transformation from the triangle onto a line. A smooth function that does that cannot be injective, so you would lose a lot of information. But because the points lie only on a <em>surface</em>, there is an injective function that gives you a two-dimensional encoding like the one described in the beginning.</p>
  <p>Feel free to follow up if things are not clear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-30T16:22:49.390Z
Answers.Text: Thanks for the help! Do you think it would make sense to just flatten the whole vector of distributions and use that as a feature vector? From what I've heard you generally want to reduce the dimensions of your feature vector, use less feature but more sharp ones. Especially if I wanted to combine the distribution information with some more stuff, the feature vector will just explode right? How can any model make use of this, especially without millions of samples? I only have a few hundred
Answers.UserDisplayName: mavex857
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-30T18:24:22.813Z
Answers.Text: Without more details on your project its hard to make suggestions. Questions that might be important to ask: Do you want the output of the model to depend on all the distributions equally, or is there some more detailed structure where one distribution is somehow 'closer' to another one like in a network? Are the distributions in some specific ordering? If the answer to the last two questions is no, then you probably either have to flatten, or you could try something like [DeepSets](https://arxiv.org/abs/1703.06114) which operates on sets of objects.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-29T23:07:21.223Z
Answers.LastActivityDate: 2023-10-29T23:07:21.223Z
Answers.UserDisplayName: Chillston
Body: <p>I have a series of multiple probability distribution like this:</p>
  <pre><code>[
  [0.2, 0.3, 0.5],
  [0.1, 0.2, 0.7],
  ...
  ]
  </code></pre>
  <p>Do you have any suggestions how I can represent this series in a feature vector that I want to use for a machine learning problem?
  Can each distribution in the series can be scaled down to a scalar value without losing to much information, to obtain a feature vector like this?:</p>
  <pre><code>[
  scalar representation of dist 1,
  scalar representation of dist 2,
  ...
  ]
  </code></pre>
  <p>Or can I use the series as is and train my model with a multi-dimensional input?</p>
  
Tags: <machine-learning><feature-extraction><features>
Title: Feature vector representation of probability distribution
UserDisplayName: mavex857
ThreadText: # Feature vector representation of probability distribution
  
  (Asked by: username_0 on 2023-10-29)
  
  I have a series of multiple probability distribution like this:
  
  ```
  [
  [0.2, 0.3, 0.5],
  [0.1, 0.2, 0.7],
  ...
  ]
  
  ```
  
  Do you have any suggestions how I can represent this series in a feature vector that I want to use for a machine learning problem?
  Can each distribution in the series can be scaled down to a scalar value without losing to much information, to obtain a feature vector like this?:
  
  ```
  [
  scalar representation of dist 1,
  scalar representation of dist 2,
  ...
  ]
  
  ```
  
  Or can I use the series as is and train my model with a multi-dimensional input?
  
  **Tags:** machine-learning, feature-extraction, features
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-29. Score: 1
  
  You will lose a lot of information mapping the 3-dimensional probability distributions onto scalars. The best you can do without loss of information is 2-dimensions per distribution. To get the 2-dimensional encoding, you can just drop any of the three dimensions, as its value is fully determined by the other two dimensions (because they have to sum to one). Here is the mathematical reasoning:
  
  Your input distributions $X$ consists of $N$ elements with $3$ dimensions each, so $X \in \mathbb{R}^{N \times 3}$. Also, each of these distributions $x\_i \in X$ sums to one ($\sum\_j x\_{ij} = 1$), and every entry is bigger than zero $x\_{ij} > 0$.
  Geometrically, this means that every vector $x\_i$ lies on the surface of a triangle in $\mathbb{R}^3$ that spans between the basis vectors of $\mathbb{R}^3$.
  
  A function that maps each of the distributions $x\_i$ onto a scalar would therefore be a transformation from the triangle onto a line. A smooth function that does that cannot be injective, so you would lose a lot of information. But because the points lie only on a *surface*, there is an injective function that gives you a two-dimensional encoding like the one described in the beginning.
  
  Feel free to follow up if things are not clear.
  
  > **Comments:**
  > **username_0**: Thanks for the help! Do you think it would make sense to just flatten the whole vector of distributions and use that as a feature vector? From what I've heard you generally want to reduce the dimensions of your feature vector, use less feature but more sharp ones. Especially if I wanted to combine the distribution information with some more stuff, the feature vector will just explode right? How can any model make use of this, especially without millions of samples? I only have a few hundred  
  > **username_1**: Without more details on your project its hard to make suggestions. Questions that might be important to ask: Do you want the output of the model to depend on all the distributions equally, or is there some more detailed structure where one distribution is somehow 'closer' to another one like in a network? Are the distributions in some specific ordering? If the answer to the last two questions is no, then you probably either have to flatten, or you could try something like [DeepSets](https://arxiv.org/abs/1703.06114) which operates on sets of objects.  
  
  ---
  
  