Answers.Body: <p><strong>LINEAR COMBINATION WITH AN OFFSET (&quot;BIAS&quot;)</strong></p>
  <p>Let's look at a simple situation where just a 2-vector is fed to a node. Draw it out.</p>
  <p><a href="https://i.sstatic.net/gYWFDcUI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gYWFDcUI.png" alt="2 -&gt; 1 neural net" /></a></p>
  <p>Let <span class="math-container">$\sigma$</span> be an acitvation function. The the output node on the right takes a value equal to:</p>
  <p><span class="math-container">$$
  \sigma\left(
  \omega_{\text{red}}x_{\text{red}} +
  \omega_{\text{blue}}x_{\text{blue}} +
  b_{\text{purple}}
  \right)
  $$</span></p>
  <p>The <span class="math-container">$\omega_{\text{red}}x_{\text{red}}$</span> is equal to some scalar multiple of the first vector component (in red).</p>
  <p>The <span class="math-container">$\omega_{\text{blue}}x_{\text{blue}}$</span> is equal to some scalar multiple of the second vector component (in blue).</p>
  <p>The <span class="math-container">$b_{\text{purple}}$</span> is an offset or bias term (hence the &quot;b&quot;).</p>
  <p>For a general input vector of dimension <span class="math-container">$p$</span>, you wind up with something like <span class="math-container">$
  \sigma\left(
  b +
  \overset{p}{\underset{j=1}{\sum}} \omega_j x_j
  \right)
  $</span>.</p>
  <p>The <span class="math-container">$b +
  \overset{p}{\underset{j=1}{\sum}} \omega_j x_j$</span> is a single number, so the activation function only takes a scalar input, rather than a vector.</p>
  <p>The <span class="math-container">$\omega$</span> weights and <span class="math-container">$b$</span> bias are learned through optimization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-25T11:07:16.487Z
Answers.LastActivityDate: 2024-07-25T11:07:16.487Z
Answers.UserDisplayName: Dave
Answers.Body: <p>Fine thing to develop your algorithms from scratch, fine thing.</p>
  <p>In deep learning, particularly in natural language processing (NLP) tasks, it is common to represent words as vectors and feed these vectors into neural network nodes. This technique is known as word embeddings or word vector representations. I can explain the concept to you based on the principles outlined in the book <em>Deep Learning</em> by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.</p>
  <p>In traditional neural networks, each node receives a single number as input, typically representing a feature or an activation value. However, when working with text data, a single number is not sufficient to capture the rich semantic meaning of words. Instead, we need a way to represent words in a continuous vector space, where similar words are close together and dissimilar words are far apart.</p>
  <p>Word embeddings provide a solution to this problem. They are dense vector representations that capture semantic relationships between words. These vectors are learned from data and can be considered as a projection of words into a lower-dimensional space, where each dimension represents a different aspect of meaning.</p>
  <p>One popular word embedding algorithm is Word2Vec, which has two main variants: Continuous Bag-of-Words (CBOW) and Skip-gram. Both algorithms aim to learn word embeddings by predicting the context of a word within a given sentence or a window of surrounding words.</p>
  <p>The CBOW model, for instance, takes the context words as input and predicts the target word. In this case, the context words are encoded as vectors and summed together before being fed into a single node in the neural network. The network learns to associate the context vector with the corresponding target word. The training process adjusts the word embeddings in such a way that words with similar contexts have similar vector representations.</p>
  <p>By using word embeddings, neural networks can leverage the semantic information captured in the vector space to perform various NLP tasks. These embeddings can capture syntactic and semantic relationships, analogies, and contextual information between words.</p>
  <p>For example, Let's consider a simple example using the Word2Vec CBOW model to learn word embeddings from a small corpus.</p>
  <p>Suppose we have the following corpus consisting of three sentences:</p>
  <ul>
  <li>&quot;I love deep learning.&quot;</li>
  <li>&quot;Deep learning is fascinating.&quot;</li>
  <li>&quot;Neural networks are powerful.&quot;</li>
  </ul>
  <p>To create word embeddings, we need to define a window size, which determines the number of context words we consider on each side of the target word. Let's assume a window size of 1 for this example.</p>
  <p>First, we need to construct a vocabulary from our corpus. In this case, our vocabulary would consist of the following unique words:</p>
  <pre><code>[&quot;I&quot;, &quot;love&quot;, &quot;deep&quot;, &quot;learning&quot;, &quot;is&quot;, &quot;fascinating&quot;, &quot;neural&quot;, &quot;networks&quot;, &quot;are&quot;, &quot;powerful&quot;]
  </code></pre>
  <p>Next, we assign a one-hot encoding to each word in the vocabulary. A one-hot encoding represents each word as a binary vector where only the position corresponding to the word's index is 1, and all other positions are 0. For example, the one-hot encoding for the word &quot;learning&quot; would be <code>[0, 0, 0, 1, 0, 0, 0, 0, 0, 0]</code>.</p>
  <p>Now, we can create training examples for our CBOW model. Let's take the second sentence, <code>&quot;Deep learning is fascinating,&quot;</code> as an example.</p>
  <p>For each target word in the sentence, we consider the window of context words. With a window size of 1, we have the following training examples:</p>
  <ul>
  <li>Input (context) words: [&quot;deep&quot;, &quot;is&quot;]</li>
  <li>Target (center) word: &quot;learning&quot;</li>
  </ul>
  <p>To encode the context words as vectors, we use their respective one-hot encodings:</p>
  <ul>
  <li>Context word &quot;deep&quot;: <code>[0, 0, 1, 0, 0, 0, 0, 0, 0, 0]</code></li>
  <li>Context word &quot;is&quot;: <code>[0, 0, 0, 0, 1, 0, 0, 0, 0, 0]</code></li>
  </ul>
  <p>To feed these context word vectors into a single node in the neural network, we can simply sum them together:</p>
  <ul>
  <li>Sum of context word vectors: <code>[0, 0, 1, 0, 1, 0, 0, 0, 0, 0]</code></li>
  </ul>
  <p>The target word, &quot;learning,&quot; will be used as the output label for training.</p>
  <p>We repeat this process for all target words in the corpus, considering the context words within the defined window size. The neural network then learns to associate the summed context vectors with the corresponding target words by adjusting the word embeddings.</p>
  <p>After training, we obtain word embeddings where words with similar meanings or contexts are closer together in the vector space. These embeddings can be used for various NLP tasks or as input to other downstream models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-01T11:39:09.147Z
Answers.Text: I understand how CBOW and Word2Vec works, I was asking about how a whole vector of numbers gets fed into a single node.
Answers.UserDisplayName: Jake StBu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-01T07:45:16.310Z
Answers.LastActivityDate: 2023-07-01T07:45:16.310Z
Answers.UserDisplayName: Revolucion for Monica
Body: <p>I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm not really sure how this works, because in all the books I've ever read, and all the projects that I've developed, only a single number has ever gone into each node. So, how does this work?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-02T06:52:57.997Z
Comments.Text: where have you seen this? maybe mentioning the reference and the quotes (or the codes) can help clarify whether it really is feeding a whole vector into a single node
Comments.UserDisplayName: Sanyou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-28T07:24:55.743Z
Comments.Text: If the node is $R\to R$ it is probably some transform prior to it and then applied elementwise (or on scalar). If the node is $R^m\to R^n$ I think pretty much anything can happen, as long as you can compute the appropriate jacobians.
Comments.UserDisplayName: Emil
Tags: <natural-language-processing><feedforward-neural-networks><vectors>
Title: How does a multidimensional vector get fed into a single node in a neural network?
UserDisplayName: Jake StBu
ThreadText: # How does a multidimensional vector get fed into a single node in a neural network?
  
  (Asked by: username_0 on 2023-07-01)
  
  I mostly develop neural networks completely from scratch, like without libraries. I've been seeing, especially in NLP tasks, entire vectors, often representing words, get fed into a single node. I'm not really sure how this works, because in all the books I've ever read, and all the projects that I've developed, only a single number has ever gone into each node. So, how does this work?
  
  **Tags:** natural-language-processing, feedforward-neural-networks, vectors
  > **Comments:**
  > **username_1**: where have you seen this? maybe mentioning the reference and the quotes (or the codes) can help clarify whether it really is feeding a whole vector into a single node  
  > **username_2**: If the node is $R\to R$ it is probably some transform prior to it and then applied elementwise (or on scalar). If the node is $R^m\to R^n$ I think pretty much anything can happen, as long as you can compute the appropriate jacobians.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-07-25. Score: 0
  
  **LINEAR COMBINATION WITH AN OFFSET ("BIAS")**
  
  Let's look at a simple situation where just a 2-vector is fed to a node. Draw it out.
  
  [![2 -> 1 neural net](https://i.sstatic.net/gYWFDcUI.png)](https://i.sstatic.net/gYWFDcUI.png)
  
  Let $\sigma$ be an acitvation function. The the output node on the right takes a value equal to:
  
  $$
  \sigma\left(
  \omega\_{\text{red}}x\_{\text{red}} +
  \omega\_{\text{blue}}x\_{\text{blue}} +
  b\_{\text{purple}}
  \right)
  $$
  
  The $\omega\_{\text{red}}x\_{\text{red}}$ is equal to some scalar multiple of the first vector component (in red).
  
  The $\omega\_{\text{blue}}x\_{\text{blue}}$ is equal to some scalar multiple of the second vector component (in blue).
  
  The $b\_{\text{purple}}$ is an offset or bias term (hence the "b").
  
  For a general input vector of dimension $p$, you wind up with something like $
  \sigma\left(
  b +
  \overset{p}{\underset{j=1}{\sum}} \omega\_j x\_j
  \right)
  $.
  
  The $b +
  \overset{p}{\underset{j=1}{\sum}} \omega\_j x\_j$ is a single number, so the activation function only takes a scalar input, rather than a vector.
  
  The $\omega$ weights and $b$ bias are learned through optimization.
  
  
  ---
  
  ### Answer by username_4 on 2023-07-01. Score: 0
  
  Fine thing to develop your algorithms from scratch, fine thing.
  In deep learning, particularly in natural language processing (NLP) tasks, it is common to represent words as vectors and feed these vectors into neural network nodes. This technique is known as word embeddings or word vector representations. I can explain the concept to you based on the principles outlined in the book Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
  In traditional neural networks, each node receives a single number as input, typically representing a feature or an activation value. However, when working with text data, a single number is not sufficient to capture the rich semantic meaning of words. Instead, we need a way to represent words in a continuous vector space, where similar words are close together and dissimilar words are far apart.
  Word embeddings provide a solution to this problem. They are dense vector representations that capture semantic relationships between words. These vectors are learned from data and can be considered as a projection of words into a lower-dimensional space, where each dimension represents a different aspect of meaning.
  One popular word embedding algorithm is Word2Vec, which has two main variants: Continuous Bag-of-Words (CBOW) and Skip-gram. Both algorithms aim to learn word embeddings by predicting the context of a word within a given sentence or a window of surrounding words.
  The CBOW model, for instance, takes the context words as input and predicts the target word. In this case, the context words are encoded as vectors and summed together before being fed into a single node in the neural network. The network learns to associate the context vector with the corresponding target word. The training process adjusts the word embeddings in such a way that words with similar contexts have similar vector representations.
  By using word embeddings, neural networks can leverage the semantic information captured in the vector space to perform various NLP tasks. These embeddings can capture syntactic and semantic relationships, analogies, and contextual information between words.
  For example, Let's consider a simple example using the Word2Vec CBOW model to learn word embeddings from a small corpus.
  Suppose we have the following corpus consisting of three sentences:
  
  "I love deep learning."
  "Deep learning is fascinating."
  "Neural networks are powerful."
  
  To create word embeddings, we need to define a window size, which determines the number of context words we consider on each side of the target word. Let's assume a window size of 1 for this example.
  First, we need to construct a vocabulary from our corpus. In this case, our vocabulary would consist of the following unique words:
  ["I", "love", "deep", "learning", "is", "fascinating", "neural", "networks", "are", "powerful"]
  
  Next, we assign a one-hot encoding to each word in the vocabulary. A one-hot encoding represents each word as a binary vector where only the position corresponding to the word's index is 1, and all other positions are 0. For example, the one-hot encoding for the word "learning" would be [0, 0, 0, 1, 0, 0, 0, 0, 0, 0].
  Now, we can create training examples for our CBOW model. Let's take the second sentence, "Deep learning is fascinating," as an example.
  For each target word in the sentence, we consider the window of context words. With a window size of 1, we have the following training examples:
  
  Input (context) words: ["deep", "is"]
  Target (center) word: "learning"
  
  To encode the context words as vectors, we use their respective one-hot encodings:
  
  Context word "deep": [0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
  Context word "is": [0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
  
  To feed these context word vectors into a single node in the neural network, we can simply sum them together:
  
  Sum of context word vectors: [0, 0, 1, 0, 1, 0, 0, 0, 0, 0]
  
  The target word, "learning," will be used as the output label for training.
  We repeat this process for all target words in the corpus, considering the context words within the defined window size. The neural network then learns to associate the summed context vectors with the corresponding target words by adjusting the word embeddings.
  After training, we obtain word embeddings where words with similar meanings or contexts are closer together in the vector space. These embeddings can be used for various NLP tasks or as input to other downstream models.
  
  > **Comments:**
  > **username_0**: I understand how CBOW and Word2Vec works, I was asking about how a whole vector of numbers gets fed into a single node.  
  
  ---
  
  