Answers.Body: <p>There are known architectures to implement this idea, namely, seamese networks, and also a training strategy, known as contrastive learning, that relies on the idea of comparing the output of neural networks. I will explain both of them briefly.</p>
  <p>The idea of seamese networks is exactly what you mentioned. You have a single model, <span class="math-container">$m$</span>, that receives two inputs, <span class="math-container">$x_{1}$</span> and <span class="math-container">$x_{2}$</span>. Uppon the application of <span class="math-container">$m$</span>, one has <span class="math-container">$h_{i} = m(x_{i})$</span>.</p>
  <p>Now, seamese networks have been employed in at least two situations that I know of. The first one is known as Contrastive Learning, and seeks to learn a classifier from pairs of examples. A pair <span class="math-container">$(x_{i}, x_{j})$</span> is said to be a positive pair if they belong to the same class, otherwise it is called a negative example. The idea is to encode both examples with <span class="math-container">$m$</span>, then comparing the learned representations through a similarity measure <span class="math-container">$sim(h_{i}, h_{j})$</span>. As an example of similarity measure, take the inner product,</p>
  <p><span class="math-container">$$sim(h_{i}, h_{j}) = \sum_{k=1}^{p}h_{ik}h_{jk}$$</span></p>
  <p>Using this, the contrastive loss reads as,</p>
  <p><span class="math-container">$$\ell(h_{i}, h_{j}) = \dfrac{exp(sim(h_{i}, h_{j}) / \tau)}{\sum_{n=1}^{N}exp(sim(h_{i}, h_{n}) / \tau)}$$</span></p>
  <p>where <span class="math-container">$\tau &gt; 0$</span> is known as the temperature, in allusion to Statistical Mechanics. The principle behind this type of learning strategy is the the model <span class="math-container">$m$</span> will learn to encoder inputs in such a way that positive pairs (samples from the same class) are mapped close together in the latent space, whereas negative pairs will be far apart. For reference you can consult <a href="https://arxiv.org/pdf/1911.05722.pdf" rel="nofollow noreferrer">[He et al., 2020]</a></p>
  <p>The second application that I know of is metric learning. In this case, suppose that you have a metric <span class="math-container">$d$</span> that is useful, but very costly to calculate over the input space (e.g. <span class="math-container">$d(x_{1}, x_{2})$</span> takes a lot of time to calculate). This is the case, for instance, with the Wasserstein distance. The idea is to precompute a bunch of values <span class="math-container">$d_{ij} = d(x_{i}, x_{j})$</span>, and to train a seamese network so that <span class="math-container">$\lVert h_{i} - h_{j} \rVert \approx d_{ij}$</span>, that way we may measure <span class="math-container">$d_{ij}$</span> by computing the Euclidean distance over the latent space. This is roughly the idea behind the work of <a href="https://openreview.net/pdf?id=SJyEH91A-" rel="nofollow noreferrer">[Courty et al., 2017]</a></p>
  <h1>References</h1>
  <p>[He et al., 2020] He, Kaiming, et al. &quot;Momentum contrast for unsupervised visual representation learning.&quot; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.</p>
  <p>[Courty et al., 2017] Courty, Nicolas, Rémi Flamary, and Mélanie Ducoffe. &quot;Learning wasserstein embeddings.&quot; arXiv preprint arXiv:1710.07457 (2017).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-21T09:02:14.513Z
Answers.LastActivityDate: 2021-12-21T09:02:14.513Z
Answers.UserDisplayName: Eduardo Montesuma
Body: <p>Is there a deep learning architecture where outputs of the same model with two different inputs are used for error calculation (backpropagation)?</p>
  <p>Workflow:</p>
  <p>Input1 -----&gt; Model ------&gt; Output1</p>
  <p>Input2 -----&gt; Model ------&gt; Output2</p>
  <p>Loss = criterion(Output1, Output2)</p>
  <p>Backpropagation(Loss)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-20T09:07:41.040Z
Comments.Text: There is at least one model training process that I can think of that works like this in practice. I am not sure what the downvote was for, but perhaps you could explain what your original problem is, and what you want to understand about the model. Otherwise the direct answer to your question is "yes" with no details.
Comments.UserDisplayName: Neil Slater
Tags: <deep-learning><backpropagation><models>
Title: Deep Learning Architecture where outputs from two different inputs are used for error calculation
UserDisplayName: Aditya Prakash
ThreadText: # Deep Learning Architecture where outputs from two different inputs are used for error calculation
  
  (Asked by: username_0 on 2021-12-20)
  
  Is there a deep learning architecture where outputs of the same model with two different inputs are used for error calculation (backpropagation)?
  
  Workflow:
  
  Input1 -----> Model ------> Output1
  
  Input2 -----> Model ------> Output2
  
  Loss = criterion(Output1, Output2)
  
  Backpropagation(Loss)
  
  **Tags:** deep-learning, backpropagation, models
  
  > **Comments:**
  > **username_1**: There is at least one model training process that I can think of that works like this in practice. I am not sure what the downvote was for, but perhaps you could explain what your original problem is, and what you want to understand about the model. Otherwise the direct answer to your question is "yes" with no details.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-12-21. Score: 1
  
  There are known architectures to implement this idea, namely, seamese networks, and also a training strategy, known as contrastive learning, that relies on the idea of comparing the output of neural networks. I will explain both of them briefly.
  
  The idea of seamese networks is exactly what you mentioned. You have a single model, $m$, that receives two inputs, $x\_{1}$ and $x\_{2}$. Uppon the application of $m$, one has $h\_{i} = m(x\_{i})$.
  
  Now, seamese networks have been employed in at least two situations that I know of. The first one is known as Contrastive Learning, and seeks to learn a classifier from pairs of examples. A pair $(x\_{i}, x\_{j})$ is said to be a positive pair if they belong to the same class, otherwise it is called a negative example. The idea is to encode both examples with $m$, then comparing the learned representations through a similarity measure $sim(h\_{i}, h\_{j})$. As an example of similarity measure, take the inner product,
  
  $$sim(h\_{i}, h\_{j}) = \sum\_{k=1}^{p}h\_{ik}h\_{jk}$$
  
  Using this, the contrastive loss reads as,
  
  $$\ell(h\_{i}, h\_{j}) = \dfrac{exp(sim(h\_{i}, h\_{j}) / \tau)}{\sum\_{n=1}^{N}exp(sim(h\_{i}, h\_{n}) / \tau)}$$
  
  where $\tau > 0$ is known as the temperature, in allusion to Statistical Mechanics. The principle behind this type of learning strategy is the the model $m$ will learn to encoder inputs in such a way that positive pairs (samples from the same class) are mapped close together in the latent space, whereas negative pairs will be far apart. For reference you can consult [[He et al., 2020]](https://arxiv.org/pdf/1911.05722.pdf)
  
  The second application that I know of is metric learning. In this case, suppose that you have a metric $d$ that is useful, but very costly to calculate over the input space (e.g. $d(x\_{1}, x\_{2})$ takes a lot of time to calculate). This is the case, for instance, with the Wasserstein distance. The idea is to precompute a bunch of values $d\_{ij} = d(x\_{i}, x\_{j})$, and to train a seamese network so that $\lVert h\_{i} - h\_{j} \rVert \approx d\_{ij}$, that way we may measure $d\_{ij}$ by computing the Euclidean distance over the latent space. This is roughly the idea behind the work of [[Courty et al., 2017]](https://openreview.net/pdf?id=SJyEH91A-)
  
  # References
  
  [He et al., 2020] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
  
  [Courty et al., 2017] Courty, Nicolas, Rémi Flamary, and Mélanie Ducoffe. "Learning wasserstein embeddings." arXiv preprint arXiv:1710.07457 (2017).
  
  
  ---
  
  