Answers.Body: <p>ChatGPT May 12 Version says:</p>
  <p><em>The cost function you have specified is a pairwise loss function that penalizes the difference between the output of the ANN for any two input instances. It is similar to the contrastive loss function used in Siamese Networks. Siamese Networks are a type of neural network architecture that consists of two or more identical subnetworks that share the same weights and are used to compare two inputs.</em></p>
  <p><em>Therefore, a Siamese Network is a suitable choice for minimizing the cost function and satisfying the given constraint. In particular, you can use a Siamese Network with a shared encoder that maps each input instance to a fixed-length vector representation, followed by a distance metric that computes the pairwise distance between the encoded representations. The distance metric can be implemented using a fully connected layer or a custom loss function.</em></p>
  <p><em>To satisfy the given constraint, you can add a constraint layer to the network that enforces a minimum output value for each input instance. The constraint layer can be implemented using a fully connected layer with a fixed bias term or a custom activation function that saturates at the minimum output value.</em></p>
  <p><em>Overall, the architecture of the Siamese Network depends on the specifics of the problem, including the dimensionality of the input space and the desired level of accuracy. You may need to experiment with different architectures and hyperparameters to achieve the desired performance.</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-13T00:47:58.107Z
Answers.LastActivityDate: 2023-05-13T00:47:58.107Z
Answers.UserDisplayName: Brian O'Donnell
Answers.Body: <p>If I understand your query correctly, you want to create a latent space that groups similar objects. You should then probably look for Siamese networks. However, your loss function will need another term to <em>increase dissimilarity</em> between <em>different</em> labels. Otherwise, as pointed out by Mike NZ, the net would collapse(yes, it is possible). Perhaps <a href="https://towardsdatascience.com/how-to-choose-your-loss-when-designing-a-siamese-neural-net-contrastive-triplet-or-quadruplet-ecba11944ec" rel="nofollow noreferrer">this</a> will give some insights.</p>
  <p>Note that the above method is not completely unsupervised. There are, in fact, a few claims of unsupervised classifications via clustering, although your objective function would look very different. You could go through <a href="https://arxiv.org/pdf/2005.12320v2.pdf" rel="nofollow noreferrer">this</a> paper(called SCAN) for more details.</p>
  <p>Hope it helps.</p>
  <p>[Edit]</p>
  <p>If you want a (lower-dimensional)representation of objects themselves, browsing through <a href="https://machinelearningmastery.com/dimensionality-reduction-for-machine-learning/" rel="nofollow noreferrer">this</a> could help. For a complex problem, linear reductions like PCA, although helpful, aren't probably what you're looking for. Here you can try training autoencoders. The loss function would work, along with some regularization term.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T12:20:58.253Z
Answers.Text: @ cybershiptrooper, the problem I want to solve is as follows: assume that we have a set S consisting of N number of people and for them we define a d-dimensional feature vector. The feature vector includes features characterizing the "quality" of people. I believe that although each person in S  has different feature values, they are of similar quality. Hence, I  want to design a ANN which will minimize their dissimilarity.
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T17:20:50.630Z
Answers.Text: @user3489173 Oh, so you want to do something like dimensionality reduction? Try using autoencoder type architectures. I have added some more details in my answer.
Answers.UserDisplayName: cybershiptrooper
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T18:03:50.323Z
Answers.Text: @ cybershiptrooper  It has nothing to do with dimensionality reduction. The problem is as follows. Assume that we have a set S consisting of N number of people and for them we define a d-dimensional feature vector. The feature vector includes features characterizing the "quality" of people. I believe that although each person in S has different feature values, they are of similar  "high"  quality. Hence, I want to design a ANN which will minimize their dissimilarity.
  
  Then, I want to  use the developed model for quality quantification of a given arbitrary  person.
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T09:10:02.587Z
Answers.LastActivityDate: 2021-05-02T17:26:52.510Z
Answers.LastEditDate: 2021-05-02T17:26:52.510Z
Answers.UserDisplayName: cybershiptrooper
Body: <p>Let <span class="math-container">$\mathcal{S}$</span> be the training data set, where each input <span class="math-container">$u^i \in \mathcal{S}$</span> has <span class="math-container">$d$</span> features.</p>
  <p>I want to design an ANN so that the cost function below is minimized (the sum of the square of pairwise differences between model outputs)  and the given constraint is satisfied, where <span class="math-container">$w$</span> is the ANN model parameter vector.</p>
  <p><span class="math-container">\begin{align}
  \min _{w}&amp; \sum_{\{i, j\} \in \mathcal{S}}\left(f\left(w, u^{i}\right)-f\left(w, u^{j}\right)\right)^{2} \\
  &amp;f\left(w, u^{i}\right) \geq q_{\min }, \quad i \in \mathcal{S}
  \end{align}</span></p>
  <p>What kind of ANN is suitable for this purpose?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-01T23:20:56.360Z
Comments.Text: If the network outputs qmin constantly for every prediction, wouldn't this minimise your loss function?
Comments.UserDisplayName: Mike NZ
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-01T23:30:37.540Z
Comments.Text: @ Mike NZ, yes but outputing  qmin constantly for every prediction is impossible. My question was what type of ANN to use.
Comments.UserDisplayName: user3489173
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-01T23:35:56.230Z
Comments.Text: @ Mike NZ, in fact what I need  is to find an ANN structure which minimizes the dissimilarity of network outputs when all training data are considered.
Comments.UserDisplayName: user3489173
Tags: <neural-networks><machine-learning><architecture><model-request><constrained-optimization>
Title: Which neural network can I use to solve this constrained optimisation problem?
UserDisplayName: user3489173
ThreadText: # Which neural network can I use to solve this constrained optimisation problem?
  
  (Asked by: username_0 on 2021-05-01)
  
  Let $\mathcal{S}$ be the training data set, where each input $u^i \in \mathcal{S}$ has $d$ features.
  
  I want to design an ANN so that the cost function below is minimized (the sum of the square of pairwise differences between model outputs) and the given constraint is satisfied, where $w$ is the ANN model parameter vector.
  
  \begin{align}
  \min \_{w}& \sum\_{\{i, j\} \in \mathcal{S}}\left(f\left(w, u^{i}\right)-f\left(w, u^{j}\right)\right)^{2} \\
  &f\left(w, u^{i}\right) \geq q\_{\min }, \quad i \in \mathcal{S}
  \end{align}
  
  What kind of ANN is suitable for this purpose?
  
  **Tags:** neural-networks, machine-learning, architecture, model-request, constrained-optimization
  > **Comments:**
  > **username_1**: If the network outputs qmin constantly for every prediction, wouldn't this minimise your loss function?  
  > **username_0**: @ Mike NZ, yes but outputing  qmin constantly for every prediction is impossible. My question was what type of ANN to use.  
  > **username_0**: @ Mike NZ, in fact what I need  is to find an ANN structure which minimizes the dissimilarity of network outputs when all training data are considered.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-05-13. Score: 0
  
  ChatGPT May 12 Version says:
  
  *The cost function you have specified is a pairwise loss function that penalizes the difference between the output of the ANN for any two input instances. It is similar to the contrastive loss function used in Siamese Networks. Siamese Networks are a type of neural network architecture that consists of two or more identical subnetworks that share the same weights and are used to compare two inputs.*
  
  *Therefore, a Siamese Network is a suitable choice for minimizing the cost function and satisfying the given constraint. In particular, you can use a Siamese Network with a shared encoder that maps each input instance to a fixed-length vector representation, followed by a distance metric that computes the pairwise distance between the encoded representations. The distance metric can be implemented using a fully connected layer or a custom loss function.*
  
  *To satisfy the given constraint, you can add a constraint layer to the network that enforces a minimum output value for each input instance. The constraint layer can be implemented using a fully connected layer with a fixed bias term or a custom activation function that saturates at the minimum output value.*
  
  *Overall, the architecture of the Siamese Network depends on the specifics of the problem, including the dimensionality of the input space and the desired level of accuracy. You may need to experiment with different architectures and hyperparameters to achieve the desired performance.*
  
  
  ---
  
  ### Answer by username_3 on 2021-05-02. Score: 0
  
  If I understand your query correctly, you want to create a latent space that groups similar objects. You should then probably look for Siamese networks. However, your loss function will need another term to *increase dissimilarity* between *different* labels. Otherwise, as pointed out by Mike NZ, the net would collapse(yes, it is possible). Perhaps [this](https://towardsdatascience.com/how-to-choose-your-loss-when-designing-a-siamese-neural-net-contrastive-triplet-or-quadruplet-ecba11944ec) will give some insights.
  
  Note that the above method is not completely unsupervised. There are, in fact, a few claims of unsupervised classifications via clustering, although your objective function would look very different. You could go through [this](https://arxiv.org/pdf/2005.12320v2.pdf) paper(called SCAN) for more details.
  
  Hope it helps.
  
  [Edit]
  
  If you want a (lower-dimensional)representation of objects themselves, browsing through [this](https://machinelearningmastery.com/dimensionality-reduction-for-machine-learning/) could help. For a complex problem, linear reductions like PCA, although helpful, aren't probably what you're looking for. Here you can try training autoencoders. The loss function would work, along with some regularization term.
  
  > **Comments:**
  > **username_0**: @ cybershiptrooper, the problem I want to solve is as follows: assume that we have a set S consisting of N number of people and for them we define a d-dimensional feature vector. The feature vector includes features characterizing the "quality" of people. I believe that although each person in S  has different feature values, they are of similar quality. Hence, I  want to design a ANN which will minimize their dissimilarity.  
  > **username_3**: @user3489173 Oh, so you want to do something like dimensionality reduction? Try using autoencoder type architectures. I have added some more details in my answer.  
  > **username_0**: @ cybershiptrooper  It has nothing to do with dimensionality reduction. The problem is as follows. Assume that we have a set S consisting of N number of people and for them we define a d-dimensional feature vector. The feature vector includes features characterizing the "quality" of people. I believe that although each person in S has different feature values, they are of similar  "high"  quality. Hence, I want to design a ANN which will minimize their dissimilarity.  Then, I want to  use the developed model for quality quantification of a given arbitrary  person.  
  
  ---
  
  