Answers.Body: <p>Right, before performing any search operation you have to build the vector database by training a model that learns to describe your data (text, images, etc) as <em>vectors embeddings</em>.</p>
  <p>Learning an embedding of you data can be done in multiple ways:</p>
  <ol>
  <li>You take a <strong>pre-trained classifier</strong>,  then remove the classification head, and consider as embeddings the output of the last (usually dense) layer. For example, if you're looking for embeddings of images you can take a ImageNet pre-trained VGG (or ResNet) and take 4096d output of the penultimate dense layer. Then to perform searches or queries you can either: 1) use kNN, 2) clustering, or even 3) train another neural net to learn which samples should be similar and which other dissimilar. A similar approach would be to learn an <em>auto-encoder</em>, and then consider as embeddings the encoder's output.</li>
  <li><strong>Self-supervised</strong> and <strong>Contrastive learning:</strong> In SSL (e.g. SimCLR, SimSiam) you usually take advantage of data augmentations to learn a representation/embedding of the data you can use kNN or clustering on top. With contrastive learning the thing is more explicit, you use a contrastive loss that involves pair of samples either positive-positive or negative-positive to ensure the embeddings are close in PP-pairs, and far in NP-pairs: as you mentioned. This is usually done with Siamese networks and a margin objective. Another approach is to use a triplet loss, in which the training triplets are positive-anchor-negative to prevent collapse of embeddings (i.e. a trivial solution.) Once you have a trained model you can threshold a distance in embedding space to say if two embeddings belong to the same class, for example.</li>
  <li>Another approach is <strong>metric learning</strong>, in which you aim at learning a distance function on the embedding space.</li>
  </ol>
  <p>Indeed, what I've listed here are basic ideas behind more complicated and realistic methods and tools powered by deep learning and neural networks. So, you may want to look at the following tutorials: <a href="https://keras.io/examples/vision/near_dup_search" rel="nofollow noreferrer">1</a>, <a href="https://keras.io/examples/vision/semantic_image_clustering" rel="nofollow noreferrer">2</a>, <a href="https://keras.io/examples/vision/siamese_contrastive" rel="nofollow noreferrer">3</a>, <a href="https://keras.io/examples/vision/siamese_network" rel="nofollow noreferrer">4</a>, <a href="https://keras.io/examples/vision/metric_learning" rel="nofollow noreferrer">5</a>, and <a href="https://keras.io/examples/vision/metric_learning_tf_similarity" rel="nofollow noreferrer">6</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-07T10:06:58.380Z
Answers.LastActivityDate: 2023-05-07T10:06:58.380Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>If I correctly understand the concept of a vector database, it is simply a vector embedding space where through some distance metric, a query can find similar results. Yet, an encoder for an embedding space needs to be trained, so is something similar done here where the loss function involves the expected proximity between a picture of a cat, and the query &quot;cat&quot; for example. The loss here is the difference between how far the two actually are minus how far it should be apart? Is this how I train my own, since I can't afford to pay for Pinecone?</p>
  
Tags: <vector-space>
Title: Are vector databases trained?
UserDisplayName: User
ThreadText: # Are vector databases trained?
  
  (Asked by: username_0 on 2023-05-05)
  
  If I correctly understand the concept of a vector database, it is simply a vector embedding space where through some distance metric, a query can find similar results. Yet, an encoder for an embedding space needs to be trained, so is something similar done here where the loss function involves the expected proximity between a picture of a cat, and the query "cat" for example. The loss here is the difference between how far the two actually are minus how far it should be apart? Is this how I train my own, since I can't afford to pay for Pinecone?
  
  **Tags:** vector-space
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-07. Score: 1 | Accepted answer
  
  Right, before performing any search operation you have to build the vector database by training a model that learns to describe your data (text, images, etc) as *vectors embeddings*.
  
  Learning an embedding of you data can be done in multiple ways:
  
  1. You take a **pre-trained classifier**, then remove the classification head, and consider as embeddings the output of the last (usually dense) layer. For example, if you're looking for embeddings of images you can take a ImageNet pre-trained VGG (or ResNet) and take 4096d output of the penultimate dense layer. Then to perform searches or queries you can either: 1) use kNN, 2) clustering, or even 3) train another neural net to learn which samples should be similar and which other dissimilar. A similar approach would be to learn an *auto-encoder*, and then consider as embeddings the encoder's output.
  2. **Self-supervised** and **Contrastive learning:** In SSL (e.g. SimCLR, SimSiam) you usually take advantage of data augmentations to learn a representation/embedding of the data you can use kNN or clustering on top. With contrastive learning the thing is more explicit, you use a contrastive loss that involves pair of samples either positive-positive or negative-positive to ensure the embeddings are close in PP-pairs, and far in NP-pairs: as you mentioned. This is usually done with Siamese networks and a margin objective. Another approach is to use a triplet loss, in which the training triplets are positive-anchor-negative to prevent collapse of embeddings (i.e. a trivial solution.) Once you have a trained model you can threshold a distance in embedding space to say if two embeddings belong to the same class, for example.
  3. Another approach is **metric learning**, in which you aim at learning a distance function on the embedding space.
  
  Indeed, what I've listed here are basic ideas behind more complicated and realistic methods and tools powered by deep learning and neural networks. So, you may want to look at the following tutorials: [1](https://keras.io/examples/vision/near_dup_search), [2](https://keras.io/examples/vision/semantic_image_clustering), [3](https://keras.io/examples/vision/siamese_contrastive), [4](https://keras.io/examples/vision/siamese_network), [5](https://keras.io/examples/vision/metric_learning), and [6](https://keras.io/examples/vision/metric_learning_tf_similarity).
  
  
  ---
  
  