Answers.Body: <p>At least the traditional Lucene full-text search is not vector based but an inverted index structure which can sort document relevance ranking via scoring functions such as the term frequency-inverse document frequency (TF-IDF) or the Okapi BM25 algorithm. On the other hand vector-based LLMs often use techniques such as cosine similarity instead for the new <em>semantic</em> relevance ranking which the traditional full-text search cannot satisfactorily achieve.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-13T21:24:53.733Z
Answers.LastActivityDate: 2023-07-13T21:24:53.733Z
Answers.UserDisplayName: cinch
Answers.Body: <p>They are used. Tf-IDF and BM25 are pretty strong baselines for document retrieval, and papers on semantic search often include these methods as a comparison point.</p>
  <p>e.g., papers like <a href="https://arxiv.org/abs/2004.04906" rel="nofollow noreferrer">this one</a> on open domain QA retrieve documents based on the question being asked. Vector representations perform better, but BM25 is still pretty solid, especially when performance is a bottleneck.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-16T04:09:29.410Z
Answers.LastActivityDate: 2023-07-16T04:12:54.013Z
Answers.LastEditDate: 2023-07-16T04:12:54.013Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>Because Lucene does not do a comparison of MEANING. It is, essentially, still a text search.</p>
  <p>Vector databases are an attempt to find similar MEANING, even if the words are not identical. That is a totally different area of search. A standard vector (going by OpenAI here) has more than 1500 dimensions and identifies the similarity based on every dimension - Lucene would not even know the difference between &quot;large village&quot; and &quot;city&quot; while in reality they are QUITE similar. Or would not know that a &quot;House&quot; is close in meaning to a &quot;Building&quot;.</p>
  <blockquote>
  <p>Is there any objective study</p>
  </blockquote>
  <p>No one reading the documentation would need that - they simply do too many different things. Heck, Lucene does not even HAVE semantic search built in. The difference is not in the search - it is in what is searched and how the vector for a vector database is constructed to try to capture the meaning, not the similarity in words. At the minimum you would need Lucene + a semantic search mechanism - and then you would have to calculate something like a similarity along a lot of axis. Also known as a vector.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T09:32:00.237Z
Answers.LastActivityDate: 2023-07-14T09:32:00.237Z
Answers.UserDisplayName: TomTom
Body: <p>w.r.t. LLM applications using the RAG (retriever-augmented-generation) architecture, people have started taken it for granted that it will be powered by a vector database. e.g., see <a href="https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/" rel="noreferrer">this</a>:</p>
  <blockquote>
  <p>The most important piece of the preprocessing pipeline, from a systems standpoint, is the vector database.</p>
  </blockquote>
  <p>Why can't lucene index (full-text search) be used for the retriever? Is there any objective study that has been done comparing quality of results using full-text search vs. using a vector database?</p>
  <p>As I was writing this, even lucene seems to have jumped on the vector bandwagon. see <a href="https://www.apachecon.com/acna2022/slides/04_lucene_vector_search_sokolov.pdf" rel="noreferrer">this</a></p>
  
Tags: <large-language-models>
Title: Why can't Lucene search be used to power LLM applications?
UserDisplayName: morpheus
ThreadText: # Why can't Lucene search be used to power LLM applications?
  
  (Asked by: username_0 on 2023-07-13)
  
  w.r.t. LLM applications using the RAG (retriever-augmented-generation) architecture, people have started taken it for granted that it will be powered by a vector database. e.g., see [this](https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/):
  
  > The most important piece of the preprocessing pipeline, from a systems standpoint, is the vector database.
  
  Why can't lucene index (full-text search) be used for the retriever? Is there any objective study that has been done comparing quality of results using full-text search vs. using a vector database?
  
  As I was writing this, even lucene seems to have jumped on the vector bandwagon. see [this](https://www.apachecon.com/acna2022/slides/04_lucene_vector_search_sokolov.pdf)
  
  **Tags:** large-language-models
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-07-13. Score: 3
  
  At least the traditional Lucene full-text search is not vector based but an inverted index structure which can sort document relevance ranking via scoring functions such as the term frequency-inverse document frequency (TF-IDF) or the Okapi BM25 algorithm. On the other hand vector-based LLMs often use techniques such as cosine similarity instead for the new *semantic* relevance ranking which the traditional full-text search cannot satisfactorily achieve.
  
  
  ---
  
  ### Answer by username_2 on 2023-07-16. Score: 2
  
  They are used. Tf-IDF and BM25 are pretty strong baselines for document retrieval, and papers on semantic search often include these methods as a comparison point.
  
  e.g., papers like [this one](https://arxiv.org/abs/2004.04906) on open domain QA retrieve documents based on the question being asked. Vector representations perform better, but BM25 is still pretty solid, especially when performance is a bottleneck.
  
  
  ---
  
  ### Answer by username_3 on 2023-07-14. Score: 1
  
  Because Lucene does not do a comparison of MEANING. It is, essentially, still a text search.
  
  Vector databases are an attempt to find similar MEANING, even if the words are not identical. That is a totally different area of search. A standard vector (going by OpenAI here) has more than 1500 dimensions and identifies the similarity based on every dimension - Lucene would not even know the difference between "large village" and "city" while in reality they are QUITE similar. Or would not know that a "House" is close in meaning to a "Building".
  
  > Is there any objective study
  
  No one reading the documentation would need that - they simply do too many different things. Heck, Lucene does not even HAVE semantic search built in. The difference is not in the search - it is in what is searched and how the vector for a vector database is constructed to try to capture the meaning, not the similarity in words. At the minimum you would need Lucene + a semantic search mechanism - and then you would have to calculate something like a similarity along a lot of axis. Also known as a vector.
  
  
  ---
  
  