<p>At least the traditional Lucene full-text search is not vector based but an inverted index structure which can sort document relevance ranking via scoring functions such as the term frequency-inverse document frequency (TF-IDF) or the Okapi BM25 algorithm. On the other hand vector-based LLMs often use techniques such as cosine similarity instead for the new <em>semantic</em> relevance ranking which the traditional full-text search cannot satisfactorily achieve.</p>

CC BY-SA 4.0
2023-07-13T21:24:53.733Z
2023-07-13T21:24:53.733Z
cinch
<p>They are used. Tf-IDF and BM25 are pretty strong baselines for document retrieval, and papers on semantic search often include these methods as a comparison point.</p>
<p>e.g., papers like <a href="https://arxiv.org/abs/2004.04906" rel="nofollow noreferrer">this one</a> on open domain QA retrieve documents based on the question being asked. Vector representations perform better, but BM25 is still pretty solid, especially when performance is a bottleneck.</p>

CC BY-SA 4.0
2023-07-16T04:09:29.410Z
2023-07-16T04:12:54.013Z
2023-07-16T04:12:54.013Z
Alexander Wan
<p>Because Lucene does not do a comparison of MEANING. It is, essentially, still a text search.</p>
<p>Vector databases are an attempt to find similar MEANING, even if the words are not identical. That is a totally different area of search. A standard vector (going by OpenAI here) has more than 1500 dimensions and identifies the similarity based on every dimension - Lucene would not even know the difference between &quot;large village&quot; and &quot;city&quot; while in reality they are QUITE similar. Or would not know that a &quot;House&quot; is close in meaning to a &quot;Building&quot;.</p>
<blockquote>
<p>Is there any objective study</p>
</blockquote>
<p>No one reading the documentation would need that - they simply do too many different things. Heck, Lucene does not even HAVE semantic search built in. The difference is not in the search - it is in what is searched and how the vector for a vector database is constructed to try to capture the meaning, not the similarity in words. At the minimum you would need Lucene + a semantic search mechanism - and then you would have to calculate something like a similarity along a lot of axis. Also known as a vector.</p>

CC BY-SA 4.0
2023-07-14T09:32:00.237Z
2023-07-14T09:32:00.237Z
TomTom
<p>w.r.t. LLM applications using the RAG (retriever-augmented-generation) architecture, people have started taken it for granted that it will be powered by a vector database. e.g., see <a href="https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/" rel="noreferrer">this</a>:</p>
<blockquote>
<p>The most important piece of the preprocessing pipeline, from a systems standpoint, is the vector database.</p>
</blockquote>
<p>Why can't lucene index (full-text search) be used for the retriever? Is there any objective study that has been done comparing quality of results using full-text search vs. using a vector database?</p>
<p>As I was writing this, even lucene seems to have jumped on the vector bandwagon. see <a href="https://www.apachecon.com/acna2022/slides/04_lucene_vector_search_sokolov.pdf" rel="noreferrer">this</a></p>

<large-language-models>
Why can't Lucene search be used to power LLM applications?
morpheus
# Why can't Lucene search be used to power LLM applications?

(Asked by: username_0 on 2023-07-13)

w.r.t. LLM applications using the RAG (retriever-augmented-generation) architecture, people have started taken it for granted that it will be powered by a vector database. e.g., see [this](https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/):

> The most important piece of the preprocessing pipeline, from a systems standpoint, is the vector database.

Why can't lucene index (full-text search) be used for the retriever? Is there any objective study that has been done comparing quality of results using full-text search vs. using a vector database?

As I was writing this, even lucene seems to have jumped on the vector bandwagon. see [this](https://www.apachecon.com/acna2022/slides/04_lucene_vector_search_sokolov.pdf)

**Tags:** large-language-models


---

## 3 answers

### Answer by username_1 on 2023-07-13. Score: 3

At least the traditional Lucene full-text search is not vector based but an inverted index structure which can sort document relevance ranking via scoring functions such as the term frequency-inverse document frequency (TF-IDF) or the Okapi BM25 algorithm. On the other hand vector-based LLMs often use techniques such as cosine similarity instead for the new *semantic* relevance ranking which the traditional full-text search cannot satisfactorily achieve.


---

### Answer by username_2 on 2023-07-16. Score: 2

They are used. Tf-IDF and BM25 are pretty strong baselines for document retrieval, and papers on semantic search often include these methods as a comparison point.

e.g., papers like [this one](https://arxiv.org/abs/2004.04906) on open domain QA retrieve documents based on the question being asked. Vector representations perform better, but BM25 is still pretty solid, especially when performance is a bottleneck.


---

### Answer by username_3 on 2023-07-14. Score: 1

Because Lucene does not do a comparison of MEANING. It is, essentially, still a text search.

Vector databases are an attempt to find similar MEANING, even if the words are not identical. That is a totally different area of search. A standard vector (going by OpenAI here) has more than 1500 dimensions and identifies the similarity based on every dimension - Lucene would not even know the difference between "large village" and "city" while in reality they are QUITE similar. Or would not know that a "House" is close in meaning to a "Building".

> Is there any objective study

No one reading the documentation would need that - they simply do too many different things. Heck, Lucene does not even HAVE semantic search built in. The difference is not in the search - it is in what is searched and how the vector for a vector database is constructed to try to capture the meaning, not the similarity in words. At the minimum you would need Lucene + a semantic search mechanism - and then you would have to calculate something like a similarity along a lot of axis. Also known as a vector.


---

