Please read the topic category description 17 to understand what this is all about


Create your own search engine
In Chapter 5 of the  Course, you learned how to use FAISS to find documents that are most semantically similar to a given query. The goal of this project is to extend this idea to build a retrieval and reranking system, where the retriever returns possibly relevant results, while the reranker evaluates the how relevant these hits are to the query.
An example of the architecture might looks as follows (taken from the sentence-transformers library):

image945×279 8.53 KB


Model(s)
The sentence-transformers models on the Hub 9 are great for the reranking task.

Datasets
Wikipedia is usually a good corpus to test retrieval systems on and you can find a dump in various languages here:

wikipedia 12


Challenges
Implementing the full retriever-reranking architecture might be a challenge, so a simpler place to start is with a single long document. You can then chunk that document into paragraphs and compute the relevancy scores across each paragraph

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 3 that allows a user to enter a search query about a document (or a whole corpus of documents), and returns the top 5 most relevant paragraphs.
Don’t forget to push all your models and datasets to the Hub so others can build on them!


Additional resources

https://www.sbert.net/examples/applications/retrieve_rerank/README.html#retrieve-re-rank 24


Discord channel
To chat and organise with other people interested in this project, head over to our Discord 9 and:


Follow the instructions on the #join-course channel


Join the neural-search-engine channel


Just make sure you comment here to indicate that you’ll be contributing to this project