Answers.Body: <p>If your goal is to quickly find the most relevant papers from abstract dataset, you don't need to train a new QA model. You can use vector database and embedding model to efficiently search your abstracts.</p>
  <p>Suggested approach:</p>
  <ol>
  <li><p>Create embeddings: transform all abstract (or chunks of abstracts) into numerical representations called embeddings. These embeddings capture the semantic meaning of the text. Pre-trained models will do that for you.</p>
  </li>
  <li><p>Vector Database: store these embeddings in a vector database designed for similarity search (for example ChromaDB, Pinecone). Add each vector embedding with metadata, including the paper title and potentially other relevant information.</p>
  </li>
  <li><p>Retrieval: when you have a query/question, convert it into an embedding using the same embedding model. Use the vector database's retriever to find the most similar abstract embeddings. The retriever will return the top-k most relevant abstracts, ranked by their similarity to your query. They will also have paper title as metadata.</p>
  </li>
  <li><p>So now you know top relevant papers. Read them!</p>
  </li>
  </ol>
  <p>Why this approach:</p>
  <ul>
  <li>Searching a vector database is much faster than running a QA model over every abstract.</li>
  <li>Once you create and store the embeddings, you can easily experiment with different queries without having to reprocess all abstracts each time.</li>
  <li>Also, you don't need to train a new model, saving time/resources.</li>
  </ul>
  <p>This approach is used for  Retrieval Augmented Generation(RAG) with LLMs. These retrieved abstracts, along with question itself, are passed to LLM to generate informative answers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-28T20:20:37.813Z
Answers.Text: So, as I see and if I understand correctly, I can apply this directly to my csv file that contain all my paper abstracts
  I will try your idea, thank you for your help
Answers.UserDisplayName: Someone_1313
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T20:37:32.513Z
Answers.Text: I just created a script to perform this task, now that I see the complete picture, it looks like a promising way to accomplish this task, thank you again for your help!!!!!!!
Answers.UserDisplayName: Someone_1313
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-28T18:45:58.860Z
Answers.LastActivityDate: 2024-06-28T18:45:58.860Z
Answers.UserDisplayName: Kulin Patel
Body: <p>I have a data frame that contains different numbers of columns, the important ones are:</p>
  <pre><code>Title Abstract
  </code></pre>
  <p>The length (number of words) varies between each row, with a total of 850 rows, but all are scientific papers related to science (immunology context).
  I created this file using a Python script that checks for different combinations of keywords in a specific context to search in the literature the papers in a straightforward way.
  To move forward I would like to reduce the number of rows, in this case, the number of papers that are useful for my research. With this in mind, I started using transformers to analyze each abstract in my data frame and check if it is useful for my research based on a specific question and context, so my focus is the transformers related to the <strong>question-answering</strong> process.</p>
  <p>I'm completely new in this world of neural network architecture (transformers), so I do not know if I'm doing the process correctly or if the approach that I chose is correct.
  Based on the research that I did my current approach is:</p>
  <pre><code>1. Use transformers from Hugging Face.
  2. Select a dataset that works with Biological data context.
  3. Select a model that is for the question-answering process in the context of biological data. 
  4. Upload the dataset in my workspace and train the model (?)
  5. Define your question and define the context.
  6. Upload your dataset (data frame) that you want to analyze and generate output with long and short answers (?)
  </code></pre>
  <p>So far, I think that I'm in point 3, fixing some issues related to the question-answering model (not related to this post). However, I doubt if all of this is necessary to perform my goal, <strong>reduce the number of papers that are useful for my research (that I need to include to perform discussion process in a paper writing process) based on a question and context.</strong></p>
  <p>Is this process correct for my goal?</p>
  <p>Do you know if is there a better way?</p>
  <p>If the process is correct, what model do you recommend for biological context?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-27T17:32:59.697Z
Comments.Text: Hi, Welcome to AI Stack Exchange. What's ultimate objective? finding relevant papers  among 850 papers based on query/context? or make QA model? Just checking what you expect form model - generated answer or relevant paper.
Comments.UserDisplayName: Kulin Patel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-27T19:43:15.473Z
Comments.Text: Thank you for your reply, my main goal is to be able to check each abstract in my data frame an select the ones that answer my question-context information, because I selected those 850 papers based on keyword criteria, for example, keywords 1, 2, 3..., so now I want to check which one of them really work for me and be able to reduce the number of papers to read
Comments.UserDisplayName: Someone_1313
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-11T20:26:28.307Z
Comments.Text: @KulinPatel Could I have your feedback on my code? Here is the link if you have time to check it:
  https://stackoverflow.com/questions/78737322/issues-with-irrelevant-similarity-search-results-using-rag-and-llms-on-pubmed-da
Comments.UserDisplayName: Someone_1313
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-21T20:26:54.947Z
Comments.Text: I can't see the question.
Comments.UserDisplayName: Kulin Patel
Tags: <transformer><question-answering>
Title: Filter scientific papers based on abstract information
UserDisplayName: Someone_1313
ThreadText: # Filter scientific papers based on abstract information
  
  (Asked by: username_0 on 2024-06-25)
  
  I have a data frame that contains different numbers of columns, the important ones are:
  Title Abstract
  
  The length (number of words) varies between each row, with a total of 850 rows, but all are scientific papers related to science (immunology context).
  I created this file using a Python script that checks for different combinations of keywords in a specific context to search in the literature the papers in a straightforward way.
  To move forward I would like to reduce the number of rows, in this case, the number of papers that are useful for my research. With this in mind, I started using transformers to analyze each abstract in my data frame and check if it is useful for my research based on a specific question and context, so my focus is the transformers related to the question-answering process.
  I'm completely new in this world of neural network architecture (transformers), so I do not know if I'm doing the process correctly or if the approach that I chose is correct.
  Based on the research that I did my current approach is:
  1. Use transformers from Hugging Face.
  2. Select a dataset that works with Biological data context.
  3. Select a model that is for the question-answering process in the context of biological data. 
  4. Upload the dataset in my workspace and train the model (?)
  5. Define your question and define the context.
  6. Upload your dataset (data frame) that you want to analyze and generate output with long and short answers (?)
  
  So far, I think that I'm in point 3, fixing some issues related to the question-answering model (not related to this post). However, I doubt if all of this is necessary to perform my goal, reduce the number of papers that are useful for my research (that I need to include to perform discussion process in a paper writing process) based on a question and context.
  Is this process correct for my goal?
  Do you know if is there a better way?
  If the process is correct, what model do you recommend for biological context?
  
  **Tags:** transformer, question-answering
  > **Comments:**
  > **username_1**: Hi, Welcome to AI Stack Exchange. What's ultimate objective? finding relevant papers  among 850 papers based on query/context? or make QA model? Just checking what you expect form model - generated answer or relevant paper.  
  > **username_0**: Thank you for your reply, my main goal is to be able to check each abstract in my data frame an select the ones that answer my question-context information, because I selected those 850 papers based on keyword criteria, for example, keywords 1, 2, 3..., so now I want to check which one of them really work for me and be able to reduce the number of papers to read  
  > **username_0**: @KulinPatel Could I have your feedback on my code? Here is the link if you have time to check it: https://stackoverflow.com/questions/78737322/issues-with-irrelevant-similarity-search-results-using-rag-and-llms-on-pubmed-da  
  > **username_1**: I can't see the question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-28. Score: 1 | Accepted answer
  
  If your goal is to quickly find the most relevant papers from abstract dataset, you don't need to train a new QA model. You can use vector database and embedding model to efficiently search your abstracts.
  
  Suggested approach:
  
  1. Create embeddings: transform all abstract (or chunks of abstracts) into numerical representations called embeddings. These embeddings capture the semantic meaning of the text. Pre-trained models will do that for you.
  2. Vector Database: store these embeddings in a vector database designed for similarity search (for example ChromaDB, Pinecone). Add each vector embedding with metadata, including the paper title and potentially other relevant information.
  3. Retrieval: when you have a query/question, convert it into an embedding using the same embedding model. Use the vector database's retriever to find the most similar abstract embeddings. The retriever will return the top-k most relevant abstracts, ranked by their similarity to your query. They will also have paper title as metadata.
  4. So now you know top relevant papers. Read them!
  
  Why this approach:
  
  * Searching a vector database is much faster than running a QA model over every abstract.
  * Once you create and store the embeddings, you can easily experiment with different queries without having to reprocess all abstracts each time.
  * Also, you don't need to train a new model, saving time/resources.
  
  This approach is used for Retrieval Augmented Generation(RAG) with LLMs. These retrieved abstracts, along with question itself, are passed to LLM to generate informative answers.
  
  > **Comments:**
  > **username_0**: So, as I see and if I understand correctly, I can apply this directly to my csv file that contain all my paper abstracts I will try your idea, thank you for your help  
  > **username_0**: I just created a script to perform this task, now that I see the complete picture, it looks like a promising way to accomplish this task, thank you again for your help!!!!!!!  
  
  ---
  
  