Answers.Body: <p>I had to execute a JavaScript code that opens the pdf link in a new window/tab. After that, just closed this newly opened window/tab and continued with the script execution. Here is the updated code:</p>
  <pre><code>import os
  import time
  import csv
  
  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.firefox.options import Options
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  ncbi_url = &quot;https://www.ncbi.nlm.nih.gov/&quot;
  
  download_dir = &quot;pdf_downloaded&quot;
  
  if not os.path.exists(download_dir):
      os.mkdir(download_dir)
      
  options = Options()
  options.set_preference('permissions.default.stylesheet', 2)
  options.set_preference('permissions.default.image', 2)
  options.set_preference('browser.download.folderList', 2)
  options.set_preference('browser.download.dir', os.path.abspath(download_dir))
  options.set_preference('browser.download.manager.showWhenStarting', False)
  options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
  options.set_preference('pdfjs.disabled', True)
  
  ###############################################################################
  def download_pmc_pdf(pmcid_key):
      browser = webdriver.Firefox(options=options)
      pmc_url = ncbi_url + &quot;pmc/articles/&quot; + pmcid_key + &quot;/&quot;
      browser.get(pmc_url)
      try:
          wait = WebDriverWait(browser, 10)
          pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))
  
          # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
          for link in pdf_links:
              href = link.get_attribute('href')
              if '.pdf' in href:
                  # Open the pdf link in a new window
                  browser.execute_script(f'window.open(&quot;{href}&quot;,&quot;_blank&quot;);')
                  break
  
          time.sleep(15)  # delay for 10 seconds
          # After the file is downloaded:
          with open(f'{download_dir}/file_mapping.csv', 'a', newline='') as file:
              writer = csv.writer(file)
              writer.writerow([pmcid_key, os.path.basename(href)])
          ## Switch to the new window and close it 
          ## (this is a hack to deal with bowser freezing upon downloading)
          browser.switch_to.window(browser.window_handles[-1])
          browser.close()
  
      except Exception as e:
          print(e)
      finally:
          browser.quit()  # quit the browser outside the loop    
  ###############################################################################
  pmcid = &quot;2788555&quot;   
  download_pmc_pdf(pmcid)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T15:57:30.880Z
Answers.LastActivityDate: 2023-07-14T16:02:37.583Z
Answers.LastEditDate: 2023-07-14T16:02:37.583Z
Answers.UserDisplayName: Wasim Aftab
Answers.Body: <p>Pubmed Central stores pdfs for all available open access articles on a public-facing ftp server (<a href="https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/" rel="nofollow noreferrer">https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/</a>). You can query the database using the OA Web Service API, e.g. with cURL</p>
  <pre><code># for pubmed id &quot;10026098&quot;
  curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098
  &lt;OA&gt;&lt;responseDate&gt;2023-07-17 01:39:13&lt;/responseDate&gt;&lt;request id=&quot;PMC10026098&quot;&gt;https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098&lt;/request&gt;&lt;records returned-count=&quot;2&quot; total-count=&quot;2&quot;&gt;&lt;record id=&quot;PMC10026098&quot; citation=&quot;Ther Adv Med Oncol. 2023 Mar 18; 15:17588359231161409&quot; license=&quot;CC BY-NC&quot; retracted=&quot;no&quot;&gt;&lt;link format=&quot;tgz&quot; updated=&quot;2023-03-23 12:10:30&quot; href=&quot;ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/83/eb/PMC10026098.tar.gz&quot; /&gt;&lt;link format=&quot;pdf&quot; updated=&quot;2023-03-20 09:34:33&quot; href=&quot;ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf&quot; /&gt;&lt;/record&gt;&lt;/records&gt;&lt;/OA&gt;
  
  # get the pdf link from the API (ftp storage)
  curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\&quot;\(ftp.*.pdf\)\&quot;.*/\1/p'
  ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf
  
  # download the pdf
  pdf=&quot;<span class="math-container">$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
  curl -O "$</span>f&quot;
  
  # download many pdfs from a list of ids
  cat list_of_ids.txt
  10026098
  9341183
  
  while read -r id_from_list
  do
      pdf=&quot;<span class="math-container">$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$</span>id_from_list | sed -n 's/.*\&quot;\(ftp.*.pdf\)\&quot;.*/\1/p')&quot;
      curl -O &quot;$pdf&quot;
  done &lt; list_of_ids.txt
  </code></pre>
  <p>This approach is preferable to selenium scraping for many reasons, and if you have any problems the NCBI provides documentation (<a href="https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/#indart" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/#indart</a>) and offers support (you can email them at pubmedcentral@ncbi.nlm.nih.gov).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T13:05:17.273Z
Answers.Text: Thank you for your time! Unfortunately, it did not work, returning the error `curl: (23) Failed writing received data to disk/application`
  Furthermore, I've noticed a potential issue. What if the article listed on PubMed is not available on PMC but on other platforms, such as the publisher's website? For example, PMID:37230009
Answers.UserDisplayName: Wasim Aftab
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T23:15:03.573Z
Answers.Text: You're welcome; just trying to help. That cURL error is very likely related to your curl install and file permissions (e.g. https://askubuntu.com/questions/1356327/cant-write-to-a-hidden-path-using-curl); you could try using `wget` instead, e.g. `wget -O - https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 2> /dev/null | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p'` should work. And as for articles on Pubmed that aren't on PMC, your question specifies 'articles on PMC', so that is what my answer addresses. If you want to download pdfs from any source, that is a totally different question.
Answers.UserDisplayName: jared_mamrot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T14:41:55.783Z
Answers.Text: Your solution is crashing because pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')" is empty for the pmcids that I used: 8968431, 8570779, 8290288, 8254498. But it worked for the pmcid that you provided. It seems pdfs corresponding to my pmcids are not present on the public-facing ftp server because when I tried your code with some of the pdfs that are listed there it worked. So, for my use case, browser automation is the only way so far
Answers.UserDisplayName: Wasim Aftab
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T22:57:13.080Z
Answers.Text: Ohh! Yep - I see the problem - only Open Access articles are accessible via the API (`<OA><responseDate>2023-07-19 18:55:39</responseDate><request>https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=8968431</request><error code="idIsNotOpenAccess">identifier 'PMC8968431' is not Open Access</error></OA>`). Thanks for taking the time to check!
Answers.UserDisplayName: jared_mamrot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T06:09:16.797Z
Answers.LastActivityDate: 2023-07-17T06:09:16.797Z
Answers.UserDisplayName: jared_mamrot
Body: <p>I'm trying to download pdf from pubmed central using selenium webdriver in python. The code is provided below which downloads the pdf fine but, it does not quit automatically. i.e. it is sort of stuck executing <code>browser.get(href)</code> line in <code>download_pmc_pdf()</code>. What could be the reason?</p>
  <pre><code>import os
  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.firefox.options import Options
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  ncbi_url = &quot;https://www.ncbi.nlm.nih.gov/&quot;
  
  download_dir = &quot;pdf_downloaded&quot;
  
  if not os.path.exists(download_dir):
      os.mkdir(download_dir)
      
  options = Options()
  options.set_preference('permissions.default.stylesheet', 2)
  options.set_preference('permissions.default.image', 2)
  options.set_preference('browser.download.folderList', 2)
  options.set_preference('browser.download.dir', os.path.abspath(download_dir))
  options.set_preference('browser.download.manager.showWhenStarting', False)
  options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
  options.set_preference('pdfjs.disabled', True)
  
  browser = webdriver.Firefox(options=options)
  
  ###############################################################################
  def download_pmc_pdf(pmcid_key):
      pmc_url = ncbi_url + &quot;pmc/articles/&quot; + pmcid_key + &quot;/&quot;
      browser.get(pmc_url)
      try:
          wait = WebDriverWait(browser, 10)
          pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))
          
          # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
          for link in pdf_links:
              href = link.get_attribute('href')
              if '.pdf' in href:
                  browser.get(href)
                  break  # exit the loop as we have found the PDF link
          print(&quot;line beofre browser.quit()&quot;)
          browser.quit()  # quit the browser outside the loop
      except Exception as e:
          print(e)
  
  
      
  ###############################################################################
  pmcid = &quot;2788555&quot;            
  download_pmc_pdf(pmcid)
  browser.quit()
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-14T14:25:14.893Z
Comments.Text: This might be better-suited to stackoverflow since it's primarily a selenium question rather than a bioinformatics question, but for what it's worth, I see the same behavior when I try your script.  (I've used selenium a bit before but never tried `get` for anything other than HTML.)  I don't suppose the e-utils support PDF paper download, do they?
Comments.UserDisplayName: Jesse
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-14T15:50:21.883Z
Comments.Text: To the best of my knowledge, E-utils primarily supports text, not PDF downloads. For some PMC-listed articles, the XML, mediline records lack full text, hence the need for browser automation. Though e-utils is my preferred choice, this situation demands an alternative. Nonetheless, the issue is now resolved
Comments.UserDisplayName: Wasim Aftab
Tags: <python>
Title: selenium webdriver is stuck after downloading and saving a pdf file from pubmed central website
ThreadText: # selenium webdriver is stuck after downloading and saving a pdf file from pubmed central website
  
  (Asked by: username_0 on 2023-07-13)
  
  I'm trying to download pdf from pubmed central using selenium webdriver in python. The code is provided below which downloads the pdf fine but, it does not quit automatically. i.e. it is sort of stuck executing `browser.get(href)` line in `download_pmc_pdf()`. What could be the reason?
  
  ```
  import os
  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.firefox.options import Options
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  ncbi_url = "https://www.ncbi.nlm.nih.gov/"
  
  download_dir = "pdf_downloaded"
  
  if not os.path.exists(download_dir):
      os.mkdir(download_dir)
      
  options = Options()
  options.set_preference('permissions.default.stylesheet', 2)
  options.set_preference('permissions.default.image', 2)
  options.set_preference('browser.download.folderList', 2)
  options.set_preference('browser.download.dir', os.path.abspath(download_dir))
  options.set_preference('browser.download.manager.showWhenStarting', False)
  options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
  options.set_preference('pdfjs.disabled', True)
  
  browser = webdriver.Firefox(options=options)
  
  ###############################################################################
  def download_pmc_pdf(pmcid_key):
      pmc_url = ncbi_url + "pmc/articles/" + pmcid_key + "/"
      browser.get(pmc_url)
      try:
          wait = WebDriverWait(browser, 10)
          pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))
          
          # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
          for link in pdf_links:
              href = link.get_attribute('href')
              if '.pdf' in href:
                  browser.get(href)
                  break  # exit the loop as we have found the PDF link
          print("line beofre browser.quit()")
          browser.quit()  # quit the browser outside the loop
      except Exception as e:
          print(e)
  
      
  ###############################################################################
  pmcid = "2788555"            
  download_pmc_pdf(pmcid)
  browser.quit()
  
  ```
  
  **Tags:** python
  
  > **Comments:**
  > **username_1**: This might be better-suited to stackoverflow since it's primarily a selenium question rather than a bioinformatics question, but for what it's worth, I see the same behavior when I try your script.  (I've used selenium a bit before but never tried `get` for anything other than HTML.)  I don't suppose the e-utils support PDF paper download, do they?  
  > **username_0**: To the best of my knowledge, E-utils primarily supports text, not PDF downloads. For some PMC-listed articles, the XML, mediline records lack full text, hence the need for browser automation. Though e-utils is my preferred choice, this situation demands an alternative. Nonetheless, the issue is now resolved  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-07-14. Score: 2
  
  I had to execute a JavaScript code that opens the pdf link in a new window/tab. After that, just closed this newly opened window/tab and continued with the script execution. Here is the updated code:
  
  ```
  import os
  import time
  import csv
  
  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.firefox.options import Options
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  ncbi_url = "https://www.ncbi.nlm.nih.gov/"
  
  download_dir = "pdf_downloaded"
  
  if not os.path.exists(download_dir):
      os.mkdir(download_dir)
      
  options = Options()
  options.set_preference('permissions.default.stylesheet', 2)
  options.set_preference('permissions.default.image', 2)
  options.set_preference('browser.download.folderList', 2)
  options.set_preference('browser.download.dir', os.path.abspath(download_dir))
  options.set_preference('browser.download.manager.showWhenStarting', False)
  options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
  options.set_preference('pdfjs.disabled', True)
  
  ###############################################################################
  def download_pmc_pdf(pmcid_key):
      browser = webdriver.Firefox(options=options)
      pmc_url = ncbi_url + "pmc/articles/" + pmcid_key + "/"
      browser.get(pmc_url)
      try:
          wait = WebDriverWait(browser, 10)
          pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))
  
          # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
          for link in pdf_links:
              href = link.get_attribute('href')
              if '.pdf' in href:
                  # Open the pdf link in a new window
                  browser.execute_script(f'window.open("{href}","_blank");')
                  break
  
          time.sleep(15)  # delay for 10 seconds
          # After the file is downloaded:
          with open(f'{download_dir}/file_mapping.csv', 'a', newline='') as file:
              writer = csv.writer(file)
              writer.writerow([pmcid_key, os.path.basename(href)])
          ## Switch to the new window and close it 
          ## (this is a hack to deal with bowser freezing upon downloading)
          browser.switch_to.window(browser.window_handles[-1])
          browser.close()
  
      except Exception as e:
          print(e)
      finally:
          browser.quit()  # quit the browser outside the loop    
  ###############################################################################
  pmcid = "2788555"   
  download_pmc_pdf(pmcid)
  
  ```
  
  
  ---
  
  ### Answer by username_2 on 2023-07-17. Score: 1
  
  Pubmed Central stores pdfs for all available open access articles on a public-facing ftp server (<https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/>). You can query the database using the OA Web Service API, e.g. with cURL
  
  ```
  # for pubmed id "10026098"
  curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098
  <OA><responseDate>2023-07-17 01:39:13</responseDate><request id="PMC10026098">https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098</request><records returned-count="2" total-count="2"><record id="PMC10026098" citation="Ther Adv Med Oncol. 2023 Mar 18; 15:17588359231161409" license="CC BY-NC" retracted="no"><link format="tgz" updated="2023-03-23 12:10:30" href="ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/83/eb/PMC10026098.tar.gz" /><link format="pdf" updated="2023-03-20 09:34:33" href="ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf" /></record></records></OA>
  
  # get the pdf link from the API (ftp storage)
  curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p'
  ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf
  
  # download the pdf
  pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
  curl -O "$f"
  
  # download many pdfs from a list of ids
  cat list_of_ids.txt
  10026098
  9341183
  
  while read -r id_from_list
  do
      pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
      curl -O "$pdf"
  done < list_of_ids.txt
  
  ```
  
  This approach is preferable to selenium scraping for many reasons, and if you have any problems the NCBI provides documentation (<https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/#indart>) and offers support (you can email them at pubmedcentral@ncbi.nlm.nih.gov).
  
  > **Comments:**
  > **username_0**: Thank you for your time! Unfortunately, it did not work, returning the error `curl: (23) Failed writing received data to disk/application` Furthermore, I've noticed a potential issue. What if the article listed on PubMed is not available on PMC but on other platforms, such as the publisher's website? For example, PMID:37230009  
  > **username_2**: You're welcome; just trying to help. That cURL error is very likely related to your curl install and file permissions (e.g. https://askubuntu.com/questions/1356327/cant-write-to-a-hidden-path-using-curl); you could try using `wget` instead, e.g. `wget -O - https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 2> /dev/null | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p'` should work. And as for articles on Pubmed that aren't on PMC, your question specifies 'articles on PMC', so that is what my answer addresses. If you want to download pdfs from any source, that is a totally different question.  
  > **username_0**: Your solution is crashing because pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')" is empty for the pmcids that I used: 8968431, 8570779, 8290288, 8254498. But it worked for the pmcid that you provided. It seems pdfs corresponding to my pmcids are not present on the public-facing ftp server because when I tried your code with some of the pdfs that are listed there it worked. So, for my use case, browser automation is the only way so far  
  > **username_2**: Ohh! Yep - I see the problem - only Open Access articles are accessible via the API (`<OA><responseDate>2023-07-19 18:55:39</responseDate><request>https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=8968431</request><error code="idIsNotOpenAccess">identifier 'PMC8968431' is not Open Access</error></OA>`). Thanks for taking the time to check!  
  
  ---
  
  