Answers.Body: <p>The idea for this code is to first convert PIDs to PMCIDs, then run the <code>tidypmc</code> in a loop over the PMCIDs. The only problem is that <code>tidypmc</code> failed to retrieve tables from most of the IDs in your example list.</p>
  <pre><code>library(tidyverse)
  library(tidypmc)
  library(httr)
  library(jsonlite)
  example_pids &lt;- c(30024784, 30555165, 30510081, 31688884, 31516032, 28588019, 29286103) %&gt;% as.character()
  
  #-- Convert to PMC ids
  convertPIDtoPMCID &lt;- function(pids) {
      #-------- Make API request
      pids4query &lt;- paste(pids, collapse = &quot;%0D%0A&quot;)
      idconv_req &lt;- paste0(&quot;https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/?ids=&quot;, pids4query, &quot;&amp;idtype=pmid&amp;format=json&amp;versions=no&amp;showaiid=no&amp;tool=&amp;email=&amp;.submit=Submit&quot;)
      pids_json &lt;- GET(idconv_req) %&gt;% content(&quot;text&quot;) %&gt;% fromJSON()
      #-------- Get info from JSON
      pid2pmc &lt;- pids_json<span class="math-container">$records %&gt;% 
          select(pmcid, pmid) %&gt;% 
          as.data.frame()
      rownames(pid2pmc) &lt;- pid2pmc$</span>pmid
      pmcids &lt;- pmcids &lt;- pid2pmc[pids, &quot;pmcid&quot;]
      return(pmcids)
  }
  example_pmcids &lt;- convertPIDtoPMCID(example_pids)
  
  #-- Try to get data with tidypmc
  pub_tables &lt;- lapply(example_pmcids, function(pmc_id) {
      message(&quot;-- Trying &quot;, pmc_id, &quot;...&quot;)
      doc &lt;- tryCatch(pmc_xml(pmc_id), 
                      error = function(e) {
                          message(&quot;------ Failed to recover PMCID&quot;)
                          return(NULL)
                          })
      if(!is.null(doc)) { 
          #-- If succeed, try to get table
          tables &lt;- pmc_table(doc)
          if(!is.null(tables)) {
              #-- If succeed, try to get table name
              table_caps &lt;- pmc_caption(doc) %&gt;%
                  filter(tag == &quot;table&quot;)
              names(tables) &lt;- paste(table_caps<span class="math-container">$label, table_caps$</span>text, sep = &quot; - &quot;)
          }
          return(tables) 
      } else {
          #-- If fail, return NA
          return(NA)
      }
  })
  names(pub_tables) &lt;- example_pids
  
  #-- Inspect results
  pub_tables<span class="math-container">$`30555165`$</span>`Table 1 - Patient demographic and baseline characteristics`
  pub_tables<span class="math-container">$`29286103`$</span>`Table I - Sample summary.`
  </code></pre>
  <p>Tables will require quite a bit of tidying after this. Good luck!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T19:20:05.190Z
Answers.Text: https://github.com/ropensci/tidypmc/issues/3 this answer from the library developer
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T21:29:46.830Z
Answers.Text: `b <-epmc_search(query = 'acute myeloid leukemia drug studies',output = 'parsed',limit = 200)`
      `a <- b %>% select(pmid,pmcid)`
  `a <- a[complete.cases(a),]`
    `c <- a$pmcid`
     `Error in names(tables) <- paste(table_caps$label, table_caps$text, sep = " - ") : 
        'names' attribute [4] must be the same length as the vector [2]`                                           Im getting this error when i run the pub_tables chunk
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T07:26:53.133Z
Answers.Text: after multiple trial and errors when I did this       `#names(tables) <- paste(table_caps$label, table_caps$text, sep = " - ")` i get no error I do see a final list why is it so?
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T09:48:11.943Z
Answers.Text: That line is just for embellishment, you shouldn't have a problem removing it. It adds the title of the table. But it might fail if the metadata doesn't have the table title. So feel free to remove it.
Answers.UserDisplayName: csgroen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T10:56:48.940Z
Answers.Text: this happen when i don't learn R properly i was breaking my head few hours .thank you for the clarification
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T15:44:36.920Z
Answers.Text: Most publications with a Pubmed ID are not open access and you cannot mine the text, so be sure to filter results using OPEN_ACCESS:Y in Europe PMC
Answers.UserDisplayName: Chris S.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-29T08:06:09.847Z
Answers.Text: @csgroen is there any way simple way to save the pub_tables which is a list into tabular form ?
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T08:00:11.837Z
Answers.Text: @krushnachChandra Tabular data represents m observations of n variables. The issue here is that each table has different variables and observations. How can you join tables that have different columns and different rows? You can't. That's why we get a list.
Answers.UserDisplayName: csgroen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-30T09:59:30.583Z
Answers.Text: @csgroen I found a a temporary way just tot store https://stackoverflow.com/questions/56374683/write-list-of-lists-to-separate-csv-files
   yes you are right we can;'t join unless there is some variable which is common across tables. I'm exploring Chris way where he combined them into data-frame which can probably the parsed
Answers.UserDisplayName: kcm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-27T15:23:35.993Z
Answers.LastActivityDate: 2020-07-27T15:23:35.993Z
Answers.UserDisplayName: csgroen
Body: <p>I'm trying to use this <a href="https://ropensci.org/tutorials/europepmc_tutorial/" rel="nofollow noreferrer">europmc r libray</a> where I have a list of pmids to look for. I tried with pubtator but its bit complicated.In Europmc i can all the annotated terms etc.</p>
  <pre><code>library(&quot;europepmc&quot;)
  
  example list of PMIDS
  30024784
  30555165
  30510081
  31688884
  31516032
  28588019
  29286103
  </code></pre>
  <p>Now what I'm doing is looking each ID using epmc_details function which is not the way i would do if i have to look for hundreds.</p>
  <pre><code>epmc_details(ext_id = '30510081')
  </code></pre>
  <p>I have to question how can I run the epmc_details in a loop or some other way where i can look for PMIDS one by one and save the result in a data frame.</p>
  <p>The epmc_details is returned as a list. The structure of the list is as such</p>
  <pre><code>[1] &quot;basic&quot;           &quot;author_details&quot;  &quot;journal_info&quot;    &quot;ftx&quot;             &quot;chemical&quot;        &quot;mesh_topic&quot;      &quot;mesh_qualifiers&quot;
  [8] &quot;comments&quot;        &quot;grants&quot;    
  </code></pre>
  <p>I would only like to save basic,chemical,mesh_topic,mesh_qualifiers in one data frame.</p>
  <p>For example if my first id is this <code>30510081</code> the dataframe should have first column as my ID which is basically <code>basic[1]</code>and rest of the information appended to the next columns.</p>
  <p>such as</p>
  <pre><code> ID chemical mesh_qualifiers mesh_topic gene 
  </code></pre>
  <p>Any suggestion or help would be really and highly appreciated</p>
  <p>I was looking at europmc site through the browser</p>
  <p>this was one of my <a href="https://europepmc.org/article/MED/32700753" rel="nofollow noreferrer">query</a></p>
  <p>when i highlight the keyterms i do see in the abstract itself all the keyterms are getting annotated but when i do the same query search through R I do see empty results as such why there is a difference?</p>
  <pre><code>$chemical
  # A tibble: 0 x 0
  
  $mesh_topic
  # A tibble: 0 x 0
  
  $mesh_qualifiers
  # A tibble: 0 x 0
  </code></pre>
  <p>I found better way of getting data from pubmed using the <a href="https://docs.ropensci.org/tidypmc/index.html" rel="nofollow noreferrer">tidypmc</a> library.</p>
  <pre><code>library(tidypmc)
  doc &lt;- pmc_xml(&quot;PMC6365492&quot;)
  doc
  txt &lt;- pmc_text(doc)
  txt
  
  count(txt, &quot;section&quot;)
  
  cap1 &lt;- pmc_caption(doc)
  filter(cap1, sentence == 1)
  
  tab1 &lt;- pmc_table(doc)
  
  sapply(tab1, nrow)
  
  tab1[[1]]
  attributes(tab1[[2]])
  
  collapse_rows(tab1, na.string=&quot;-&quot;)
  
  library(tibble)
  x &lt;- xml_name(xml_find_all(doc, &quot;//*&quot;))
  tibble(tag=x) %&gt;% count(&quot;tag&quot;)
  
  library(tidytext)
  x1 &lt;- unnest_tokens(txt, word, text) %&gt;%
    anti_join(stop_words) %&gt;%
    filter(!word %in% 1:100)
  #  Joining, by = &quot;word&quot;
  #filter(x1, str_detect(section, &quot;Case description&quot;)) 
  
  
  filter(x1, str_detect(section, &quot;Results&quot;))
  count(a$word)
  
  tbls &lt;- pmc_table(doc)
  map_int(tbls, nrow)
  
  tbls[[1]]
  
  collapse_rows(tbls, na.string=&quot;-&quot;)
  </code></pre>
  <p>But if i understand it can use one PMC id at a time. Again keeping my original question how can i put this in a loop to query lets say i have 100 PMCID and get it result and store in a dataframe.</p>
  <p>After using tidypmc i found that i parse all the publication based on attributes or tags. Such as title,abstract, results etc etc.</p>
  <p>Lets say I'm interested in the table tags information where they have metadata of patients as well as others.So if a paper contain multiple tables I would like to store each of them in a data-frame under the respective publication.Since I have multiple IDs to search and do the save as above mentioned. How to put this through a loop or can it be done without using loop ?</p>
  <p>Any suggestion or help would be really appreciated as always.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-26T08:55:49.003Z
Comments.Text: I think the content of that article are not available in pubmed, you can only download a pdf through some link.. again.. it's my guess.. maybe if you do more you will see this repeats itself
Comments.UserDisplayName: StupidWolf
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T10:56:44.170Z
Comments.Text: @StupidWolf i found tidypmc is better but yes it will only look for article which has PMCID . I have updated my question and query, I would be glad if you can have a look and give your suggestion.
Comments.UserDisplayName: kcm
Tags: <r><public-databases><parsing>
Title: europmc and tidypmc R library for extracting or making metadata from publication
ThreadText: # europmc and tidypmc R library for extracting or making metadata from publication
  
  (Asked by: username_0 on 2020-07-24)
  
  I'm trying to use this [europmc r libray](https://ropensci.org/tutorials/europepmc_tutorial/) where I have a list of pmids to look for. I tried with pubtator but its bit complicated.In Europmc i can all the annotated terms etc.
  
  ```
  library("europepmc")
  
  example list of PMIDS
  30024784
  30555165
  30510081
  31688884
  31516032
  28588019
  29286103
  
  ```
  
  Now what I'm doing is looking each ID using epmc\_details function which is not the way i would do if i have to look for hundreds.
  
  ```
  epmc_details(ext_id = '30510081')
  
  ```
  
  I have to question how can I run the epmc\_details in a loop or some other way where i can look for PMIDS one by one and save the result in a data frame.
  
  The epmc\_details is returned as a list. The structure of the list is as such
  
  ```
  [1] "basic"           "author_details"  "journal_info"    "ftx"             "chemical"        "mesh_topic"      "mesh_qualifiers"
  [8] "comments"        "grants"    
  
  ```
  
  I would only like to save basic,chemical,mesh\_topic,mesh\_qualifiers in one data frame.
  
  For example if my first id is this `30510081` the dataframe should have first column as my ID which is basically `basic[1]`and rest of the information appended to the next columns.
  
  such as
  
  ```
   ID chemical mesh_qualifiers mesh_topic gene 
  
  ```
  
  Any suggestion or help would be really and highly appreciated
  
  I was looking at europmc site through the browser
  
  this was one of my [query](https://europepmc.org/article/MED/32700753)
  
  when i highlight the keyterms i do see in the abstract itself all the keyterms are getting annotated but when i do the same query search through R I do see empty results as such why there is a difference?
  
  ```
  $chemical
  # A tibble: 0 x 0
  
  $mesh_topic
  # A tibble: 0 x 0
  
  $mesh_qualifiers
  # A tibble: 0 x 0
  
  ```
  
  I found better way of getting data from pubmed using the [tidypmc](https://docs.ropensci.org/tidypmc/index.html) library.
  
  ```
  library(tidypmc)
  doc <- pmc_xml("PMC6365492")
  doc
  txt <- pmc_text(doc)
  txt
  
  count(txt, "section")
  
  cap1 <- pmc_caption(doc)
  filter(cap1, sentence == 1)
  
  tab1 <- pmc_table(doc)
  
  sapply(tab1, nrow)
  
  tab1[[1]]
  attributes(tab1[[2]])
  
  collapse_rows(tab1, na.string="-")
  
  library(tibble)
  x <- xml_name(xml_find_all(doc, "//*"))
  tibble(tag=x) %>% count("tag")
  
  library(tidytext)
  x1 <- unnest_tokens(txt, word, text) %>%
    anti_join(stop_words) %>%
    filter(!word %in% 1:100)
  #  Joining, by = "word"
  #filter(x1, str_detect(section, "Case description")) 
  
  filter(x1, str_detect(section, "Results"))
  count(a$word)
  
  tbls <- pmc_table(doc)
  map_int(tbls, nrow)
  
  tbls[[1]]
  
  collapse_rows(tbls, na.string="-")
  
  ```
  
  But if i understand it can use one PMC id at a time. Again keeping my original question how can i put this in a loop to query lets say i have 100 PMCID and get it result and store in a dataframe.
  
  After using tidypmc i found that i parse all the publication based on attributes or tags. Such as title,abstract, results etc etc.
  
  Lets say I'm interested in the table tags information where they have metadata of patients as well as others.So if a paper contain multiple tables I would like to store each of them in a data-frame under the respective publication.Since I have multiple IDs to search and do the save as above mentioned. How to put this through a loop or can it be done without using loop ?
  
  Any suggestion or help would be really appreciated as always.
  
  **Tags:** r, public-databases, parsing
  
  > **Comments:**
  > **username_1**: I think the content of that article are not available in pubmed, you can only download a pdf through some link.. again.. it's my guess.. maybe if you do more you will see this repeats itself  
  > **username_0**: @username_1 i found tidypmc is better but yes it will only look for article which has PMCID . I have updated my question and query, I would be glad if you can have a look and give your suggestion.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-27. Score: 3 | Accepted answer
  
  The idea for this code is to first convert PIDs to PMCIDs, then run the `tidypmc` in a loop over the PMCIDs. The only problem is that `tidypmc` failed to retrieve tables from most of the IDs in your example list.
  
  ```
  library(tidyverse)
  library(tidypmc)
  library(httr)
  library(jsonlite)
  example_pids <- c(30024784, 30555165, 30510081, 31688884, 31516032, 28588019, 29286103) %>% as.character()
  
  #-- Convert to PMC ids
  convertPIDtoPMCID <- function(pids) {
      #-------- Make API request
      pids4query <- paste(pids, collapse = "%0D%0A")
      idconv_req <- paste0("https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/?ids=", pids4query, "&idtype=pmid&format=json&versions=no&showaiid=no&tool=&email=&.submit=Submit")
      pids_json <- GET(idconv_req) %>% content("text") %>% fromJSON()
      #-------- Get info from JSON
      pid2pmc <- pids_json$records %>% 
          select(pmcid, pmid) %>% 
          as.data.frame()
      rownames(pid2pmc) <- pid2pmc$pmid
      pmcids <- pmcids <- pid2pmc[pids, "pmcid"]
      return(pmcids)
  }
  example_pmcids <- convertPIDtoPMCID(example_pids)
  
  #-- Try to get data with tidypmc
  pub_tables <- lapply(example_pmcids, function(pmc_id) {
      message("-- Trying ", pmc_id, "...")
      doc <- tryCatch(pmc_xml(pmc_id), 
                      error = function(e) {
                          message("------ Failed to recover PMCID")
                          return(NULL)
                          })
      if(!is.null(doc)) { 
          #-- If succeed, try to get table
          tables <- pmc_table(doc)
          if(!is.null(tables)) {
              #-- If succeed, try to get table name
              table_caps <- pmc_caption(doc) %>%
                  filter(tag == "table")
              names(tables) <- paste(table_caps$label, table_caps$text, sep = " - ")
          }
          return(tables) 
      } else {
          #-- If fail, return NA
          return(NA)
      }
  })
  names(pub_tables) <- example_pids
  
  #-- Inspect results
  pub_tables$`30555165`$`Table 1 - Patient demographic and baseline characteristics`
  pub_tables$`29286103`$`Table I - Sample summary.`
  
  ```
  
  Tables will require quite a bit of tidying after this. Good luck!
  
  > **Comments:**
  > **username_0**: https://github.com/ropensci/tidypmc/issues/3 this answer from the library developer  
  > **username_0**: `b <-epmc_search(query = 'acute myeloid leukemia drug studies',output = 'parsed',limit = 200)`     `a <- b %>% select(pmid,pmcid)` `a <- a[complete.cases(a),]`   `c <- a$pmcid`    `Error in names(tables) <- paste(table_caps$label, table_caps$text, sep = " - ") :        'names' attribute [4] must be the same length as the vector [2]`                                           Im getting this error when i run the pub_tables chunk  
  > **username_0**: after multiple trial and errors when I did this       `#names(tables) <- paste(table_caps$label, table_caps$text, sep = " - ")` i get no error I do see a final list why is it so?  
  > **username_2**: That line is just for embellishment, you shouldn't have a problem removing it. It adds the title of the table. But it might fail if the metadata doesn't have the table title. So feel free to remove it.  
  > **username_0**: this happen when i don't learn R properly i was breaking my head few hours .thank you for the clarification  
  > **username_3**: Most publications with a Pubmed ID are not open access and you cannot mine the text, so be sure to filter results using OPEN_ACCESS:Y in Europe PMC  
  > **username_0**: @username_2 is there any way simple way to save the pub_tables which is a list into tabular form ?  
  > **username_2**: @krushnachChandra Tabular data represents m observations of n variables. The issue here is that each table has different variables and observations. How can you join tables that have different columns and different rows? You can't. That's why we get a list.  
  > **username_0**: @username_2 I found a a temporary way just tot store https://stackoverflow.com/questions/56374683/write-list-of-lists-to-separate-csv-files  yes you are right we can;'t join unless there is some variable which is common across tables. I'm exploring username_3 way where he combined them into data-frame which can probably the parsed  
  
  ---
  
  