Answers.Body: <p>Okay, its done. What you are looking for is as follows:</p>
  <pre><code>import requests
  from pathlib import Path
  
  # Test code    
  def get_all_keys(d):
      for key, value in d.items():
          yield key
          if isinstance(value, dict):
              yield from get_all_keys(value)
  
  def requestIt(links): 
      cancer_all = requests.get(links)
      return cancer_all.json() 
  
  if __name__ == '__main__':
  
      path = Path('/PathToDir/test', 'link.html')
      with open (path, 'r') as fin:
          links = fin.readlines()
          
          
      data = list(map(requestIt, links))
  
      # Test code
      for x in get_all_keys(data[0]['_embedded']['terms'][0]):
          print(x)
  
      # Answer code
      for i,d in enumerate(data):
          print (f'Link {i + 1} ') 
          for obo_id in d['_embedded']['terms']:
              print (obo_id['obo_id'])
  </code></pre>
  <p><strong>Output</strong></p>
  <blockquote>
  <p><strong>Link 1</strong>
  NCIT:C8263
  NCIT:C9163
  NCIT:C5630</p>
  </blockquote>
  <blockquote>
  <p><strong>Link 2</strong>
  NCIT:C7175
  NCIT:C68700
  NCIT:C68696
  NCIT:C68697
  NCIT:C68698
  NCIT:C68699
  NCIT:C9155
  NCIT:C9288
  NCIT:C9287
  NCIT:C9019
  NCIT:C9018
  NCIT:C36055
  NCIT:C36058
  NCIT:C36057
  NCIT:C36056
  NCIT:C38377
  NCIT:C122688
  NCIT:C156718
  NCIT:C156719
  NCIT:C169107</p>
  </blockquote>
  <blockquote>
  <p><strong>Link 3</strong>
  NCIT:C7056
  NCIT:C131911
  NCIT:C165799
  NCIT:C68684
  NCIT:C4998
  NCIT:C5089
  NCIT:C5095
  NCIT:C8154
  NCIT:C8604
  NCIT:C127840
  NCIT:C8489
  NCIT:C8852
  NCIT:C69144
  NCIT:C115349
  NCIT:C115367
  NCIT:C165240
  NCIT:C8926
  NCIT:C160230
  NCIT:C160231
  NCIT:C182035</p>
  </blockquote>
  <p>Where <strong>Link 1</strong> is ...</p>
  <pre><code>https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
  </code></pre>
  <p>... and so on.</p>
  <p>If the code is listed <code># Test code</code> that is what is used to interrogate the data structure. This doesn't contribute to your answer.</p>
  <p>However, given the way I've done it, it's important to understand the data structure. Thus output of <code># Test code</code> is at the base of this post. If you need another part of the data structure you simply switch <code>id_odo</code> to whatever else you are looking for (at the base of this post).</p>
  <p>There will be an easy to use OOP method based out the <code>data</code> object. I did't want to spend time investigating the object ... the question is specifically about <code>obo_id</code> so it was easier to go for that.</p>
  <p><strong>Test code output</strong></p>
  <pre><code>iri
  lang
  description
  synonyms
  annotation
  Contributing_Source
  Display_Name
  Legacy Concept Name
  Neoplastic_Status
  Preferred_Name
  Semantic_Type
  UMLS_CUI
  code
  label
  ontology_name
  ontology_prefix
  ontology_iri
  is_obsolete
  term_replaced_by
  is_defining_ontology
  has_children
  is_root
  short_form
  obo_id  # &lt;- its here
  in_subset
  obo_definition_citation
  obo_xref
  obo_synonym
  is_preferred_root
  _links
  self
  href
  parents
  href
  ancestors
  href
  hierarchicalParents
  href
  hierarchicalAncestors
  href
  jstree
  href
  graph
  href
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T18:20:12.357Z
Answers.LastActivityDate: 2023-03-11T19:41:02.550Z
Answers.LastEditDate: 2023-03-11T19:41:02.550Z
Answers.UserDisplayName: M__
Answers.Body: <p>@Megha
  The requestIt(link) function returns a list of dictionary objects.
  So your &quot;data&quot; variable is a list of dictionary objects.</p>
  <p>Every item on this list (a dict) has a key named &quot;_embedded&quot; whose value is also a dictionary.
  This dictionary (the _embedded) has a key named &quot;terms&quot; which is a list of dictionaries.
  The obo_ids you need are inside the dictionaries of the &quot;terms&quot; list.</p>
  <p>The problem with your code was that you were trying to access the &quot;obo_id&quot; key on the top level dictionary.</p>
  <p>So a simple solution would be like this:</p>
  <pre><code>with open ('urls.txt', 'r') as fin:
      links = fin.readlines()
  resps = [requestIt(link) for link in links] # this one holds all the data
  obo_ids = []
  for resp in resps:                          # for every response we got
      terms = resp['_embedded']['terms']      # choose the terms list
      for term in terms:                      # for every dict in this list
          obo_ids.append(term['obo_id'])      # extract obo_id and append it to our answer
  print(obo_ids)                              # do stuff with obo_ids
  </code></pre>
  <p>You should probably perform some exception handling inside the requestIt() function to handle broken links etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-09T17:24:21.197Z
Answers.Text: Welcome @punxx and this looks cool
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-09T17:17:50.883Z
Answers.LastActivityDate: 2023-03-09T17:17:50.883Z
Answers.UserDisplayName: punxx
Answers.Body: <p>@Megha, I have not looked through the code in too much detail, however.</p>
  <pre><code>obo = obj[&quot;obo_id&quot;] for obj in data
  </code></pre>
  <p>This isn't formal Python code. The correction would simply be to place this in a comprehension ...</p>
  <pre><code>obo = [obj[&quot;obo_id&quot;] for obj in data]
  </code></pre>
  <p>... unless you want an iterator.</p>
  <p>I think there is a deep issue with <code>obj[&quot;obo_id&quot;]</code>. If the syntax still throws an exception could you provide a print out of either <code>data</code> please, or else ....</p>
  <pre><code>obj for obj in data:
      print (obj)
      print (obj.keys)
  </code></pre>
  <p>The <code>data</code> data structure for the output of <code>get.requests</code> followed by <code>.json()</code> is kinda key here</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T10:08:33.247Z
Answers.Text: Thank you for the answer. Got the same error with obo = [obj["obo_id"] for obj in data]. Then I tried `obj for obj in data:
      print (obj)
      print (obj.keys)`. Got an error `obj for obj in data:
          ^
  SyntaxError: invalid syntax
Answers.UserDisplayName: Megha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T10:10:16.980Z
Answers.Text: Is there any way I can run the code (the cancer_id [ ] one)that worked perfectly for one link in a for loop for all the links?
Answers.UserDisplayName: Megha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T13:46:11.230Z
Answers.Text: Thanks @Megha. Did you get a print out for `print (obj)`? Then an error for `print (obj.keys)`? The idea is to check the output before the error.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T13:51:20.590Z
Answers.Text: Hi got the same error for both print statements?and there was no output, just this error
Answers.UserDisplayName: Megha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T15:44:19.070Z
Answers.Text: Hi @Megha over the weekend I will replicate your code and investigate, the object, json, dictionary ... (whatever it is) output. I should get I definitive answer, but I'm afraid I can't do that during this week due to work stuff.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-08T04:12:18.737Z
Answers.Text: @Megha `cancer_all.json()` is an object so its behaviour is very different. `obj["obo_id"]` is not going to work because you'll need to reference the instance. It's possible something like `str(obj)['repo_url']` might work. I'll need to check and examine all instances etc ...
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-06T19:04:56.693Z
Answers.LastActivityDate: 2023-03-06T19:04:56.693Z
Answers.UserDisplayName: M__
Body: <p>Hi I am trying to get the cancer ontologies (obo_id and label) from EBI-OLS. Earlier I have used the below code to get the <code>obo_id</code> terms and <code>labels</code></p>
  <pre><code>descendants = requests.get(
      &quot;https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants&quot;
  )
  </code></pre>
  <p>As there are more than 1 page so I created <code>total_pages</code></p>
  <pre><code>total_pages = descendants.json()[&quot;page&quot;][&quot;totalElements&quot;] // 1000 + 1
  </code></pre>
  <pre><code>cancer_id = []
  
  for i in range(total_pages):
      ## Check the number of id's present in 'totalElements'
      des = {&quot;page&quot;: i, &quot;size&quot;: descendants.json()[&quot;page&quot;][&quot;totalElements&quot;]}
      ## Add the number of 'totalElements' and 'pages' in the new python request for descendants
      cancer_efo = requests.get(
          &quot;https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants&quot;,
          params=des,
      )
      ## Get the number of obo_id's present in the cancer_efo.json
      obo = [obj[&quot;obo_id&quot;] for obj in cancer_efo.json()[&quot;_embedded&quot;][&quot;terms&quot;]]
      ## Get the number of cancer labels present in the cancer_efo.json
      label = [obj[&quot;label&quot;] for obj in cancer_efo.json()[&quot;_embedded&quot;][&quot;terms&quot;]]
      data = zip(obo, label)
      cancer_id.extend(data)
  </code></pre>
  <p>The above code worked pretty well with one link
  Now the issue is that I want to get ontologies for 37 descendent links and maximum links have more than 1 page. I want to get one file (cancer_id) containing all obo_id's and labels from these all 37 links.</p>
  <p>I made a separate file called <code>list.txt</code> (Right now I am copying only 3 here)</p>
  <p><code>list.txt</code></p>
  <pre><code>https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
  https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3171/descendants
  https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3457/descendants
  </code></pre>
  <p>I tried the below code but it is not working</p>
  <pre><code>import requests
  
  with open ('list.txt', 'r') as fin:
      links = fin.readlines()
  
  def requestIt(links): 
      cancer_all = requests.get(links)
      return cancer_all.json() 
  
  data = list(map(requestIt, links))
  
  obo = obj[&quot;obo_id&quot;] for obj in data
  </code></pre>
  <p>The error is</p>
  <pre><code>obo = obj[&quot;obo_id&quot;] for obj in data
                          ^
  SyntaxError: invalid syntax
  </code></pre>
  
Tags: <python><api><pandas><ontology>
Title: How to run python request for list of url's with multiple page numbers?
ThreadText: # How to run python request for list of url's with multiple page numbers?
  
  (Asked by: username_0 on 2023-03-06)
  
  Hi I am trying to get the cancer ontologies (obo\_id and label) from EBI-OLS. Earlier I have used the below code to get the `obo_id` terms and `labels`
  
  ```
  descendants = requests.get(
      "https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants"
  )
  
  ```
  
  As there are more than 1 page so I created `total_pages`
  
  ```
  total_pages = descendants.json()["page"]["totalElements"] // 1000 + 1
  
  ```
  
  ```
  cancer_id = []
  
  for i in range(total_pages):
      ## Check the number of id's present in 'totalElements'
      des = {"page": i, "size": descendants.json()["page"]["totalElements"]}
      ## Add the number of 'totalElements' and 'pages' in the new python request for descendants
      cancer_efo = requests.get(
          "https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants",
          params=des,
      )
      ## Get the number of obo_id's present in the cancer_efo.json
      obo = [obj["obo_id"] for obj in cancer_efo.json()["_embedded"]["terms"]]
      ## Get the number of cancer labels present in the cancer_efo.json
      label = [obj["label"] for obj in cancer_efo.json()["_embedded"]["terms"]]
      data = zip(obo, label)
      cancer_id.extend(data)
  
  ```
  
  The above code worked pretty well with one link
  Now the issue is that I want to get ontologies for 37 descendent links and maximum links have more than 1 page. I want to get one file (cancer\_id) containing all obo\_id's and labels from these all 37 links.
  
  I made a separate file called `list.txt` (Right now I am copying only 3 here)
  
  `list.txt`
  
  ```
  https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
  https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3171/descendants
  https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3457/descendants
  
  ```
  
  I tried the below code but it is not working
  
  ```
  import requests
  
  with open ('list.txt', 'r') as fin:
      links = fin.readlines()
  
  def requestIt(links): 
      cancer_all = requests.get(links)
      return cancer_all.json() 
  
  data = list(map(requestIt, links))
  
  obo = obj["obo_id"] for obj in data
  
  ```
  
  The error is
  
  ```
  obo = obj["obo_id"] for obj in data
                          ^
  SyntaxError: invalid syntax
  
  ```
  
  **Tags:** python, api, pandas, ontology
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-03-11. Score: 1 | Accepted answer
  
  Okay, its done. What you are looking for is as follows:
  
  ```
  import requests
  from pathlib import Path
  
  # Test code    
  def get_all_keys(d):
      for key, value in d.items():
          yield key
          if isinstance(value, dict):
              yield from get_all_keys(value)
  
  def requestIt(links): 
      cancer_all = requests.get(links)
      return cancer_all.json() 
  
  if __name__ == '__main__':
  
      path = Path('/PathToDir/test', 'link.html')
      with open (path, 'r') as fin:
          links = fin.readlines()
          
          
      data = list(map(requestIt, links))
  
      # Test code
      for x in get_all_keys(data[0]['_embedded']['terms'][0]):
          print(x)
  
      # Answer code
      for i,d in enumerate(data):
          print (f'Link {i + 1} ') 
          for obo_id in d['_embedded']['terms']:
              print (obo_id['obo_id'])
  
  ```
  
  **Output**
  
  > **Link 1**
  > NCIT:C8263
  > NCIT:C9163
  > NCIT:C5630
  
  > **Link 2**
  > NCIT:C7175
  > NCIT:C68700
  > NCIT:C68696
  > NCIT:C68697
  > NCIT:C68698
  > NCIT:C68699
  > NCIT:C9155
  > NCIT:C9288
  > NCIT:C9287
  > NCIT:C9019
  > NCIT:C9018
  > NCIT:C36055
  > NCIT:C36058
  > NCIT:C36057
  > NCIT:C36056
  > NCIT:C38377
  > NCIT:C122688
  > NCIT:C156718
  > NCIT:C156719
  > NCIT:C169107
  
  > **Link 3**
  > NCIT:C7056
  > NCIT:C131911
  > NCIT:C165799
  > NCIT:C68684
  > NCIT:C4998
  > NCIT:C5089
  > NCIT:C5095
  > NCIT:C8154
  > NCIT:C8604
  > NCIT:C127840
  > NCIT:C8489
  > NCIT:C8852
  > NCIT:C69144
  > NCIT:C115349
  > NCIT:C115367
  > NCIT:C165240
  > NCIT:C8926
  > NCIT:C160230
  > NCIT:C160231
  > NCIT:C182035
  
  Where **Link 1** is ...
  
  ```
  https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
  
  ```
  
  ... and so on.
  
  If the code is listed `# Test code` that is what is used to interrogate the data structure. This doesn't contribute to your answer.
  
  However, given the way I've done it, it's important to understand the data structure. Thus output of `# Test code` is at the base of this post. If you need another part of the data structure you simply switch `id_odo` to whatever else you are looking for (at the base of this post).
  
  There will be an easy to use OOP method based out the `data` object. I did't want to spend time investigating the object ... the question is specifically about `obo_id` so it was easier to go for that.
  
  **Test code output**
  
  ```
  iri
  lang
  description
  synonyms
  annotation
  Contributing_Source
  Display_Name
  Legacy Concept Name
  Neoplastic_Status
  Preferred_Name
  Semantic_Type
  UMLS_CUI
  code
  label
  ontology_name
  ontology_prefix
  ontology_iri
  is_obsolete
  term_replaced_by
  is_defining_ontology
  has_children
  is_root
  short_form
  obo_id  # <- its here
  in_subset
  obo_definition_citation
  obo_xref
  obo_synonym
  is_preferred_root
  _links
  self
  href
  parents
  href
  ancestors
  href
  hierarchicalParents
  href
  hierarchicalAncestors
  href
  jstree
  href
  graph
  href
  
  ```
  
  
  ---
  
  ### Answer by username_2 on 2023-03-09. Score: 2
  
  @username_0
  The requestIt(link) function returns a list of dictionary objects.
  So your "data" variable is a list of dictionary objects.
  
  Every item on this list (a dict) has a key named "\_embedded" whose value is also a dictionary.
  This dictionary (the \_embedded) has a key named "terms" which is a list of dictionaries.
  The obo\_ids you need are inside the dictionaries of the "terms" list.
  
  The problem with your code was that you were trying to access the "obo\_id" key on the top level dictionary.
  
  So a simple solution would be like this:
  
  ```
  with open ('urls.txt', 'r') as fin:
      links = fin.readlines()
  resps = [requestIt(link) for link in links] # this one holds all the data
  obo_ids = []
  for resp in resps:                          # for every response we got
      terms = resp['_embedded']['terms']      # choose the terms list
      for term in terms:                      # for every dict in this list
          obo_ids.append(term['obo_id'])      # extract obo_id and append it to our answer
  print(obo_ids)                              # do stuff with obo_ids
  
  ```
  
  You should probably perform some exception handling inside the requestIt() function to handle broken links etc.
  
  > **Comments:**
  > **username_1**: Welcome @username_2 and this looks cool  
  
  ---
  
  ### Answer by username_1 on 2023-03-06. Score: 1
  
  @username_0, I have not looked through the code in too much detail, however.
  
  ```
  obo = obj["obo_id"] for obj in data
  
  ```
  
  This isn't formal Python code. The correction would simply be to place this in a comprehension ...
  
  ```
  obo = [obj["obo_id"] for obj in data]
  
  ```
  
  ... unless you want an iterator.
  
  I think there is a deep issue with `obj["obo_id"]`. If the syntax still throws an exception could you provide a print out of either `data` please, or else ....
  
  ```
  obj for obj in data:
      print (obj)
      print (obj.keys)
  
  ```
  
  The `data` data structure for the output of `get.requests` followed by `.json()` is kinda key here
  
  > **Comments:**
  > **username_0**: Thank you for the answer. Got the same error with obo = [obj["obo_id"] for obj in data]. Then I tried `obj for obj in data:     print (obj)     print (obj.keys)`. Got an error `obj for obj in data:         ^ SyntaxError: invalid syntax  
  > **username_0**: Is there any way I can run the code (the cancer_id [ ] one)that worked perfectly for one link in a for loop for all the links?  
  > **username_1**: Thanks @username_0. Did you get a print out for `print (obj)`? Then an error for `print (obj.keys)`? The idea is to check the output before the error.  
  > **username_0**: Hi got the same error for both print statements?and there was no output, just this error  
  > **username_1**: Hi @username_0 over the weekend I will replicate your code and investigate, the object, json, dictionary ... (whatever it is) output. I should get I definitive answer, but I'm afraid I can't do that during this week due to work stuff.  
  > **username_1**: @username_0 `cancer_all.json()` is an object so its behaviour is very different. `obj["obo_id"]` is not going to work because you'll need to reference the instance. It's possible something like `str(obj)['repo_url']` might work. I'll need to check and examine all instances etc ...  
  
  ---
  
  