Answers.Body: <p>It looks like the issue comes from one of the fields I was filtering for. When I remove that filter the code works again. Still need to investigate why that happened.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-24T17:34:20.107Z
Answers.Text: Could you kindly expand on the answer please, specifically which field?
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T19:23:29.907Z
Answers.Text: It really isn't clear. Possibly something in self-reported data, field 20002. I ended up using a workaround by filtering for fields separately and joining them with df.union(df2)
Answers.UserDisplayName: eclarette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-24T17:01:36.297Z
Answers.LastActivityDate: 2023-07-24T17:01:36.297Z
Answers.UserDisplayName: eclarette
Answers.Body: <p>Generally remote biological servers, for example UCSC genome server, will not perform SQL queries for a free account. They'll allow downloads, but not queries. That is probably the same reason here.</p>
  <p>The reason is simply these are often huge databases and queries can take up a lot of their computational resources, particularly if everyone is doing them. I don't know BioBank UK, but suggest its common practice.</p>
  <p>If thats true the only difference with Biobank and UCSC is the UCSC server refusals are stated very clearly, &quot;this operation ain't permitted for free accounts&quot; (paraphrasing). What might be happening in an API error is not being picked up and relayed to you. Depends how the API is written so it's just reporting a timeout.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T22:11:00.073Z
Answers.Text: The account is paid, not free
Answers.UserDisplayName: eclarette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T22:14:05.727Z
Answers.Text: @eclarette If you login directly you will receive a more informative message than going through an API, i.e. perform the SQL query there. The API isn't reporting the error message.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T04:34:20.253Z
Answers.LastActivityDate: 2023-07-23T12:38:09.787Z
Answers.LastEditDate: 2023-07-23T12:38:09.787Z
Answers.UserDisplayName: M__
Body: <p>I'm working with UK Biobank data. I made a cohort with the Cohort Browser and tried to extract the data with this code  <code>df = participant.retrieve_fields(names=field_names, filter_sql = cohort.sql, coding_values = &quot;replace&quot;, engine=dxdata.connect())</code>.</p>
  <p>This is part of the text of the error: <code>Py4JJavaError: An error occurred while calling 04100.showString. org.apache.spark. SparkException: Exception thrown in awaitResult:</code>. The full error message is cut off.</p>
  <p>However, if I remove the filter - <code>df = participant.retrieve_fields(names=field_names, coding_values = &quot;replace&quot;, engine=dxdata.connect())</code> – there is no issue.</p>
  <p>What kind of error is this and why would filtering for a cohort help?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-22T01:36:15.627Z
Comments.Text: Thanks for the question and the edit. This is Python API for a Biobank - looks very interesting. Just to clarify the implication is you can connect to Biobank ok, but when you add a filter this stops the operation. However the code that connects okay has not been shown. Is that all correct?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-23T03:23:21.120Z
Comments.Text: Yes, that's correct
Comments.UserDisplayName: eclarette
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-23T04:27:58.500Z
Comments.Text: Okay, it's a bit hard without knowing the underlying code.
Comments.UserDisplayName: M__
Tags: <python>
Title: Filtering cohorts in PySpark returns Py4JJavaError
ThreadText: # Filtering cohorts in PySpark returns Py4JJavaError
  
  (Asked by: username_0 on 2023-07-21)
  
  I'm working with UK Biobank data. I made a cohort with the Cohort Browser and tried to extract the data with this code `df = participant.retrieve_fields(names=field_names, filter_sql = cohort.sql, coding_values = "replace", engine=dxdata.connect())`.
  
  This is part of the text of the error: `Py4JJavaError: An error occurred while calling 04100.showString. org.apache.spark. SparkException: Exception thrown in awaitResult:`. The full error message is cut off.
  
  However, if I remove the filter - `df = participant.retrieve_fields(names=field_names, coding_values = "replace", engine=dxdata.connect())` – there is no issue.
  
  What kind of error is this and why would filtering for a cohort help?
  
  **Tags:** python
  
  > **Comments:**
  > **username_1**: Thanks for the question and the edit. This is Python API for a Biobank - looks very interesting. Just to clarify the implication is you can connect to Biobank ok, but when you add a filter this stops the operation. However the code that connects okay has not been shown. Is that all correct?  
  > **username_0**: Yes, that's correct  
  > **username_1**: Okay, it's a bit hard without knowing the underlying code.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-07-24. Score: 1
  
  It looks like the issue comes from one of the fields I was filtering for. When I remove that filter the code works again. Still need to investigate why that happened.
  
  > **Comments:**
  > **username_1**: Could you kindly expand on the answer please, specifically which field?  
  > **username_0**: It really isn't clear. Possibly something in self-reported data, field 20002. I ended up using a workaround by filtering for fields separately and joining them with df.union(df2)  
  
  ---
  
  ### Answer by username_1 on 2023-07-23. Score: 0
  
  Generally remote biological servers, for example UCSC genome server, will not perform SQL queries for a free account. They'll allow downloads, but not queries. That is probably the same reason here.
  
  The reason is simply these are often huge databases and queries can take up a lot of their computational resources, particularly if everyone is doing them. I don't know BioBank UK, but suggest its common practice.
  
  If thats true the only difference with Biobank and UCSC is the UCSC server refusals are stated very clearly, "this operation ain't permitted for free accounts" (paraphrasing). What might be happening in an API error is not being picked up and relayed to you. Depends how the API is written so it's just reporting a timeout.
  
  > **Comments:**
  > **username_0**: The account is paid, not free  
  > **username_1**: @username_0 If you login directly you will receive a more informative message than going through an API, i.e. perform the SQL query there. The API isn't reporting the error message.  
  
  ---
  
  