Answers.Body: <p>30MB isn't a lot of data at all, given the issue is the EBI servers and not your institutes provision - you would have noticed this before on other projects if it was an institute issue. It therefore suggests someone else is hogging the EBI capacity and there is no provision to bypass this.</p>
  <p>The best solution is contact them directly</p>
  <blockquote>
  <p>ecovid19@ebi.ac.uk</p>
  </blockquote>
  <p>... and state the download size and download speed.</p>
  <p>There could be a bot sucking up the database (that is what it sounds like), which would be easy to write given FTP is permitted. They will have the ability to block an IP and NCBI have done this before albeit a long time ago.</p>
  <p>Finally filling the user experience form would help EBI <a href="https://docs.google.com/forms/d/e/1FAIpQLSeF8YcPi6xxCNQQ6KJFVEHBMBAcX-mDNiWFXXoN1a5wWc-0Zw/viewform" rel="nofollow noreferrer">which is here</a>. However the issue you described suggests escalation. NCBI are very responsive and I would be surprised if EBI did not respond quickly.</p>
  <p>Good luck with the project, sounds interesting.</p>
  <p>Generic response below if its just traditional over-demand for the service.</p>
  <hr />
  <p>I suspect that the beautiful database EBI has provided (and it is amazing) has not anticipated the level of demand for the database size. They are far from being alone.</p>
  <p>Modern web architectures, such as Spotify, dedicate enormous effort to matching demand with service provision, notably using Kubernetes. When the server is overloaded they will simply duplicate their core database onto additional machines, which they will be buying in from AWS/GCP. It can even be done dynamically without intervention. NCBI are certainly heading in that direction.</p>
  <p>If the EBI wrote this under Kubernetes (it could be a monolith however), the limiting factor is the budget in computer resource allocation (again you can increase Kubernetes provision dynamically). However, if its a monolith this will not be easy and again ultimately someone has pay, i.e. EBI.</p>
  <p>Regardless, the fact you can use FTP means EBI are hosting their own service, GCP doesn't permit this and AWS is almost certainly similar. This would mean they have to physically provided additional in-house machines. I visited EBI some time ago and at that time the then director went to some lengths describing how focused EBI were on matching their in-house storage capacity to exponential rise in sequence data. Sudden popularity of a webserver is not trivial issue if it hasn't been anticipated.</p>
  <hr />
  <p><strong>Alternatives</strong>
  Using the NCBI service COVID service is the alternative if there is an urgent need to get the project moving. You may also wish to use GISAID <a href="https://www.gisaid.org" rel="nofollow noreferrer">here</a>.</p>
  <p>If you want to use the NCBI service, I provided the detailed instructions here: <a href="https://bioinformatics.stackexchange.com/questions/18642/how-to-programatically-download-sars-cov-2-fasta-from-ncbi-genbank-via-api/18643#18643">How to programatically download SARS-CoV-2 fasta from NCBI/Genbank via API?</a></p>
  <p>It is however important to understand the possible underlying causes from EBI. I do recall the then EBI director stating their biggest issue was a user simply switching to Genbank, which from their point of view isn't cool given the amount effort going into service provision.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T22:46:08.137Z
Answers.Text: Thank you for the detailed explanation and alternatives. I have found an alternative to EBI from the answer you have suggested. It was quick to download from NCBI.
  
  I will report/inform EBI about the data speed issues and update their response here.
Answers.UserDisplayName: Rajesh M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-11T00:00:21.483Z
Answers.Text: You are welcome. It is a pity for EBI because I think they try hard, however it is also a learning curve for them
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-11T12:32:22.867Z
Answers.Text: I have gotten a response from EBI saying, "They are aware of these downloading issues with few files and they have allocated my ticket to a colleague and help me resolve it asap"
Answers.UserDisplayName: Rajesh M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-11T12:46:42.273Z
Answers.Text: That's good, fingers crossed. They will be sensitive about you switching to Genbank.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-09T13:41:33.180Z
Answers.LastActivityDate: 2022-03-09T15:52:29.503Z
Answers.LastEditDate: 2022-03-09T15:52:29.503Z
Answers.UserDisplayName: M__
Body: <p>I am working on developing a tool to classify covid variants. So I am trying to download SARS-CoV-2 sequence data along with lineage annotation from <a href="https://www.covid19dataportal.org/sequences" rel="nofollow noreferrer">Covid-19 Data Portal</a> (I could not find a better source that provides both sequence and lineage annotated data.)</p>
  <p>Since it is a very large data (given 4,050,433 samples), my idea is to first download the metadata (<code>.tsv</code> file) with accession ids and respective lineages from the portal, filter them to keep only the variants I am interested in, and pass the accession_ids as a text file that I would like to download to <a href="https://www.covid19dataportal.org/bulk-downloads" rel="nofollow noreferrer">CDP File downloader</a> as suggested in the portal.</p>
  <p>However, downloading metadata <code>.tsv</code> file from the portal is taking forever (download speed is in bytes sometimes). I have tried downloading directly as a file, using <code>wget</code> and also using <code>curl</code>. I am not sure if there is a way to download it quicker since it is losing connection in between and restarting the download. Just as a side note, the sequence data is downloading pretty fast at 4+Mbps.</p>
  <p>I hope my question is clear. Any suggestions/tips on how can I make this happen is very much appreciated. Thanks in advance.</p>
  
Tags: <covid-19><data-download><api><curl>
Title: Taking too long to download metadata file (estimated size 30MB) from Covid-19 Data Portal
ThreadText: # Taking too long to download metadata file (estimated size 30MB) from Covid-19 Data Portal
  
  (Asked by: username_0 on 2022-03-09)
  
  I am working on developing a tool to classify covid variants. So I am trying to download SARS-CoV-2 sequence data along with lineage annotation from [Covid-19 Data Portal](https://www.covid19dataportal.org/sequences) (I could not find a better source that provides both sequence and lineage annotated data.)
  
  Since it is a very large data (given 4,050,433 samples), my idea is to first download the metadata (`.tsv` file) with accession ids and respective lineages from the portal, filter them to keep only the variants I am interested in, and pass the accession\_ids as a text file that I would like to download to [CDP File downloader](https://www.covid19dataportal.org/bulk-downloads) as suggested in the portal.
  
  However, downloading metadata `.tsv` file from the portal is taking forever (download speed is in bytes sometimes). I have tried downloading directly as a file, using `wget` and also using `curl`. I am not sure if there is a way to download it quicker since it is losing connection in between and restarting the download. Just as a side note, the sequence data is downloading pretty fast at 4+Mbps.
  
  I hope my question is clear. Any suggestions/tips on how can I make this happen is very much appreciated. Thanks in advance.
  
  **Tags:** covid-19, data-download, api, curl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-09. Score: 0 | Accepted answer
  
  30MB isn't a lot of data at all, given the issue is the EBI servers and not your institutes provision - you would have noticed this before on other projects if it was an institute issue. It therefore suggests someone else is hogging the EBI capacity and there is no provision to bypass this.
  
  The best solution is contact them directly
  
  > ecovid19@ebi.ac.uk
  
  ... and state the download size and download speed.
  
  There could be a bot sucking up the database (that is what it sounds like), which would be easy to write given FTP is permitted. They will have the ability to block an IP and NCBI have done this before albeit a long time ago.
  
  Finally filling the user experience form would help EBI [which is here](https://docs.google.com/forms/d/e/1FAIpQLSeF8YcPi6xxCNQQ6KJFVEHBMBAcX-mDNiWFXXoN1a5wWc-0Zw/viewform). However the issue you described suggests escalation. NCBI are very responsive and I would be surprised if EBI did not respond quickly.
  
  Good luck with the project, sounds interesting.
  
  Generic response below if its just traditional over-demand for the service.
  
  ---
  
  I suspect that the beautiful database EBI has provided (and it is amazing) has not anticipated the level of demand for the database size. They are far from being alone.
  
  Modern web architectures, such as Spotify, dedicate enormous effort to matching demand with service provision, notably using Kubernetes. When the server is overloaded they will simply duplicate their core database onto additional machines, which they will be buying in from AWS/GCP. It can even be done dynamically without intervention. NCBI are certainly heading in that direction.
  
  If the EBI wrote this under Kubernetes (it could be a monolith however), the limiting factor is the budget in computer resource allocation (again you can increase Kubernetes provision dynamically). However, if its a monolith this will not be easy and again ultimately someone has pay, i.e. EBI.
  
  Regardless, the fact you can use FTP means EBI are hosting their own service, GCP doesn't permit this and AWS is almost certainly similar. This would mean they have to physically provided additional in-house machines. I visited EBI some time ago and at that time the then director went to some lengths describing how focused EBI were on matching their in-house storage capacity to exponential rise in sequence data. Sudden popularity of a webserver is not trivial issue if it hasn't been anticipated.
  
  ---
  
  **Alternatives**
  Using the NCBI service COVID service is the alternative if there is an urgent need to get the project moving. You may also wish to use GISAID [here](https://www.gisaid.org).
  
  If you want to use the NCBI service, I provided the detailed instructions here: [How to programatically download SARS-CoV-2 fasta from NCBI/Genbank via API?](https://bioinformatics.stackexchange.com/questions/18642/how-to-programatically-download-sars-cov-2-fasta-from-ncbi-genbank-via-api/18643#18643)
  
  It is however important to understand the possible underlying causes from EBI. I do recall the then EBI director stating their biggest issue was a user simply switching to Genbank, which from their point of view isn't cool given the amount effort going into service provision.
  
  > **Comments:**
  > **username_0**: Thank you for the detailed explanation and alternatives. I have found an alternative to EBI from the answer you have suggested. It was quick to download from NCBI.  I will report/inform EBI about the data speed issues and update their response here.  
  > **username_1**: You are welcome. It is a pity for EBI because I think they try hard, however it is also a learning curve for them  
  > **username_0**: I have gotten a response from EBI saying, "They are aware of these downloading issues with few files and they have allocated my ticket to a colleague and help me resolve it asap"  
  > **username_1**: That's good, fingers crossed. They will be sensitive about you switching to Genbank.  
  
  ---
  
  