Answers.Body: <p>The cURL command to obtain the download link returns a JSON containing the URL. This can be parsed on the command line using <a href="https://stedolan.github.io/jq/" rel="nofollow noreferrer">jq</a> and used to construct a command to download the file. For example, with a list of URLs in a file (one per line) called <em>urls.txt</em>, you could try the following:</p>
  <pre><code>while IFS= read -r url ; do
      &gt;&amp;2 echo &quot;Downloading: ${url}&quot;
  
      curl -sSH @cosmic_header &quot;<span class="math-container">${url}" |
      jq --arg base "$</span>(basename &quot;<span class="math-container">${url}")" -r '"curl -o \($</span>base) \(.url | @sh)&quot;' |
      sh
  done &lt; urls.txt
  </code></pre>
  <p>Note that the above stores the HTTP header in a file called <em>cosmic_header</em>, which can be created using:</p>
  <pre><code>echo &quot;Authorization: Basic $(echo 'username:password' | base64)&quot; &gt; cosmic_header
  </code></pre>
  <p>This avoids exposing the authorization key on the command line when you go to download your list of files, which might be preferable if working in a shared environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-29T01:43:06.747Z
Answers.LastActivityDate: 2023-03-29T01:43:06.747Z
Answers.UserDisplayName: Steve
Body: <p>Hi I am downloading the data from a website which requires authentication of email_id and password before the download. They then provide the download link which is used to download dataset. This download link is different every time</p>
  <pre><code>### Download Cosmic Files
  
  ##First you need to login with your login details and the access key ID may vary with each login ID
  
  ##STEP-1
  echo &quot;pr@gmail.com:abc123@&quot; | base64
  #The above lines gives a authorization key cHIxNEBzYW5
  
  ##STEP-2
  # get the download link using the authorization key from step 1
  #curl -H &quot;Authorization: Basic cHIxNEBzYW5&quot; https://cancer.sanger.ac.uk/cosmic/file_download/GRCh38/cosmic/v97/CosmicSample.tsv.gz
  
  #The above line gives the download link where AWS access key changes everytime like in this case it was https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&amp;Expires=1679591113&amp;Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D
  
  ##STEP-3
  # Download file with the link from step 2
  #curl -o CosmicSample.tsv &quot;https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&amp;Expires=1679591113&amp;Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D&quot;
  
  </code></pre>
  <ol>
  <li>I want to automate the script first by passing used_id and password as an string</li>
  <li>I want to replace the AWS key in the step 3 with a string that is generated from step 2</li>
  </ol>
  <p>Thank you</p>
  
Tags: <database><bash><curl>
Title: Passing strings for password, email authentication and download link
ThreadText: # Passing strings for password, email authentication and download link
  
  (Asked by: username_0 on 2023-03-28)
  
  Hi I am downloading the data from a website which requires authentication of email\_id and password before the download. They then provide the download link which is used to download dataset. This download link is different every time
  
  ```
  ### Download Cosmic Files
  
  ##First you need to login with your login details and the access key ID may vary with each login ID
  
  ##STEP-1
  echo "pr@gmail.com:abc123@" | base64
  #The above lines gives a authorization key cHIxNEBzYW5
  
  ##STEP-2
  # get the download link using the authorization key from step 1
  #curl -H "Authorization: Basic cHIxNEBzYW5" https://cancer.sanger.ac.uk/cosmic/file_download/GRCh38/cosmic/v97/CosmicSample.tsv.gz
  
  #The above line gives the download link where AWS access key changes everytime like in this case it was https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D
  
  ##STEP-3
  # Download file with the link from step 2
  #curl -o CosmicSample.tsv "https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D"
  
  ```
  
  1. I want to automate the script first by passing used\_id and password as an string
  2. I want to replace the AWS key in the step 3 with a string that is generated from step 2
  
  Thank you
  
  **Tags:** database, bash, curl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-29. Score: 3 | Accepted answer
  
  The cURL command to obtain the download link returns a JSON containing the URL. This can be parsed on the command line using [jq](https://stedolan.github.io/jq/) and used to construct a command to download the file. For example, with a list of URLs in a file (one per line) called *urls.txt*, you could try the following:
  
  ```
  while IFS= read -r url ; do
      >&2 echo "Downloading: ${url}"
  
      curl -sSH @cosmic_header "${url}" |
      jq --arg base "$(basename "${url}")" -r '"curl -o \($base) \(.url | @sh)"' |
      sh
  done < urls.txt
  
  ```
  
  Note that the above stores the HTTP header in a file called *cosmic\_header*, which can be created using:
  
  ```
  echo "Authorization: Basic $(echo 'username:password' | base64)" > cosmic_header
  
  ```
  
  This avoids exposing the authorization key on the command line when you go to download your list of files, which might be preferable if working in a shared environment.
  
  
  ---
  
  