title: R: fetching pdf documents from Companies House API
body: <p>I'm trying to fetch documents from the API using R. Appreciate the clarification of the process in <a href="http://forum.aws.chdev.org/t/fetch-a-document-api/978/4?u=kasia_kulma" rel="noreferrer">this post</a>. I've been following the above steps with partial success, but still fail the last step to get access to documents' content:</p>
  
  <blockquote>
    <ol>
    <li>Find the document filing you're interested in (e.g. make a filing history request1 for the company). Parse the response for the link to the document in the field "links" : { "document_metadata" : "link URI fragment here" }.</li>
    </ol>
  </blockquote>
  
  <p>No problem:</p>
  
  <pre><code>library(httr)
  library(jsonlite)
  library(openssl)
  
  ### retrieving filing history ####
  company_num = 'FC013908'
  key = 'my_key'
  fh_path = paste0('/company/', str_to_upper(company_num), "/filing-history")
  fh_url &lt;- modify_url("https://api.companieshouse.gov.uk/", path = fh_path)
  fh_test &lt;- GET(fh_url, authenticate(key, "")) #status_code = 200
  fh_parsed &lt;- jsonlite::fromJSON(content(fh_test, "text",encoding = "utf-8"), flatten = TRUE)
  docs &lt;- fh_parsed$items
  </code></pre>
  
  <p>Done.</p>
  
  <blockquote>
    <p>2 For a given document request the document metadata via CH Document API3. Parse the response to get the document (mime) types available and the link to the actual document data (document URI fragment).</p>
  </blockquote>
  
  <p>No problems here:</p>
  
  <pre><code>md_meta_url = docs$links.document_metadata[1]  
  key_pass &lt;- paste0(key,":")
  decoded_auth &lt;- paste0('Basic ', base64_encode(key_pass))
  
  md_test &lt;- GET(md_meta_url,
                 add_headers(Authorization = decoded_auth)
                 )
  md_test #status_code = 200!
  md_parsed &lt;- jsonlite::fromJSON(content(md_test, "text",encoding = "utf-8"), flatten = TRUE)
  </code></pre>
  
  <p>This way I can obtain the content URL:</p>
  
  <pre><code>cont_url = md_parsed$links$document
  </code></pre>
  
  <blockquote>
    <p>Request the actual document9, specifying the mime type (e.g. "application/pdf").</p>
  </blockquote>
  
  <p>I do it while NOT following the redirect and, as expected, I get the 302 status code with the  <code>location</code> header:</p>
  
  <pre><code>accept = 'application/pdf'
  cont_test &lt;- GET(cont_url, 
             add_headers(Authorization = decoded_auth,
                         Accept = accept),
             config(followlocation = FALSE)
  )
  
  final_url &lt;- cont_test$headers$location
  
  &gt; final_url
  [1] "https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/LjBouRHeXXpIYAvqYIPWL06iXaliPz6Pucp1OXCXQhI/application-pdf?AWSAccessKeyId=ASIAJX7TVURFXZTY5DNQ&amp;Expires=1529483765&amp;Signature=uUQx6RTW7XBLqx4L6pYr5tOUySg%3D&amp;x-amz-security-token=FQoDYXdzEP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGxe7meYGe3OYhNwcSK3AwcVYJUXaUMf19oVO9s4qNPWN8AHjNNd5rrZhgE9YTkF1OmzyZSL5xHbls664kDP%2Bxd7dz9PIU5O1D%2BVxoDyoYcFiS6acDnO28KpfFE56lUZNfedf1jys%2FP0SJ8f%2F50Cbn93bfOlm0MZA9%2BQ2DYQvPfkWSvrDjMyCXHbu57gpZHjQKPNRTgzGXzUUCvFwREytGMM4eThhn4Glvvx%2FA8IiLbnsvgmEKw9iAj7KWIenhoJq3cTRytUpVeipLnQoBVLau8dFYkKdAHZaYM2Tlx0z6ObRb%2BGdm7W7eOVA1bFXuUXmUmnAHruDIwwLlgOVN2IJ9CxmJU22lY8jrEm%2BUivtrdp2oofn32PryBEJ8jJOg9cIpLbBBx%2FeOkng9zJwnZbute7Nmh%2BnaY2btsId6JjraFNsTvR%2B1qEZX9uuznUdJdqgVfTMj2gGrAmntwk0JAkILlvamzjWC%2F9vAqK7Xvt8aC6hlIMB2vdzTCU9Jf%2FrIMTClTJkk0BzBuvJ86t1l%2BXb4rF5Pab%2FegFpJ6nvZKqde%2F77wMMiTyG35EndmYx4AWqTIh9EofYwKZa9uciNvRT0E2%2BYnT5jZMo%2BdWn2QU%3D"
  </code></pre>
  
  <p>However, when I try to </p>
  
  <blockquote>
    <p>Request this URI from Amazon again passing the content type you want again.
    I get 400 error: </p>
  </blockquote>
  
  <pre><code> final_test &lt;- GET(final_url, 
                   add_headers(Authorization = decoded_auth,
                               Accept = accept
                               ))
  
  &gt; final_test
  Response [https://s3-eu-west-1.amazonaws.com/document-api-images-prod/docs/LjBouRHeXXpIYAvqYIPWL06iXaliPz6Pucp1OXCXQhI/application-pdf?AWSAccessKeyId=ASIAJX7TVURFXZTY5DNQ&amp;Expires=1529483765&amp;Signature=uUQx6RTW7XBLqx4L6pYr5tOUySg%3D&amp;x-amz-security-token=FQoDYXdzEP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDGxe7meYGe3OYhNwcSK3AwcVYJUXaUMf19oVO9s4qNPWN8AHjNNd5rrZhgE9YTkF1OmzyZSL5xHbls664kDP%2Bxd7dz9PIU5O1D%2BVxoDyoYcFiS6acDnO28KpfFE56lUZNfedf1jys%2FP0SJ8f%2F50Cbn93bfOlm0MZA9%2BQ2DYQvPfkWSvrDjMyCXHbu57gpZHjQKPNRTgzGXzUUCvFwREytGMM4eThhn4Glvvx%2FA8IiLbnsvgmEKw9iAj7KWIenhoJq3cTRytUpVeipLnQoBVLau8dFYkKdAHZaYM2Tlx0z6ObRb%2BGdm7W7eOVA1bFXuUXmUmnAHruDIwwLlgOVN2IJ9CxmJU22lY8jrEm%2BUivtrdp2oofn32PryBEJ8jJOg9cIpLbBBx%2FeOkng9zJwnZbute7Nmh%2BnaY2btsId6JjraFNsTvR%2B1qEZX9uuznUdJdqgVfTMj2gGrAmntwk0JAkILlvamzjWC%2F9vAqK7Xvt8aC6hlIMB2vdzTCU9Jf%2FrIMTClTJkk0BzBuvJ86t1l%2BXb4rF5Pab%2FegFpJ6nvZKqde%2F77wMMiTyG35EndmYx4AWqTIh9EofYwKZa9uciNvRT0E2%2BYnT5jZMo%2BdWn2QU%3D]
    Date: 2018-06-20 08:37
    Status: 400
    Content-Type: application/xml
    Size: 523 B
  &lt;BINARY BODY&gt;
  </code></pre>
  
  <p>Needless to say, executing </p>
  
  <pre><code>browseURL(final_test$url)
  </code></pre>
  
  <p>returns <code>Access Denied</code> error. I suspect it may have something to do with Amazon authorization problems similar to those described <a href="http://forum.aws.chdev.org/t/document-images/26/21?u=kasia_kulma" rel="noreferrer">here</a>. Any ideas how to solve this final hurdle?</p>
  
  <p>Thanks!</p>