title: How to download a HTTP resource to a file with Akka Streams and HTTP?
body: <p>Over the past few days I have been trying to figure out the best way to download a HTTP resource to a file using Akka Streams and HTTP.</p>
  
  <p>Initially I started with the <a href="http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.2/scala/http/client-side/request-level.html#Future-Based_Variant" rel="noreferrer">Future-Based Variant</a> and that looked something like this:</p>
  
  <pre><code>def downloadViaFutures(uri: Uri, file: File): Future[Long] = {
    val request = Get(uri)
    val responseFuture = Http().singleRequest(request)
    responseFuture.flatMap { response =&gt;
      val source = response.entity.dataBytes
      source.runWith(FileIO.toFile(file))
    }
  }
  </code></pre>
  
  <p>That was kind of okay but once I learnt more about pure Akka Streams I wanted to try and use the <a href="http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.2/scala/http/client-side/request-level.html#Flow-Based_Variant" rel="noreferrer">Flow-Based Variant</a> to create a stream starting from a <code>Source[HttpRequest]</code>. At first this completely stumped me until I stumbled upon the <code>flatMapConcat</code> flow transformation. This ended up a little more verbose:</p>
  
  <pre><code>def responseOrFail[T](in: (Try[HttpResponse], T)): (HttpResponse, T) = in match {
    case (responseTry, context) =&gt; (responseTry.get, context)
  }
  
  def responseToByteSource[T](in: (HttpResponse, T)): Source[ByteString, Any] = in match {
    case (response, _) =&gt; response.entity.dataBytes
  }
  
  def downloadViaFlow(uri: Uri, file: File): Future[Long] = {
    val request = Get(uri)
    val source = Source.single((request, ()))
    val requestResponseFlow = Http().superPool[Unit]()
    source.
      via(requestResponseFlow).
      map(responseOrFail).
      flatMapConcat(responseToByteSource).
      runWith(FileIO.toFile(file))
  }
  </code></pre>
  
  <p>Then I wanted to get a little tricky and use the <code>Content-Disposition</code> header.</p>
  
  <p>Going back to the Future-Based Variant:</p>
  
  <pre><code>def destinationFile(downloadDir: File, response: HttpResponse): File = {
    val fileName = response.header[ContentDisposition].get.value
    val file = new File(downloadDir, fileName)
    file.createNewFile()
    file
  }
  
  def downloadViaFutures2(uri: Uri, downloadDir: File): Future[Long] = {
    val request = Get(uri)
    val responseFuture = Http().singleRequest(request)
    responseFuture.flatMap { response =&gt;
      val file = destinationFile(downloadDir, response)
      val source = response.entity.dataBytes
      source.runWith(FileIO.toFile(file))
    }
  }
  </code></pre>
  
  <p>But now I have no idea how to do this with the Future-Based Variant. This is as far as I got:</p>
  
  <pre><code>def responseToByteSourceWithDest[T](in: (HttpResponse, T), downloadDir: File): Source[(ByteString, File), Any] = in match {
    case (response, _) =&gt;
      val source = responseToByteSource(in)
      val file = destinationFile(downloadDir, response)
      source.map((_, file))
  }
  
  def downloadViaFlow2(uri: Uri, downloadDir: File): Future[Long] = {
    val request = Get(uri)
    val source = Source.single((request, ()))
    val requestResponseFlow = Http().superPool[Unit]()
    val sourceWithDest: Source[(ByteString, File), Unit] = source.
      via(requestResponseFlow).
      map(responseOrFail).
      flatMapConcat(responseToByteSourceWithDest(_, downloadDir))
    sourceWithDest.runWith(???)
  }
  </code></pre>
  
  <p>So now I have a <code>Source</code> that will emit one or more <code>(ByteString, File)</code> elements for each <code>File</code> (I say each <code>File</code> since there is no reason the original <code>Source</code> has to be a single <code>HttpRequest</code>).</p>
  
  <p>Is there anyway to take these and route them to a dynamic <code>Sink</code>?</p>
  
  <p>I'm thinking something like <code>flatMapConcat</code>, such as:</p>
  
  <pre><code>def runWithMap[T, Mat2](f: T =&gt; Graph[SinkShape[Out], Mat2])(implicit materializer: Materializer): Mat2 = ???
  </code></pre>
  
  <p>So that I could complete <code>downloadViaFlow2</code> with:</p>
  
  <pre><code>def destToSink(destination: File): Sink[(ByteString, File), Future[Long]] = {
    val sink = FileIO.toFile(destination, true)
    Flow[(ByteString, File)].map(_._1).toMat(sink)(Keep.right)
  }
  sourceWithDest.runWithMap {
    case (_, file) =&gt; destToSink(file)
  }
  </code></pre>