title: How to retrieve partial response with System.Net.HttpClient
body: <p>I'm trying to use the new HttpClient class (in .NET 4.5) to retrieve partial responses from the server in order to check the content. I need to limit the size of data retrieved to the first few bytes of content in the HTTP requests to limit the bandwidth usage.</p>
  
  <p>I've been unable to accomplish this. I have tried using GetAsync(url, HttpCompletionOption.ResponseHeadersRead) then use Content.ReadAsStream() in an attempt to only read the headers and then read the response stream in a small chunk. I also tried  GetStreamAsync() and then reading the Content stream with a small chunk (1000 bytes). </p>
  
  <p>In both cases it appears that HttpClient is pulling and buffering the entire HTTP response rather than just reading the requested byte count from the stream.</p>
  
  <p>Initially I was using Fiddler to monitor the data, but realized that Fiddler might actually be causing the entire content to be proxied. I switched to using System.Net tracing (which shows):</p>
  
  <pre><code>ConnectStream#6044116::ConnectStream(Buffered 16712 bytes.)
  </code></pre>
  
  <p>which is the full size rather than just the 1000 bytes read. I've also double checked in Wireshark to verify that indeed the the full content is being pulled over the wire and it is. With larger content (like a 110k link) I get about 20k of data before the TCP/IP stream is truncated.</p>
  
  <p>The two ways I've tried to read the data:</p>
  
  <pre><code>response = await client.GetAsync(site.Url, HttpCompletionOption.ResponseHeadersRead);
  var stream = await response.Content.ReadAsStreamAsync();
  
  var buffer = new byte[1000];                                        
  var count = await stream.ReadAsync(buffer, 0, buffer.Length);
  response.Close()  // close ASAP
  result.LastResponse = Encoding.UTF8.GetString(buffer);
  </code></pre>
  
  <p>and:</p>
  
  <pre><code>var stream = await client.GetStreamAsync(site.Url);
  var buffer = new byte[1000];
  var count = await stream.ReadAsync(buffer, 0, buffer.Length);
  result.LastResponse = Encoding.UTF8.GetString(buffer);
  </code></pre>
  
  <p>Both of them produce nearly identical .NET trace's which include the buffered read.</p>
  
  <p>Is it possible to have HttpClient actually read only a small chunk of an Http Repsonse, rather than the entire response in order to not use the full bandwidth? IOW is there a way to disable any buffering on the HTTP connection using either HttpClient or HttpWebRequest?</p>
  
  <p><strong>Update:</strong>
  After some more extensive testing it looks like both HttpClient and HttpWebRequest buffer the first few TCP/IP frames - presumably to ensure the HTTP header is captured. So if you return a small enough request, it tends to get loaded completely just because it's in that inital bufferred read. But when loading a larger content url, the content does get truncated. For HttpClient it's around 20k, for HttpWebRequest somewhere around 8k for me. </p>
  
  <p>Using TcpClient doesn't have any buffering issues. When using it I get content read at the size of the read plus a bit extra for the nearest buffer size overlap, but that does include the HTTP header. Using TcpClient is not really an option for me as we have to deal with SSL, Redirects, Auth, Chunked content etc. At that point I'd be looking at implementing a full custom HTTP client just to turn of buffering.</p>