title: Read from a huge MemoryStream in C#
body: <p>I use a BinaryReader (<code>MemoryStream(MyByteArray)</code>) to read variable sized records and process them all in memory. This works well as long as my bytestream, which is in the array, is less than about 1.7 GB in size. After that (which is the maximum size of an integer in my 64-bit system) you cannot create a larger bytearray, although I have enough real memory. So my solution has been to read the bytestream and split it into several byte arrays.</p>
  
  <p>Now however, I cannot "read" across the byte array boundaries, and, as my data is in a variable format, I cannot ensure that byte arrays always finish on a whole record.</p>
  
  <p>This must be a common problem for people processing very large datasets and still have the need for speed.</p>
  
  <p>How do I handle this problem?</p>