title: What's best way to update a very large file with c#
body: <p>I'm not asking about only reading a large file or reading/writing a xml file which I know there are Xml related classes for handling that. Let me give a more specific description of what I'm trying to do:</p>
  
  <p>I have a very large file size that is about 10TB, which I can not load into memory at once. Meaning, I could not do as below:</p>
  
  <pre><code>        var lines = File.ReadAllLines("LargeFile.txt");
          var t = 1 &lt;&lt; 40;
          for(var i= t; i&lt; 2 * t; i++)
          {
              lines[i] = someWork(); //
          }
  
          File.WriteAllLines("LargeFile.txt", lines);
  </code></pre>
  
  <p>I want to read and update lines in a range between 1 and 2TB.</p>
  
  <p>What's the best approach doing this? Examples of .Net classes or 3rd party libraries would be helpful.
  I'm also interested in how other languages handle this problem as well.</p>
  
  <hr>
  
  <p>I tried David's suggestion by using position.
  However, i feel it doesn't work. 
  1. the size of FileStream seems fixed, I can modify the bytes, but it will overwrite byte by byte. it my newdata size is large/less than original line of data. I won't be able to update correctly.
  2. I didn't find a O(1) way to convert line num to position num. it still take me O(n) to find the position.</p>
  
  <p>below is my try</p>
  
  <pre><code>    public static void ReadWrite()
      {
          var fn = "LargeFile.txt";
          File.WriteAllLines(fn, Enumerable.Range(1, 20).Select(x =&gt; x.ToString()));
  
          var targetLine = 11; // zero based
          long pos = -1;
          using (var fs = new FileStream(fn, FileMode.Open, FileAccess.Read, FileShare.Read))
          {
              while (fs.Position != fs.Length)
              {
                  if (targetLine == 0)
                  {
                      pos = fs.Position +1; // move pos to begin of next line;
                  }
  
                  // still take average O(N) time to scan whole file to find the position.
                  // I'm not sure if there is better way. to redirect to the pos of x line by O(1) time.
                  if (fs.ReadByte() == '\n')
                  {
                      targetLine--;
                  }
              }
          }
  
          using (var fs = new FileStream(fn, FileMode.Open, FileAccess.ReadWrite))
          {
              var data = Encoding.UTF8.GetBytes("999"); 
              fs.Position = pos;
              // if the modify data has differnt size compare to the current one
              // it will overwrite next lines of data
              fs.Write(data, 0, data.Length);
          }
      }
  </code></pre>