title: How can I avoid file fragmentation while writing hundreds of multi-megabyte files?
body: <p>My question is similar to <a href="https://stackoverflow.com/questions/4107950/how-to-avoid-hard-disc-fragmentation">How to avoid hard disc fragmentation?</a>, but I will be generating several hundred files per day that can range in size from 2 MB to 100+ MB (that questioner implied his files were smaller as he was more worried about cluttering his disk, my problem is performance reading these files).  These files are written a little bit at a time (logging data), which is the best way to create fragmentation.  (A database is not an option.)  I have code to defragment after they are completely written, but performance suffers for files being read back the same day.</p>
  
  <p>It appears that the way to do it is suggested by <a href="https://stackoverflow.com/questions/917309/how-can-i-limit-file-fragmentation-while-working-with-net">How can I limit file fragmentation while working with .NET?</a>; tho they are short on details (and I'm in C++).  I'd use SetFilePointerEx() and SetEndOfFile() to size the file to 2MB to start with, and then when the file reaches allocated size, I'll resize based on observed growth rates.  Then when writing is complete, resize to the actual data size.</p>
  
  <p>One pitfall I see (actually <a href="http://www.cplusplus.com/forum/windows/22114/" rel="nofollow noreferrer">http://www.cplusplus.com/forum/windows/22114/</a> pointed it out) is what happens if <strike>my app crashes or</strike> the computer shuts down.  Now I've got undetermined data in my file, and no way to detect it through Windows.  This suggests I create a file to track how much data has been written, either per file or a single file.  Is there a better strategy?  Perhaps writing enough zeros after each write to be able to detect later (and then backing up to be ready for the next write)?</p>
  
  <p>Do you see any other gotchas I missed?</p>