title: Large Object Heap and String Objects coming from a queue
body: <p>I have a windows console app that is supposed to run without restarts for days and months. The app retrieves "work" from an MSMQ and process it. There are 30 threads that process a work chunk simultaneously.</p>
  
  <p>Each work chunk coming from the MSMQ is approximately 200kb most of which is allocated in a single String object.</p>
  
  <p>I have noticed that after processing about 3-4 thousands of these work chunks the memory consumption of the application is ridiculously high consuming 1 - 1.5 gb of memory. </p>
  
  <p>I run the app through a profiler and noticed that most of this memory (maybe a gig or so) is unused in the large object heap but the structure is fragmented.</p>
  
  <p>I have found that 90% of these unused (garbage collected) bytes were previously allocated String. I started suspecting then that the strings coming in from the MSMQ were allocated, used and then deallocated and are therefore the cause of the fragmentation.</p>
  
  <p>I understand that things like GC.Collect(2 or GC.Max...) wont help since they gc the large object heap but don't compact it (which is the problem here). So I think that what I need is to cache these Strings and re-use them somehow but since Strings are immutable I would have to use StringBuilders.</p>
  
  <p>My question is: Is there anyway to not change the underlying structure (i.e. using the MSMQ as this is something I cant change) and still avoid initializing a new String everytime to avoid fragmenting the LOH?</p>
  
  <p>Thanks,
  Yannis</p>
  
  <p><strong>UPDATE: About how these "work" chunks are currently retrieved</strong></p>
  
  <p>Currently these are stored as WorkChunk objects in the MSMQ. Each of these objects contains a String called Contents and another String called Headers. These are actual textual data. I can change the storage structure to something else if needed and potentially the underlying storage mechanism if needed to something else than an MSMQ.</p>
  
  <p>On the worker nodes side currently we do </p>
  
  <p>WorkChunk chunk = _Queue.Receive();</p>
  
  <p>So there is little we can cache at this stage. If we changed the structure(s) somehow then I suppose we could do a bit of progress. In any case, we will have to sort out this problem so we will do whatever is needed to avoid throwing out months of work.</p>
  
  <p><strong>UPDATE:</strong> I went on to try some of the suggestions below and noticed that this issue cannot be reproduced on my local machine (running Windows 7 x64 and 64bit app). this makes things so much more difficult - if anyone knows why then it would really help repdocung this issue locally.</p>