title: PipelineBuffer does not release memory
body: <p>I'm following this excellent post by <a href="http://www.software-architects.com/devblog/team/rainer-stropek" rel="nofollow">Rainer Stropek</a> on 1. creating a custom SSIS dataflow item &amp; 2. pulling data from Azure table storage.
  <a href="http://www.software-architects.com/devblog/2010/11/12/Custom-SSIS-Data-Source-For-Loading-Azure-Tables-Into-SQL-Server" rel="nofollow">http://www.software-architects.com/devblog/2010/11/12/Custom-SSIS-Data-Source-For-Loading-Azure-Tables-Into-SQL-Server</a></p>
  
  <p>I'll paste the code that would be of any interest to anyone who's attempting to answer my question:</p>
  
  <pre><code>GenericTableContext context = new GenericTableContext(cloudStorageAccount.TableEndpoint.AbsoluteUri, cloudStorageAccount.Credentials);
  
  public override void PrimeOutput(int outputs, int[] outputIDs, PipelineBuffer[] buffers) {
    IDTSOutput100 output = ComponentMetaData.OutputCollection[0];
    PipelineBuffer buffer = buffers[0];
    foreach(var item in this.context.CreateQuery&lt;GenericEntity&gt;("SomeTable") {
      buffer.AddRow();
      for (int x = 0; x &lt; columnInformation.Count; x++) {
        var ci = (ColumnInfo) columnInformation[x];
        var value = item[ci.ColumnName].Value;
        if (value != null) {
          buffer[ci.BufferColumnIndex] = value;
        } else {
          buffer.SetNull(ci.BufferColumnIndex);
        }
      }
    }
  </code></pre>
  
  <p>Now the <strong>Question:</strong></p>
  
  <p>My data set is very large; let's say a million records. I have an SSIS package in its simplest form, i.e. My custom data source writing data to a flat file destination. It has no transformations.</p>
  
  <p>As I run my package, I can see records getting written on to the CSV file in batches of approx 10,000 records (DefaultBufferMaxRows).</p>
  
  <p>My problem is, although these records are written, it still seems to remain in memory. Which ultimately causes my package to slow down and come to a halt. All the memory in my server is used up. </p>
  
  <p>Now I'm no SSIS expert, but i'm assuming that the records that I write in to the PipelineBuffer are not getting released after they are consumed. the word "Buffer" implies to me that it could be "flushed". But I could not find any API to do this.</p>
  
  <p>So I guess my question is, How can I release the memory allocated to records that have already been written on to the flat file so I have enough memory for the remaining records?</p>
  
  <p>Thank you,
  Nandun</p>