title: Loading huge 4Gb XML file using vtd-xml
body: <p>I am evaluating vtd-xml as a possible solution for a large data migration project. The input data is in xml format and if vtd-xml is viable it would save a lot of dev time. I run the example Process Huge XML Documents (Bigger than 2GB) from vtd-xml website: <a href="http://vtd-xml.sourceforge.net/codeSample/cs12.html" rel="nofollow">http://vtd-xml.sourceforge.net/codeSample/cs12.html</a>.</p>
  
  <p>I successfully process 500Mb but get the dreaded java.lang.OutOfMemoryError: Java heap space error with a 4Gb file.</p>
  
  <blockquote>
    <blockquote>
      <ol>
      <li>JVM Arguments: -Xmn100M -Xms500M -Xmx2048M. </li>
      <li>JVM Arguments: -Xmn100M -Xms500M -Xmx4096M. </li>
      </ol>
    </blockquote>
  </blockquote>
  
  <p>And with Maven:</p>
  
  <blockquote>
    <blockquote>
      <ol>
      <li>set MAVEN_OPTS=-Xmn100M -Xms500M -Xmx2048M </li>
      <li>set MAVEN_OPTS=-Xmn100M -Xms500M -Xmx4096M </li>
      </ol>
    </blockquote>
  </blockquote>
  
  <p>NOTE: I have tested it with various combinations of the JVM arguments. </p>
  
  <p>I have studied the vtd-xml site and API docs and browsed numerous questions here and elsewhere. All the awnsers point to setting the JVM memory higher or adding more physical memory. The vtd-xml website refer to memory usage of 1.3x-1.5x the xml file size but if using 64bit one should be able to process much larger files than available memerory. Surely it would also not be feasible to add 64Gb memory to process a 35Gb xml file.</p>
  
  <blockquote>
    <blockquote>
      <p>Environment:</p>
      
      <p>Windows 7 64 bit. 6Gb RAM. (Closed all other apps, 85% memory avaibale)</p>
      
      <p>java version "1.7.0_09"</p>
      
      <p>Java(TM) SE Runtime Environment (build 1.7.0_09-b05)</p>
      
      <p>Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)</p>
      
      <p>Eclipse Indigo</p>
      
      <p>Maven 2</p>
    </blockquote>
  </blockquote>
  
  <p>Running the example from both Eclipse and Maven throws the Out of memory exception.</p>
  
  <p>Example code:</p>
  
  <pre><code> import com.ximpleware.extended.VTDGenHuge;
   import com.ximpleware.extended.VTDNavHuge;
   import com.ximpleware.extended.XMLMemMappedBuffer;
  
   public class App {
  
  /* first read is the longer version of loading the XML file */
  public static void first_read() throws Exception{
  XMLMemMappedBuffer xb = new XMLMemMappedBuffer();
      VTDGenHuge vg = new VTDGenHuge();
      xb.readFile("C:\\Temp\\partial_dbdump.xml");
      vg.setDoc(xb);
      vg.parse(true);
      VTDNavHuge vn = vg.getNav();
      System.out.println("text data ===&gt;" + vn.toString(vn.getText()));
  }   
  
  /* second read is the shorter version of loading the XML file */
  public static void second_read() throws Exception{
      VTDGenHuge vg = new VTDGenHuge();
      if (vg.parseFile("C:\\Temp\\partial_dbdump.xml",true,VTDGenHuge.MEM_MAPPED)){
          VTDNavHuge vn = vg.getNav();
          System.out.println("text data ===&gt;" + vn.toString(vn.getText()));
      }
  }
  
  public static void main(String[] s) throws Exception{
      first_read();
      //second_read();
  }
  
  }
  </code></pre>
  
  <p>Error:  </p>
  
  <pre><code> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
  at com.ximpleware.extended.FastLongBuffer.append(FastLongBuffer.java:209)
  at com.ximpleware.extended.VTDGenHuge.writeVTD(VTDGenHuge.java:3389)
  at com.ximpleware.extended.VTDGenHuge.parse(VTDGenHuge.java:1653)
  at com.epiuse.dbload.App.first_read(App.java:14)
  at com.epiuse.dbload.App.main(App.java:29)
  </code></pre>
  
  <p>Any help would be appreciated. </p>