title: Creating index in hive 0.9
body: <p>I am trying to create index on tables in Hive 0.9. One table has 1 billion rows, another has 30 Million rows. The command I used is (other than creating the table and so on) </p>
  
  <pre><code>  CREATE INDEX DEAL_IDX_1 ON TABLE DEAL (ID) AS 
    'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler'
    WITH DEFERRED REBUILD;
  
    alter index DEAL_IDX_1 ON DEAL rebuild;
  
    set hive.optimize.autoindex=true;
    set hive.optimize.index.filter=true;
  </code></pre>
  
  <p>For the 30 Mill. row table, the rebuilding process looks alright (mapper and reducer both finished) until in the end it prints </p>
  
  <pre><code>  Invalid alter operation: Unable to alter index.
    FAILED: Execution Error, return code 1 
    from org.apache.hadoop.hive.ql.exec.DDLTask
  </code></pre>
  
  <p>Checking the log, and it had the error </p>
  
  <pre><code>java.lang.ClassNotFoundException: org.apache.derby.jdbc.EmbeddedDriver"
  </code></pre>
  
  <p>Not sure why this error was encountered, but anyway, I added the derby-version.jar:</p>
  
  <pre><code>add jar /path/derby-version.jar
  </code></pre>
  
  <p>The reported error was resolved, but still got another error:</p>
  
  <pre><code>org.apache.hadoop.hive.ql.exec.FileSinkOperator: 
  StatsPublishing error: cannot connect to database
  </code></pre>
  
  <p>Not sure how to solve the problem. I do see the created index table under hive/warehouse though. </p>
  
  <p>For the 1 Billion row table, it is another story. The mapper just got stuck at 2% or so. And error showed</p>
  
  <pre><code>FATAL org.apache.hadoop.mapred.Child: Error running child : 
  java.lang.OutOfMemoryError: Java heap space 
  </code></pre>
  
  <p>I attempted to enforce max heap size, as well as max mapr memory (see the settings mentioned somewhere but not in hive's configuration settings):</p>
  
  <pre><code>set mapred.child.java.opts =  -Xmx6024m
  set mapred.job.map.memory.mb=6000;
  set mapred.job.reduce.memory.mb=4000;
  </code></pre>
  
  <p>However, this is not help. The mapper would still got stuck at 2% with the same error. </p>