title: Oozie Hive workflow intermitent
body: <p>I currently have a problem where intermittently my oozie workflow will be unable to connect to my hive metastore. It seems like it is running out of connections to the hive-metastore?</p>
  
  <blockquote>
    <p>Caused by: MetaException(message:Could not connect to meta store using
    any of the URIs provided. Most recent failure:
    org.apache.thrift.transport.TTransportException:
    java.net.ConnectException: Connection refused
                                  at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
                                  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:277)
                                  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:163)
                                  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                                  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
                                  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
                                  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
                                  at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082)
                                  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:51)
                                  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
                                  at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140)
                                  at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151)
                                  at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:1013)
                                  at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:1000)
                                  at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:8732)
                                  at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8097)
                                  at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
                                  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:443)
                                  at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
                                  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:908)
                                  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
                                  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
                                  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
                                  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347)
                                  at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:445)
                                  at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:455)
                                  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:711)
                                  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
                                  at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:261)
                                  at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:238)
                                  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
                                  at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:49)
                                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                  at java.lang.reflect.Method.invoke(Method.java:597)
                                  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
                                  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
                                  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
                                  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
                                  at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
                                  at java.security.AccessController.doPrivileged(Native Method)
                                  at javax.security.auth.Subject.doAs(Subject.java:396)
                                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
                                  at org.apache.hadoop.mapred.Child.main(Child.java:262)
                                  Caused by: java.net.ConnectException: Connection refused
                                  at java.net.PlainSocketImpl.socketConnect(Native Method)
                                  at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
                                  at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
                                  at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
                                  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                                  at java.net.Socket.connect(Socket.java:529)
                                  at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
                                  ... 44 more
                                  )
                                  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:323)
                                  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:163)
                                  ... 42 more
                                  FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate
    org.apache.hadoop.hive.metastore.HiveMetaStoreClient</p>
  </blockquote>
  
  <p>I can run hive from the command line. Use hue to communicate to the hive-metastore and execute queries. This only seems to happen half way through my oozie workflows. Each and every hive action i commit has the hive-site.xml config as oozie.hive.defaults.</p>
  
  <p>hive-site.xml</p>
  
  <pre><code> &lt;property&gt;   &lt;name&gt;hive.metastore.uris&lt;/name&gt;  
   &lt;value&gt;thrift://localhost:9083&lt;/value&gt;   &lt;description&gt;Thrift uri for
   the remote metastore. Used by metastore client to connect to remote
   metastore.&lt;/description&gt; &lt;/property&gt;
  
   &lt;property&gt;   &lt;name&gt;datanucleus.fixedDatastore&lt;/name&gt;
       &lt;value&gt;false&lt;/value&gt; &lt;/property&gt;
  
   &lt;property&gt;   &lt;name&gt;javax.jdo.option.ConnectionURL&lt;/name&gt;  
   &lt;value&gt;jdbc:mysql://localhost:3306/metastore&lt;/value&gt;  
   &lt;description&gt;JDBC connect string for a JDBC metastore&lt;/description&gt;
   &lt;/property&gt;
  
   &lt;property&gt;   &lt;name&gt;javax.jdo.option.ConnectionDriverName&lt;/name&gt;  
   &lt;value&gt;com.mysql.jdbc.Driver&lt;/value&gt;   &lt;description&gt;Driver class name
   for a JDBC metastore&lt;/description&gt; &lt;/property&gt;
  
   &lt;property&gt;   &lt;name&gt;javax.jdo.option.ConnectionUserName&lt;/name&gt;  
   &lt;value&gt;hive&lt;/value&gt;   &lt;description&gt;username to use against metastore
   database&lt;/description&gt; &lt;/property&gt;
  
   &lt;property&gt;   &lt;name&gt;javax.jdo.option.ConnectionPassword&lt;/name&gt;  
   &lt;value&gt;hive&lt;/value&gt;   &lt;description&gt;password to use against metastore
   database&lt;/description&gt; &lt;/property&gt;
  </code></pre>
  
  <p>hive-metastore, hive-server2, mysql-server and oozie all run on the same host for the moment so localhost works. Any ideas? I have oozie share lib default true enabled and the sharelibs has been created.</p>
  
  <p>CDH 4.2.1 packages</p>
  
  <p>-oozie: oozie-3.3.0</p>
  
  <p>-hive-metastore: hive-metastore-0.10.0</p>
  
  <p>-hive-server2: hive-server2-0.10.0</p>
  
  <p>-mysql-server: mysql-server-5.1.69-1</p>
  
  <p>Any help would be greatly appreciated</p>