I have a newly installed hdp 3.0.0 on a single cluster.
Everything starts without issue, except hiveserver2.
Logs are saying that hiveserver2 is timing out due to a failure of zookeeper not finding the /hiveserver2 node.
I can connect to the zookeeper client and when I list out the node it is empty ( [] ).
Here's an excerpt from the relevant part of the logs:
2018-08-30 11:51:35,220 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/ --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://mnhost3.corp.pitechsol.com:8020' | head -1'] {} 2018-08-30 11:51:56,704 - call returned (0, '') 2018-08-30 11:51:56,704 - Execute['/var/lib/ambari-agent/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.err /var/run/hive/hive-server.pid /usr/hdp/current/hive-server2/conf/ /etc/tez/conf'] {'environment': {'HIVE_BIN': 'hive', 'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112', 'HADOOP_HOME': u'/usr/hdp/current/hadoop-client'}, 'not_if': 'ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps -p >/dev/null 2>&1', 'user': 'hive', 'path': [u'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/3.0.0.0-1634/hadoop/bin']} 2018-08-30 11:51:56,799 - Execute['://mnhost3.corp.pitechsol.com/hive?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true' hive [PROTECTED] com.mysql.jdbc.Driver'] {'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 5, 'try_sleep': 10} 2018-08-30 11:51:57,499 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {} 2018-08-30 11:51:58,720 - call returned (1, '') 2018-08-30 11:51:58,721 - Will retry 29 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s) 2018-08-30 11:52:08,732 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {} 2018-08-30 11:52:09,601 - call returned (1, '') 2018-08-30 11:52:09,612 - Will retry 28 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s) 2018-08-30 11:52:19,623 - call['/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server mnhost3.corp.pitechsol.com:2181 ls /hiveserver2 | grep 'serverUri=''] {} 2018-08-30 11:52:20,815 - call returned (1, '')
I guess the real question I have is why is the /hiveserver2 node not getting initialized within zookeeper? Where/When/What/How is the the hiveserver2 node created/initialized?
I've made no changes from the default parameters on the install. This is on a RHEL 7.5 OS if that matters any.
Created 08-31-2018 02:58 AM
Hi @Michael north,
Not sure if its same,are you hitting :
have you configured ViewFS ? :...
I didn't do anything that would lead me to believe I configured viewFS into the system.
I am using the stock configuration as downloaded from Hortonworks for HDP 3.0.0
Michael
The znode is created when you fire up zk it read and uploads the config of most of the components. I would think you need to revise your hive setup.
Switch directory to /usr/hdp/current/zookeeper-client/
$ ./bin/zkCli.sh
Could you check your ZK namespace under the hiveserver2 path you should have an entry like below
[zk: localhost:2181(CONNECTED) o] ls /hiveserver2 [serverUri=nakuru.kenya.ke:10000;version=1.2.1000.2.6.2.0-205;sequence=0000000061] [zk: localhost:2181(CONNECTED) 1]
Can you check if HS2 can reach 2181 port of ZK host which in your case is the same machine? Have you configured a metastore for your hive either MySQL, Postgres or otherwise? if not you need to do that See my attached screenshot When the HS2 is running and only then when you restart will zk pick the configuration and the znode as shown above
HTH
I am getting the same output when performing command
ls /hiveserver2
Any possible solution acquired?
Created 05-20-2021 04:39 AM
-Try the below.
-When you start the HS2, track the logs and look for the first error in HS2.
-Parallelly ensure that your zookeeper is up and running and check the zk logs during the restart to validate if the connections are coming to ZK and whether zk is accepting or rejecting the connection.
Check your Ambari UI--->HDFS-->Config-- look for the parameter hadoop.proxyuser.hive.hosts make sure it's the HS2 host is listed here.
In my case bern.swiss.ch
Then restart the stale services and revert if you see any errors share the logs
Happy Hadooping
mine fix by ... Install Ranger
please check your stderr ....