While running a Pilot for CDSW & CDH, i'm struggling running queries on hive table.
The CDSW host has the right gateway roles installed, hive-site.xml is present with the spark config files.
What are you trying to achieve? Why are you trying to run in local mode? Can you use the expected yarn client mode instead?
"Unlike PySpark or Sparklyr, you can access a SparkContext assigned to the spark (SparkSession) and sc(SparkContext) objects on console startup, just as when using the Spark shell."
Can you access to your hive tables using the spark SparkSession object?
The goal is to run Data Analysis using Spark where part of the data is stored in hive, The idea is to use the whole cluster to distribute the workload.
I missed the fact that I was running in local mode (As said it's a pilot, i'm totally new to Cloudera stack). I assumed that the workbench was by default in yarn mode. I'll dig in the doc again.
while I do get an result for Describe Table, the description returned does not match the setting in Hive.
You can use the SparkSession object to query your Hive tables. You don't need to create a HiveContext.
I don't think that this change would fix your issue.
This doesn't seem like a problem with your CDSW installation. Could you execute the same commands from a Session terminal or directly on the CDSW host machine using the spark2-shell cli tool to verify this?
Could you double check that a Hive gateway is deployed to your CDSW host?
Can you show that the table exists from eg. beeline?
// Detected repl transcript. Paste more, or ctrl-D to finish.
I set it to Hive and CDSW is now working as expected.