title: Python -> Py4j -> Spark -> Cassandra
body: <p>I would like to test a simply Spark row count job on a test Cassandra table with only four rows just to verify that everything works.</p>
  
  <p>I can quickly get this working from Java:</p>
  
  <pre><code>    JavaSparkContext sc = new JavaSparkContext(conf);
      SparkContextJavaFunctions sparkContextJavaFunctions = CassandraJavaUtil.javaFunctions(sc);
      CassandraJavaRDD&lt;CassandraRow&gt; table = sparkContextJavaFunctions.cassandraTable("demo", "playlists");
      long count = table.count();
  </code></pre>
  
  <p>Now, I'd like to get the same thing working in Python. The Spark distribution comes with a set of unbundled PySpark source code to use Spark from Python. It uses a library called py4j to launch a Java server and marshal java commands through a TCP gateway. I'm using that gateway directly to get this working.</p>
  
  <p>I specify the following extra jars to the Java SparkSubmit host via the --driver-class-path option:</p>
  
  <pre><code>spark-cassandra-connector-java_2.11-1.2.0-rc1.jar
  spark-cassandra-connector_2.11-1.2.0-rc1.jar
  cassandra-thrift-2.1.3.jar
  cassandra-clientutil-2.1.3.jar
  cassandra-driver-core-2.1.5.jar
  libthrift-0.9.2.jar
  joda-convert-1.2.jar
  joda-time-2.3.jar
  </code></pre>
  
  <p>Here is the core Python code to do the row count test:</p>
  
  <pre><code>from pyspark.java_gateway import launch_gateway
  jvm_gateway = launch_gateway()
  sc = jvm_gateway.jvm.org.apache.spark.api.java.JavaSparkContext(conf)
  spark_cass_functions = jvm_gateway.jvm.com.datastax.spark.connector.japi.CassandraJavaUtil.javaFunctions(sc)
  table = spark_cass_functions.cassandraTable("demo", "playlists");
  </code></pre>
  
  <p>On this last line, I get the following error:</p>
  
  <pre><code>py4j.protocol.Py4JJavaError: An error occurred while calling o5.cassandraTable.
  : com.datastax.spark.connector.util.ConfigCheck$ConnectorConfigurationException: Invalid Config Variables
  Only known spark.cassandra.* variables are allowed when using the Spark Cassandra Connector.
  spark.cassandra.connection.conf.factory is not a valid Spark Cassandra Connector variable.
  No likely matches found.
  spark.cassandra.auth.conf.factory is not a valid Spark Cassandra Connector variable.
  No likely matches found.
      at com.datastax.spark.connector.util.ConfigCheck$.checkConfig(ConfigCheck.scala:38)
      at com.datastax.spark.connector.rdd.CassandraRDD.&lt;init&gt;(CassandraRDD.scala:18)
      at com.datastax.spark.connector.rdd.CassandraTableScanRDD.&lt;init&gt;(CassandraTableScanRDD.scala:59)
      at com.datastax.spark.connector.rdd.CassandraTableScanRDD$.apply(CassandraTableScanRDD.scala:182)
      at com.datastax.spark.connector.japi.SparkContextJavaFunctions.cassandraTable(SparkContextJavaFunctions.java:88)
      at com.datastax.spark.connector.japi.SparkContextJavaFunctions.cassandraTable(SparkContextJavaFunctions.java:68)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
      at py4j.Gateway.invoke(Gateway.java:259)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:207)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>Clearly, there is some configuration or setup issue. I'm not sure how to reasonably debug or investigate or what I could try. Can anyone with more Cassandra/Python/Spark expertise provide some advice? Thank you!</p>
  
  <p>EDIT: A coworker setup a spark-defaults.conf file that was the root of this. I don't fully understand why this caused problems from Python and not from Java, but it doesn't matter. I don't want that conf file and removing it resolved by issue.</p>