title: "'DataFrame' object has no attribute 'apply'" when trying to apply lambda to create new column
body: <p>I aim at adding a new column in a Pandas DataFrame, but I am facing an weird error.</p>
  
  <p>The new column is expected to be a transformation from an existing column, that can be done doing a lookup in a dictionary/hashmap.</p>
  
  <pre><code># Loading data
  df = sqlContext.read.format(...).load(train_df_path)
  
  # Instanciating the map
  some_map = {
      'a': 0, 
      'b': 1,
      'c': 1,
  }
  
  # Creating a new column using the map
  df['new_column'] = df.apply(lambda row: some_map(row.some_column_name), axis=1)
  </code></pre>
  
  <p>Which leads to the following error:</p>
  
  <pre><code>AttributeErrorTraceback (most recent call last)
  &lt;ipython-input-12-aeee412b10bf&gt; in &lt;module&gt;()
       25 df= train_df
       26 
  ---&gt; 27 df['new_column'] = df.apply(lambda row: some_map(row.some_column_name), axis=1)
  
  /usr/lib/spark/python/pyspark/sql/dataframe.py in __getattr__(self, name)
      962         if name not in self.columns:
      963             raise AttributeError(
  --&gt; 964                 "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
      965         jc = self._jdf.apply(name)
      966         return Column(jc)
  
  AttributeError: 'DataFrame' object has no attribute 'apply'
  </code></pre>
  
  <p>Other potentially useful info:
   * I am using Spark and Python 2.</p>