title: Subtracting multiple values inside a column from each other by conditional pyspark
body: <p>I am struggling with something that I think should be very trivial.</p>
  <p>I have an input dataframe as:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>id</th>
  <th>months</th>
  <th>feature_a</th>
  <th>feature_b</th>
  <th>feature_c</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>1</td>
  <td>2</td>
  <td>1</td>
  <td>3</td>
  </tr>
  <tr>
  <td>1</td>
  <td>6</td>
  <td>3</td>
  <td>2</td>
  <td>4</td>
  </tr>
  <tr>
  <td>2</td>
  <td>1</td>
  <td>2</td>
  <td>1</td>
  <td>3</td>
  </tr>
  <tr>
  <td>2</td>
  <td>6</td>
  <td>3</td>
  <td>2</td>
  <td>4</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Now I want to group by id and subtract for each feature the value where the month is 6 by where the month is 1. Resulting in an output dataframe as:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>id</th>
  <th>feature_a</th>
  <th>feature_b</th>
  <th>feature_c</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>1</td>
  <td>1</td>
  <td>1</td>
  </tr>
  <tr>
  <td>2</td>
  <td>1</td>
  <td>1</td>
  <td>1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Now I am managing to do this with the following code:</p>
  <pre><code>def get_month_diff(df, start_month=1, end_month=6):
  
      columns_to_agg = ['feature_a', 'feature_b', 'feature_c']
      result = (df
                 .groupby('id')
                 .pivot('months')
                 .agg(*[F.sum(col).alias(f'{col}') for col in columns_to_agg])
                 )
      #  Pyspark doesnt work nice with columns that have '.'s in them
      result = result.toDF(*(c.replace('.', '_') for c in result.columns))
  
      for col in columns_to_agg:
          result = result.withColumn(col, result[f&quot;{end_month}_0_{col}&quot;] - result[f&quot;{start_month}_0_{col}&quot;])
  
      return result
  </code></pre>
  <p>I don't like that I have to do the substracting of one column from another column and creating these new columns outside the first spark transformation. So I am looking for a solution.</p>
  <p>Therefore I am asking if someone could help me in the right direction for solving this?</p>