title: identify the most recent record different from the current value in a PySpark dataframe
body: <p>I have a PySpark dataframe, where each user has a certain status at a point in time, as in the dummy data below</p>
  <pre><code>    --------------------------
      |user_id| status| month  |
      --------------------------
      | 1     | A     | 12/2020|
      | 1     | A     | 11/2020|
      | 1     | B     | 10/2020|
      | 1     | B     | 09/2020|
      | 1     | A     | 08/2020|
      | 1     | C     | 07/2020|
      | 2     | A     | 12/2020|
      | 2     | A     | 11/2020|
      | 2     | A     | 10/2020|
      | 2     | B     | 09/2020|
  </code></pre>
  <p>I would like to create in my PySpark dataframe another two columns (previous_status_value and previous_status_month), which for each record indicate what was the most recent date when the user had a different status than the one in the record, and what was that value. With the above dummy data, the result would be</p>
  <pre><code>    ------------------------------------------------------------------------
      |user_id| status| month  | previous_status_value| previous_status_month|
      ------------------------------------------------------------------------
      | 1     | A     | 12/2020| B                    | 10/2020              |
      | 1     | A     | 11/2020| B                    | 10/2020              |
      | 1     | B     | 10/2020| A                    | 08/2020              |
      | 1     | B     | 09/2020| A                    | 08/2020              |
      | 1     | A     | 08/2020| C                    | 07/2020              |
      | 1     | C     | 07/2020| Null                 | Null                 |
      | 2     | A     | 12/2020| B                    | 09/2020              |
      | 2     | A     | 11/2020| B                    | 09/2020              |
      | 2     | A     | 10/2020| B                    | 09/2020              |
      | 2     | B     | 09/2020| Null                 | Null                 |
  </code></pre>
  <p>The dataframe has millions of records, so I was trying to approach this using Window functions (similar to this <a href="https://stackoverflow.com/questions/59198003/sql-or-pyspark-get-the-last-time-a-column-had-a-different-value-for-each-id">answer</a>) but didn't get there.</p>