Question: Can someone explain to me what happens if we have much bigger data than the driver size? In this case how does Spark exactly work? If it caches the data in disk memory then how will it be "In-memory computing"?
Any help will be appreciated.

Answer: 
We can use [`where`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.where.html) + [`rank`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.rank.html) on `axis=1`. [`rank`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.rank.html) with `method='min'` and `ascending=False` will establish an ordering within the row such that the smallest value is 1 and the largest is 5 (the total length of the row). We then use [`where`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.where.html) to replace all values with rank less than 3:

```
df = df.where(df.rank(axis=1, method='min', ascending=False) < 3, 0)

```

We can also use the opposite condition with [`mask`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.mask.html) to keep values that rank higher than 3 and replace those which are 3 or lower with 0:

```
df = df.mask(df.rank(axis=1, method='min', ascending=False) >= 3, 0)

```

Either option produces `df`:

```
     0    1     2     3     4
0  0.0  0.0  0.24  0.52  0.00
1  0.0  0.0  0.00  0.52  0.52

```

\*Note depending on desired behaviour we may also want `method='dense'` or `method='first'` which will change how duplicated values are handled in the ranking.

---

Setup:

```
import pandas as pd

df = pd.DataFrame({
    0: [0.21, 0.31],
    1: [0.11, 0.01],
    2: [0.24, 0.44],
    3: [0.52, 0.52],
    4: [0.12, 0.52]
})

```
You can try:

A - Use list(df["col"].unique()) and sort/sorted to get the first three values. Put it into a list.

B - Use df.loc to remove the rows with a value within this new list
(something like df.loc[df["col"].isin(a)] )