Question: I would like to scrape all the comments in this reddit page and then write the data into a csv file. I have written this code.

I noticed that all comments were 'div' elements with a class of 'RichTextJSON-root'.

Code written:

```
from bs4 import BeautifulSoup
import requests
import csv

# Reddit

source = requests.get(
    'https://www.reddit.com/r/sysadmin/comments/gjkqvj/who_has_made_the_switch_from_dell_to_lenovo/').text

soup = BeautifulSoup(source, 'html.parser')

countreddit = 0

csv_file = open('reddit_scrape.csv', 'w')
writer = csv.writer(csv_file)
writer.writerow(['Comment'])

comments = soup.find_all('div', class_='RichTextJSON-root')

for comment in comments:
    for para in comment.find_all('p'):
        paratext = para.text
        writer.writerow([paratext])
        countreddit += 1

print(f'Reddit Count: {countreddit}')
csv_file.close()

```

Link to reddit post: <https://www.reddit.com/r/sysadmin/comments/gjkqvj/who_has_made_the_switch_from_dell_to_lenovo/>

However, I have only managed to scrape till midway of the webpage, stopping at the comment 'Those using the WD15 docks would like to have a word with you.' May I ask how can I scrape till the end of this page?

I read on other posts that it could be because when beautifulsoup scrapes, it only scrapes what is rendered on the webpage and at this point in time, the webpage has not been rendered completely.

Thank you!

Answer: 
Very close. The reshaping operation should be a [`pivot_table`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html) with the aggfunc set to count rather than [`pivot`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot.html):

```
plot_df = (
    df_winter_start.pivot_table(index='month', columns='Disaster_Type',
                                values='day', aggfunc='count')
)

```

Now that the data is in the correct format:

`plot_df`

```
Disaster_Type   Ice  Snow  Winter
month                            
2              21.0   NaN     NaN
3               7.0   NaN     6.0
4               5.0   9.0     NaN
6               NaN   NaN     2.0

```

It can be plotted with stacked bars:

```
plot_df.plot.bar(stacked=True, ax=ax, zorder=3, cmap=cmap, rot=0)

```

[![plot 1 stacked](https://i.stack.imgur.com/J4Z7q.png)](https://i.stack.imgur.com/J4Z7q.png)

or as separate bars:

```
plot_df.plot.bar(ax=ax, zorder=3, cmap=cmap, rot=0)

```

[![plot 2 separate bars](https://i.stack.imgur.com/1DEH0.png)](https://i.stack.imgur.com/1DEH0.png)
Or you could group `df_winter_start` by type and month, counting up the events, and then use seaborn for plotting:

```py
df_counts = df_winter_start[['Disaster_Type', 
                             'month']].value_counts().reset_index(name='count')
df_counts

```

```
    Disaster_Type   month   count
0   Ice             2       21
1   Snow            4       9
2   Ice             3       7
3   Winter          3       6
4   Ice             4       5
5   Winter          6       2

```

```py
import seaborn as sns

cmap = {'Ice': '#1f77b4', 'Snow': '#aec7e8', 'Winter': '#ff7f0e'}

sns.barplot(data=df_counts, x='month', y='count', 
            hue='Disaster_Type', palette=cmap);

```

[![winter plot](https://i.stack.imgur.com/sDj4j.png)](https://i.stack.imgur.com/sDj4j.png)