Question: I want to extract all members' names, but I am getting the names of the members only from the first page.

```
from bs4 import BeautifulSoup
import requests

r = requests.get('https://www.acoinsite.org/life-member')
soup = BeautifulSoup(r.text,'lxml')

for data in soup.find_all('p',class_='font_7'):
    print data.text

```

I expected the output of members name from all 103 pages. Any idea how to fix it?

Answer: 
When needing to pull data from multiple pages, you can usually do 1 of the following:

**1)** Figure out the url pattern that is used when going from page to page, and iterate through those. Unfortunately that doesn't seem to be an option as the url does not change from page to page. 

**2)** If the page is dynamic and needs to be rendered first before pulling the data from the source html, or to simply simulate the use of openig a browser and clicking on elements (such as this case clicking on `►`, use Selenium to simulate opening the page. Then pull the data from page 1, then simulate clicking on next page, pull the data from page 2 and append the data from page 2 to what you have stored from page 1, etc., etc., etc. until you've reached the end of the pages. Downside to this is it's time consuming to go page, to page, to page, especially for 103 pages.

**3)** Find the GET ot POST request url to access the API and get the data in a json format as suggested by QHarr. This is probably the best way to go about it, but I couldn't find that request url he/she mentioned, and I was able to find another way.

**4)** The json data is sometimes within the html source code. Find that by searching through the `<script>` tags, then manipulate the string to pull out the json structure and use `json.loads()` to read that in. The hardest part then was to find where they actually have all the data you want, but I found it under the keys: `['wixapps']['appbuilder']['items']['BlankList_i34svvgq322']`. Then you can iterate through those items to get what you want:

My solution below is option 4:

```
from bs4 import BeautifulSoup
import requests
import json

headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "DNT": "1", "Connection": "close", "Upgrade-Insecure-Requests": "1"}
r = requests.get('https://www.acoinsite.org/life-member',headers=headers)
soup = BeautifulSoup(r.content,'html.parser')

scripts = soup.find_all('script')

for script in scripts:
    if 'var warmupData = ' in script.text:
        script_str = script.text
        script_str = script_str.split('var warmupData = ')[1]
        jsonStr = script_str.rsplit(';',1)[0]

        jsonObj = json.loads(jsonStr)

for k, v in jsonObj['wixapps']['appbuilder']['items']['BlankList_i34svvgq322'].items():
    print (v['title'])

```

**Output**

You can print off the names yourself, but quickly, I also just put it into a list and to show you then have 1030 names:

```
name_list = [ v['title'] for k, v in jsonObj['wixapps']['appbuilder']['items']['BlankList_i34svvgq322'].items() ]

print (len(name_list))
1030

```
This should solve your issue:

```
from bs4 import BeautifulSoup
import requests
headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:65.0) Gecko/20100101 Firefox/65.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "DNT": "1", "Connection": "close", "Upgrade-Insecure-Requests": "1"}
r = requests.get('https://www.acoinsite.org/life-member',headers=headers)
soup = BeautifulSoup(r.content,'lxml')
print(soup)
for data in soup.findAll('p',class_='font_7'):
    print(data.find('em',{"style":"font-style:normal;"}).text)

```

hope this helps