title: Scraping through every product on retailer website
body: <p>We are trying to scrape every product for every category on Forever 21's website. Given a product page, we know how to extract the information we need, and given a category, we can extract every product. However, we do not know how to crawl through every product category. Here is our code for a given category and getting every product:</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  import json
  #import re
  
  params = {"action": "getcategory",
            "br": "f21",
            #"category": re.compile('\S+'),
            "category": "dress",
            "pageno": 1,
            "pagesize": "",
            "sort": "",
            "fsize": "",
            "fcolor": "",
            "fprice": "",
            "fattr": ""}
  
  url = "http://www.forever21.com/Ajax/Ajax_Category.aspx"
  js = requests.get(url, params=params).json()
  soup = BeautifulSoup(js[u'CategoryHTML'], "html.parser")
  i = 0
  j = 0
  
  while len(soup.select("div.item_pic a")) != 0:
     for a in soup.select("div.item_pic a"):
        #print a["href"]
        i = i + 1
  
     params["pageno"] = params["pageno"] + 1
     j = j + 1
     js = requests.get(url, params=params).json()
     soup = BeautifulSoup(js[u'CategoryHTML'], "html.parser")
  
  print i
  print j
  </code></pre>
  
  <p>As you can see in the comments, we tried to use regular expressions for the category but had no success. i and j are just product and page counters. Any suggestions on how to modify/add to this code to get every product category?</p>