title: BeautifulSoup scraping information from multiple divs using loops into JSON
body: <p>I am scraping titles, descriptions, links, and people's names from a multiple divs that follow the same structure. I am using BeautifulSoup, and I am able to scrape everything out of the first div. However, I'm having trouble scraping from  my long list of divs, and getting the data in a portable format like CSV or JSON.</p>
  
  <p>How can I scrape each item from my long list of divs, and store that information in JSON objects together for each mp3? </p>
  
  <p>The divs look like this: </p>
  
  <pre><code>&lt;div class="audioBoxWrap clearBoth"&gt;
      &lt;h3&gt;Title 1&lt;/h3&gt;
      &lt;p&gt;Description 1&lt;/p&gt;
      &lt;div class="info" style="line-height: 1px; height: 1px; font-size: 1px;"&gt;&lt;/div&gt;
      &lt;div class="audioBox" style="display: none;"&gt;
          stuff
      &lt;/div&gt;
      &lt;div&gt; [ &lt;a href="link1.mp3"&gt;Right-click to download&lt;/a&gt;] &lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="audioBoxWrap clearBoth"&gt;
      &lt;h3&gt;Title 2&lt;/h3&gt;
      &lt;p&gt;Description 2&lt;/p&gt;
      &lt;div class="info" style="line-height: 1px; height: 1px; font-size: 1px;"&gt;&lt;/div&gt;
      &lt;div class="audioBox" style="display: none;"&gt;
          stuff
      &lt;/div&gt;
      &lt;div&gt; [ &lt;a href="link2.mp3"&gt;Right-click to download&lt;/a&gt;] &lt;/div&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>I've figured out how to scrape from the first div, but I cannot grab the info for each div. For example, my code below only spits out the h3 for the first div over and over. </p>
  
  <p>I know that I can create a python list for titles, descriptions, etc, but how do I keep the metadata structure like JSON, so that title1, link1, and description1 stay together, as well as title2's information. </p>
  
  <pre><code>with open ('soup.html', 'r') as myfile:
      html_doc = myfile.read()
  
      soup = BeautifulSoup(html_doc, 'html.parser')
  
      audio_div = soup.find_all('div', {'class':"audioBoxWrap clearBoth"})
  
      print len(audio_div)
      #create dictionary for storing scraped data. I don't know how to store the values for each mp3 separately.
  
      for i in audio_div:
          print soup.find('h3').text
  </code></pre>
  
  <p>I want my JSON to look something like this: </p>
  
  <pre><code>{  
     "podcasts":[  
        {  
           "title":"title1",
           "description":"description1",
           "link":"link1"
        },
        {  
           "title":"title2",
           "description":"description2",
           "link":"link2"
        }
     ]
  }
  </code></pre>