title: Map h4 to div siblings in dataframe Beautifulsoup python
body: <p>I'm scraping a webpage but having trouble mapping the information into a dataframe. There are no tables in the HTML. Here is an example of the HTML:</p>
  
  <pre><code>html= [
  &lt;h2&gt;Event Title&lt;h2&gt;
  &lt;div class="row"&gt;
      &lt;h4&gt;Category 1&lt;h4&gt;
      &lt;div&gt;A&lt;div&gt;
      &lt;h4&gt;Category 2&lt;h4&gt;
      &lt;div&gt;B&lt;div&gt;
      &lt;h4&gt;Category 3&lt;h4&gt;
      &lt;div&gt;C&lt;div&gt;
      &lt;h4&gt;Category 4&lt;h4&gt;
      &lt;div&gt;D&lt;div&gt;
  ]
  </code></pre>
  
  <p>Here is my code using requests and Beautifulsoup in python:</p>
  
  <pre><code>data = []
  event = soup.find('h2')
  for i in soup.find_all('div', {'class': 'row'}):
      categories = [x.text for x in i.findAll('h4')]
      info = [x.text for x in i.findAll('div')]
  
      datum = {'event': event.get_text().replace('\n', '').replace('\r', ''), 
               'categories ': categories , 
               'info ': info }
  
      data.append(datum)
  
  df = pd.DataFrame(data)
  df
  </code></pre>
  
  <p>The dataframe ends up looking like with one event title and two lists:</p>
  
  <pre><code>index - event - categories - info
  1 - Event Title - ['Category 1','Category 2','Category 3','Category 4'] - ["Category 1 \n A\n Category 2\n B\n Category 3\n C\n Category 4\n D\n"]
  </code></pre>
  
  <p>I would like it to map somehow to end up so that h4 Category 1 is related to div A. </p>
  
  <pre><code>index - event - categories - info
  1 - Event Title - Category 1 - A
  2 - Event Title - Category 2 - B
  3 - Event Title - Category 3 - C
  4 - Event Title - Category 4 - D
  </code></pre>
  
  <p>Since h4 and div are siblings and not parent-child , it is possible to separate this in my web scrape code? I have multiple pages with different event titles and the data is too large to do it by hand. </p>
  
  <p>I have also tried, among others:</p>
  
  <pre><code>data = []
  
  event = soup.find('h2').get_text()
  
  for i in soup.find_all('div', {'class': 'row'}):
      categories = [x.text for x in soup.findAll('h4')]
      cats = soup.find_all('h4')
      cat = cats[3]
      info = cat.findNextSiblings('div')
  
      datum = {'event': event, 'categories ': categories , 'info': info} 
      data.append(datum)
  
      df1 = pd.DataFrame(data)
  df1
  </code></pre>
  
  <p>The result of this one gives me a df of:</p>
  
  <pre><code>index - event - categories - info
  1 - Event Title - ['Category 1','Category 2','Category 3','Category 4'] - [&lt;div&gt;A&lt;div&gt;, &lt;div&gt;B&lt;div&gt;, &lt;div&gt;C&lt;div&gt;, &lt;div&gt;D&lt;div&gt;]
  </code></pre>
  
  <p>Here is the weblink to inspect the elements:
  <a href="https://www.ibjjfdb.com/ChampionshipResults/926/PublicResults" rel="nofollow noreferrer">https://www.ibjjfdb.com/ChampionshipResults/926/PublicResults</a> </p>
  
  <p>Any ideas would be helpful. Thank you!</p>