title: BeautifulSoup removing nested tags
body: <p>I am trying to make a generic scraper using BeautifulSoup for which I am trying to detect the tag under which directly text is available.</p>
  
  <p>Consider this example:</p>
  
  <pre><code>&lt;body&gt;
  &lt;div class="c1"&gt;
      &lt;div class="c2"&gt;
          &lt;div class="c3"&gt;
              &lt;div class="c4"&gt;
                  &lt;div class="c5"&gt;
                      &lt;h1&gt; A heading for section &lt;/h1&gt;
                  &lt;/div&gt;
                  &lt;div class="c5"&gt;
                      &lt;p&gt; Some para &lt;/p&gt;
                  &lt;/div&gt;
                  &lt;div class="c5"&gt;
                      &lt;h2&gt; Sub heading &lt;/h2&gt;
                      &lt;p&gt; &lt;span&gt; Blah Blah &lt;/span&gt; &lt;/p&gt;
                  &lt;/div&gt;
              &lt;/div&gt;
          &lt;/div&gt;
      &lt;/div&gt;
  &lt;/div&gt;
  &lt;/body&gt;
  </code></pre>
  
  <p>Here my objective is to extract (div with class c4) as it has all the textual content. Rest of the div before it c1 - c3 are just wrappers for me.</p>
  
  <p>One possible way for identifying the node, I came up is:</p>
  
  <pre><code>if node.find(re.compile("^h[1-6]"), recursive=False) is not None:
      return node.parent.parent
  </code></pre>
  
  <p>But it is too specific for this case.</p>
  
  <p>Is there any optimized way for finding text in one level of recursion. i.e. if I do something like</p>
  
  <pre><code>node.find(text=True, recursion_level=1)
  </code></pre>
  
  <p>then it should return text considering only immediate children.</p>
  
  <p>My solution so far, not sure if it holds for all cases.</p>
  
  <pre><code>def check_for_text(node):
      return node.find(text=True, recursive=False)
  
  def check_1_level_depth(node):
      if check_for_text(node):
          return check_for_text(node)
  
      return map(check_for_text, node.children)
  </code></pre>
  
  <p>For the code above: node is an element of soup that is currently under check, i.e. div, span, etc.
  Please assume that I am handling all exceptions in check_for_text() (AttributeError: 'NavigableString')</p>