Question: I'm trying to scrape a website and want to get all the descendants of a specific div class. For example, suppose I have a website that looks like the following:

[Edit: The question's author indicates in the comments that all of the `div` elements should be at the same level; I've therefore taken the liberty of closing them in this sample code.]

```
<div class = "blah">
    <p></p>
</div>

<div class = "i-want-this">
    <p></p>
    <p><a href= "http://www.google.com"></a></p>
</div>

<div class = "i-want-this">
    <p></p>
    <li></li>
        <p>meh</p>
    <li></li>
</div>

```

I want all the descendants of each instance of the div class "i-want-this" and to ignore the other divs. I can specify those divs in a find\_all

```
div = soup.find_all('div', {'class': 'i-want-this'})

```

But that just creates a list of all of that. I also see that you can grab descendants by

```
soup.div.descendants

```

But I don't know how to specify which classes of div tags I want to include. I'd appreciate any help!

Answer: 
Ultimately, I came up with this solution. The "children" object captured all children of 'div' and successive children; then I iterated over that

```
children = soup.findChildren('div', {'class': 'i-want-this'})

content = []
for item in children:   
    item = [content for content in item.text.split('\n') if len(content)>0]

    # Create string from separate list items to all be listed in content
    item = ' '.join(item)
    content.append(item)

```
I shoved your HTML into a file called `temp.htm`.

You need only a tiny part of scrapy for tasks like this: `Selector`. Just feed your HTML into it, then use its `xpath` method.

In this case I can specify one of the two `div` elements with the `class` of interest, by number, then ask for all of its descendants and then `extract` what's in those. The result in each case is a list of elements descending from the children of the `div`.

```
>>> from scrapy.selector import Selector
>>> HTML = open('temp.htm').read()
>>> selector = Selector(text=HTML)
>>> selector.xpath('.//div[@class="i-want-this"][1]/*').extract()
['<p></p>', '<p><a href="http://www.google.com"></a></p>']
>>> selector.xpath('.//div[@class="i-want-this"][2]/*').extract()
['<p></p>', '<li>', '<p>meh</p>', '<li>']

```

[The xpath cheatsheet](http://ricostacruz.com/cheatsheets/xpath.html) can be really useful at times like these.