Question: I am trying to get the product name and the quantity from a nested HTML table using selenium. My problem is some of the divs don't have any id or class names. The table I am trying to access is the **Critical Product list**. Here is what I have done but I do seem to be lost at how I can get the nested divs. *The site is in the code.*

```
options = Options()
options.add_argument('start-maximized')

driver = webdriver.Chrome(chrome_options=options, executable_path=r'/usr/local/bin/chromedriver/')
url = 'https://www.rrpcanada.org/#/' # site I'm scraping
driver.get(url)
time.sleep(150)
page = driver.page_source
driver.quit()

html_soup = BeautifulSoup(page, 'html.parser')
item_containers = html_soup.find_all('div', class_='critical-products-title hide-mobile')

if item_containers:
    for item in item_containers:
       for link in item.findAll('a', ) # need to loop the inner divs to reach the href and then get to the left and right classes to get title and quantity
        print(item)

```

Here is the image from the inspection. I want to be able to loop through all the divs and get the title and quantity.

[![enter image description here](https://i.stack.imgur.com/E1RE9.png)](https://i.stack.imgur.com/E1RE9.png)

Answer: 
You don't need beautiful soup, nor to save the page\_source.
I used a CSS selector to select all the target rows in the table and then applied list comprehension to choose the left and right sides of each row. I outputted the results to a list of tuples.

```
options = Options()
options.add_argument('start-maximized')

driver = webdriver.Chrome(chrome_options=options, executable_path=r'/usr/local/bin/chromedriver/')
url = 'https://www.rrpcanada.org/#/' # site I'm scraping
driver.get(url)
time.sleep(150)

elements = driver.find_elements_by_css_selector('#app > div:nth-child(1) > div.header-wrapper > div.header-right > div.critical-product-table-container > div.table.shorten.hide-mobile > div > div > div > a > div')

targetted_values = [(element.find_element_by_css_selector('.line-item-left').text, element.find_element_by_css_selector('.line-item-right').text) for element in elements]

driver.quit()

```

Example output of targetted\_values:

```
[('Surgical & Reusable Masks', '376,713,363 available'),
('Disposable Gloves', '66,962,093 available'),
('Gowns and Coveralls', '40,502,145 available'),
('Respirators', '22,189,273 available'),
('Surface Wipes', '20,650,831 available'),
('Face Shields', '16,535,686 available'),
('Hand Sanitizer', '11,152,890 L available'),
('Thermometers', '8,457,993 available'),
('Testing Kits', '2,110,815 available'),
('Surface Solutions', '107,452 L available'),
('Protective Barriers', '10,833 available'),
('Ventilators', '410 available')]

```
This is the selector for **`product name`**:

```
div.critical-product-table-container div.line-item-left

```

And for **`total`**:

```
div.critical-product-table-container div.line-item-right

```

But the following approach is without `BeautifulSoup`.

`time.sleep(...)` is bad practice, please use [**`WebDriverWait`**](https://selenium-python.readthedocs.io/waits.html#explicit-waits) instead.

And to pair the above two variables and perform parallel looping, I try to use the [**`zip()`**](https://docs.python.org/3.3/library/functions.html#zip) function:

```
url = 'https://www.rrpcanada.org/#/' # site I'm scraping
driver.get(url)
wait = WebDriverWait(driver, 150)
product_names = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'div.critical-product-table-container div.line-item-left')))
totals = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'div.critical-product-table-container div.line-item-right')))

for product_name, total in zip(product_names, totals):
    print(product_name.text +'--' +total.text)

driver.quit()

```

You need following import:

```
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

```