<repo_name>SteBurz/python-selenium-web-scraper<file_sep>/README.md
# python-selenium-web-scraper

This repo is an example how a web scraper with python an selenium could look like.
The program is:

- get a title of a website
- get a text of a HTML Element
- perform a click event
- input text into a field and
- how to inject custom script.

More can be watched here: https://youtu.be/Aum4-GRCzxc

## Install

How to install pip: https://phoenixnap.com/kb/install-pip-windows
Chrome Webdriver: https://sites.google.com/a/chromium.org/chromedriver/downloads
Using selenium: https://selenium-python.readthedocs.io/

<file_sep>/scraper.py
from selenium import webdriver

DRIVER_PATH = 'C:\Program Files\chromedriver.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get('example.com')

websiteTitle = driver.title
firstPost = driver.find_element_by_xpath('//*[@id="post-7"]/div/header/h2/a')

if websiteTitle:
    print(websiteTitle)
else: 
    print("Couldn't get title")


if firstPost.text:
    print(firstPost.text)
    firstPost.click()
    excerpt = driver.find_element_by_xpath('//*[@id="post-7"]/div/div/p[1]')

    if(excerpt.text):
        print(excerpt.text)

        commentTextField = driver.find_element_by_xpath('//*[@id="comment"]')
        commentNameField = driver.find_element_by_xpath('//*[@id="author"]')
        commentEmailField = driver.find_element_by_xpath('//*[@id="email"]')
        commentSubmitButton = driver.find_element_by_xpath('//*[@id="submit"]')

        commentTextField.send_keys("Some content here")
        commentNameField.send_keys("<NAME>")
        commentEmailField.send_keys("<EMAIL>")

        driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
        commentSubmitButton.click()

    else:
        print('Couldn\'t find excerpt for post.')
else:
    print('Couldn\'t find first post')

driver.quit()
