I've been reading about beautifulSoup, http headers, authentication, cookies and something about mechanize.
I'm trying to scrape my favorite art websites with python. Like deviant art which I found a scraper for. Right now I'm trying to login but the basic authentication code examples I try don't work.
So question, How do I find out what type of authentication a site uses so that I know I'm trying to login the correct way? Including things like valid user-agents when they try to block bots.
Bear with my ignorance as I'm new to HTTP, python, and scraping.
It's very unlikely that any of the sites you are interested in use basic auth. You will need a library like mechanize that manages cookies and you will need to submit the login information to the site's login page.
how to find the authentication used on a website