Setting up a python screen scraper that could work on Google App engine
I have used BeautifulSoup with great success parsing HTML.  Problem is that's all BeautifulSoup does, is parse the HTML.  I ended up writing all the http interactions using urlfetch.  
To web-scrape my target I need a full fledged code driven browser that can execute javascript on my target site's pages.  I think I'm having to dump the python app and go java so I can use HTMLUnit - prototyping underway. - mattb
I am looking to setup a automated screen scraper that will run on Google app engine using python. I want it to scrape the site and put the specified results into a Entity in app engine. I am looking for some directions on what to use. I have seen beautifulsoup but wonder if people could recommend anything else that could run on Google App engine.