I want to parse some data from a website. However, there's a certain peculiarity:
There is a dropdown list (layed out using div and child a tags, made functional with a jQuery script). Upon selecting one of the values, a subsequent text field would change its value.
I want to retrieve the first dropdown value, and the respective text field, then the next dropdown value and the updated text field, and so forth.
How would I go about this?
What happens here is, upon selecting a value form the dropdown, an AJAX request is generated and gets the data.
You can analyze the request url in your browser. If you use firefox, use firebug and take a look at the Net tab, what requests are generating and what is the url. In google chorme, look in Network tab. If you want to parse the data you have to make a request in that url.
Python/lxml: Retrieving variable data