Disclaimer here: I'm really not a programmer. I'm eager to learn, but my experience is pretty much basic on c64 20 years ago and a couple of days of learning Python.
I'm just starting out on a fairly large (for me as a beginner) screen scraping project. So far I have been using python with mechanize+lxml for my browsing/parsing. Now I'm encountering some really javascript heavy pages that doesn't show a anything without javascript enabled, which means trouble for mechanize.
From my searching I've kind come to the conclusion that I have a basically a few options:

Trying to figure out what the javascript is doing a emulate that in my code (I don't quite know where to start with this. ;-))
Using pywin32 to control internet explorer or something similar, like using the webkit-browser from pyqt4 or even using telnet and mozrepl (this seems really hard)
Switching language to perl since www::Mechanize seems be a lot more mature on per (addons and such for javascript). Don't know too much about this at all. 

If anyone has some pointers here that would be great. I understand that I need to do a lot of trial and error, but would be nice I wouldn't go too far away from the "true" answer, if there is such a thing.
For nonprogrammers, I recomment using IRobotSoft.  It is visual oriented and with full javascript support.  The shortcoming is that it runs only on Windows.  The good thing is you can become an expert just by trial and error to learn the software.
Options for handling javascript heavy pages while screen scraping