tools for Crawling popular forum/bulletin board software
I've started writing a crawler to crawl vbulletin boards. However, I am not a web programmer (json api's I can do, but that isn't really web-crawling), and as such I do not know what the best way to crawl is, and what tools are available. 

I am more than capable of writing the crawler, but the I find the underlying HTML very irregular, and so I don't want to be a victim of the structure of the HTML changing in a newer version of vbulletin.

I'm writing an interface using pycurl and beautiful soup. However, is there a better way to do this, are there any good crawlers already available for vbulletin ? (language is not a concern). A meta forum crawler (works with more than one forum type) would be even better. 

If you cannot suggest one, could you advise me, if you have the experience, from what I should expect from the stability of the underlying HTML, should I worry about a new version of vbulletin breaking my crawler ? 

Perhaps there is a better way to extract a vbulletin dataset ?
tools for Crawling popular forum/bulletin board software
===

I've started writing a crawler to crawl vbulletin boards. However, I am not a web programmer (json api's I can do, but that isn't really web-crawling), and as such I do not know what the best way to crawl is, and what tools are available. 

I am more than capable of writing the crawler, but the I find the underlying HTML very irregular, and so I don't want to be a victim of the structure of the HTML changing in a newer version of vbulletin.

I'm writing an interface using pycurl and beautiful soup. However, is there a better way to do this, are there any good crawlers already available for vbulletin ? (language is not a concern). A meta forum crawler (works with more than one forum type) would be even better. 

If you cannot suggest one, could you advise me, if you have the experience, from what I should expect from the stability of the underlying HTML, should I worry about a new version of vbulletin breaking my crawler ? 

Perhaps there is a better way to extract a vbulletin dataset ?