Question: I am trying to make an application that requires a lot of data scraping from multiple websites. I tried scraping websites using Ruby but gems such as Mechanize only seem to scrape static pages and not dynamic content. I have a couple questions regarding which of these languages, or any other language, I should use for this project (I am considering using Node because quite a few elements in the application have to be in real time).

1. Is it possible to use Ruby and/or Node to scrape dynamic content? If so which tools specifically should be used?
2. If multiple users are going to be scraping from multiple sites, which language would you recommend using?
3. On a slightly unrelated note, is it possible to combine Node and Rails?

Thanks in advance!

Answer: 
You can utilize the `capybara` gem for scraping javascript sites using ruby.

This has the advantage of being able to use actual browsers such as Firefox, Chrome and IE through the `selenium` driver. Or you can use headless browsers such as webkit (via capybara-webkit) or phantomjs (via poltergeist).

When you use capybara, just be sure to use a javascript enabled driver, such as selenium or capybara-webkit. My driver of the day is poltergeist.

There are some instructions for how to use capybara with remote sites in [their readme](https://github.com/jnicklas/capybara#calling-remote-servers).

Node vs. Ruby is a very open ended question. My answer here is suggesting Ruby because that is my experience and preference. "Combining" them could mean many things, they can be used in concert, each playing to their strengths.
When you say that mechanize can't scrape dynamic content, you really mean that it's a little bit more work to figure out which ajax requests need to be made and make them. The other side of that is that once you do you generally get a nice json response that's easy to deal with. Mechanize is also much faster than a full browser solution so my opinion is that it's usually worth the extra work.

As far as Node goes, there's potential and maybe once it's been around for a while some great libraries will become available, but I haven't seen anything yet that would make up for the ruby things I wiss miss.