Ask HN: Is Hackernews still using the file system as database?
While on the "how HN works" topic, I have a question:<p>I'm working on an HN app reader idea that I think is unique (more to come) that requires regularly getting the front page posts and comments. I wrote a script to do that through the Firebase API but holy shit it ended up needing thousands of requests to get all the data for the current state of the front page.<p>So instead, I wrote a scraper script to produce the same thing, just with 30 requests (1 per front page post). (Trying to turn these damn <tr>'s into a recursive comment tree was a quite the mindfuck btw)<p>...is it ok to scrape HN? I see nothing in the robots.txt to say it shouldn't be allowed, and I actually feel better making 30 scraping requests rather than XXXX API requests.<p>EDIT: Also happy to receive suggestions. I'm coding in Ruby and don't see an obvious way to access the Firebase DB directly (quite frankly I don't even know if what I just said makes sense) so any help is appreciated.