Here are a few suggestions based on 6+ years in adtech (which have just came to a close, never again thank you):<p>* Use a Queue. RabbitMQ is quite good. Instead of writing to files, generate data&#x2F;tasks on the queue and have them consumed by more than one client. The clients should handle inserting the data to the database. You can control the pipe by the number of clients you have consuming tasks, and&#x2F;or by rate limiting them. Break those queue consuming clients to small pieces. Its ok to queue item B on the queue while processing item A.<p>* If you data is more fluid and changing all the time, and&#x2F;or if it comes in JSON serializable format, consider switching to postgresql ^9.4, and use the JSONB columns to store this data. You can index&#x2F;query those columns and performance wise its on par (or surpasses) MongoDB.<p>* Avoid AWS at this stage. like commented by someone here - bare metal is a better friend to you. You&#x27;ll also know exactly how much you&#x27;re paying each month. no surprises. I can&#x27;t recommend Softlayer enough.<p>* Don&#x27;t over complicate things. If you can think of a simple solution to something - its preferable than the complicated solution you might have had before.<p>* If you&#x27;re going the queue route suggested above, you can pre-process the data while you get it in. If its going to be placed into buckets, do it then, if its normalised - do it then. The tasks on the queue should be atomic and idempotent. You can use something like memcached if you need your clients to communicate between eachother (like checking if a queue item is not already processed by another consumer and thus is locked).