One of the startups I worked on was a classifieds aggregation engine pulling data from external feeds.<p>We basically queued all retrieved items for processing with no attempts at avoiding data loss whatsoever - including using in memory queues for lots of things.<p>Our reasoning was that if a machine crashed, worst case was that a few listings would take up to 24 hours to update, but generally much less (we adapted crawling rate for our sources based on change frequency; so large sources of listings would get re-indexed far more frequently, so if a feed didn&#x27;t update or 24 hours it&#x27;d be because it wasn&#x27;t a source of much data), and we could force refreshes of the data.<p>Some people were horrified at the approach because the idea of ensuring consistency and not losing data is so ingrained. But the reality is that you need to measure the cost of consistency up against the value it provides. And often it&#x27;s not very valuable, <i>especially</i> when there is an authoritative source of the data to recover from and when the data will be outdated quickly anyway.<p>A lot of the time any notion of consistency is an illusion anyway - by the time the page has returned, the results are outdated - and what matters is maintaining the illusion (e.g. ensure that if a user makes an update it&#x27;s reflected in the page that&#x27;s returned).<p>The key is you need to know the tradeoffs and apply them consciously rather than get caught out by tradeoffs components you rely on makes without telling you..