Maybe you can check out HSearch - the real time, distributed open source search engine built on top of Hadoop and Hbase. Btw I am a developer of HSearch, so take it as a biased opinion.
Jiri Kehar , Owner of XTRA Software s.r.o., Java Developer at XTRA Software, s.r.o.
14. Ok, that’s cool, but… ¿Howdoes it solve my problem?
17. Philosophy	§  Always reprocess everything. ¡EVERYTHING!	§  ¿Why?	•  More bug tolerant	•  More ﬂexible	•  More eﬃcient. E.g.:	ª  With a 7200 RPM HD	–  Random IOPS – 100 –  Sequencial Read/Write – 40 MB/s	–  Hypothesis: 5 Kb register size	ª  … it is faster to rewrite all data than to perform random updates when more than 1.25% of the registers has changed. –  1 GB, 200.000 registers	»  Sequential writing: 25 sg	»  Random writing: 33 min!
20. Processor (II) §  Possible problem:	•  Very large feeds	ª Does not scale, as one task will deal with the full feed. §  Solution	•  Write a custom InputFormat that divides the feed in smaller pieces.
26. Advantages of the architecture	§  Horizontal Scalability	•  If properly programmed	§  High tolerance to failures and bugs	•  Always everything is reprocessed	§  Flexible	•  It is easy to do big changes	§  High decoupling	•  Indexes are the unique interaction between the back-end and the front-end	•  Web servers can keep running even if the back- end is broken.