title: How to read compressed bz2 (bzip2) Wikipedia dumps into stream xml record reader for hadoop map reduce
body: <p>I am working on using Hadoop Map Reduce to do research on the wikipedia data dumps (compressed in bz2 format). Since these dumps are so big (5 T), I can't decompress the xml data into HDFS and just use the StreamXmlRecordReader that hadoop provides. Hadoop does support uncompressing bz2 files, but it splits the pages arbitrarily and sends those to the mapper. Because this is xml, we need the splits to be a  tags. Is there anyway to use the built in bz2 decompression and stream xml record reader provided by hadoop together?</p>