Figure 26 Generate the Index Files with Live Mode in Virtual Cloudera Note: If you get a Java heap space error, adjust the values of the mapreduce.map.java.opts and mapreduce.reduce.java.opts in the file /etc/hadoop/conf/mapred-site.xml. Figure 27 Fix the Java Heap Error in Virtual Cloudera Use Lily Indexer for Data Indexing (Batch Mode) The live mode of Lily Indexer is simple and efficient, because it makes full use of the memory. However, it cannot deal with big data. The batch mode of Lily indexer is helpful for indexing a large dataset. Create a Lily Indexer configuration file. This step is the same as the one in live mode. Run HBaseMapReduceIndexer tool. We index the HBase table using MapReduce in batch mode. The following command helps us to output the indexes into HDFS. Key Command(s) hadoop --config /etc/hadoop/conf jar /usr/lib/hbase-solr/tools/hbase-indexer-mr-*-job.jar /etc/hbase/conf/hbase-site.xml -D 'mapred.child.java.opts=-Xmx500m' [LOCAL_DIR]/morphline-hbase-mapper.xml --zk-host 127.0.0.1/solr --log4j [LOCAL_DIR]/log4j.properties --collection [COLLECTION_NAME] --verbose --output-dir hdfs://quickstart.cloudera/user/cloudera/cs5604f16-small-index --overwrite-output-dir 1 Figure 28 Generate the Index Files with Batch Mode in Virtual Cloudera the files in HDFS. If the MapReduce job goes successfully, then multiple indexing files have already been generated in HDFS. Key Command(s) hadoop fs -ls /solr/cs5604f16-small/core_node1/data/index Figure 29 the Index Files in HDFS in Virtual Cloudera
Figure 26 Generate the Index Files with Live Mode in Virtual Cloudera Note: If you get a Java heap space error, adjust the values of the mapreduce.map.java.opts and mapreduce.reduce.java.opts in the file /etc/hadoop/conf/mapred-site.xml. Figure 27 Fix the Java Heap Error in Virtual Cloudera Use Lily Indexer for Data Indexing (Batch Mode) The live mode of Lily Indexer is simple and efficient, because it makes full use of the memory. However, it cannot deal with big data. The batch mode of Lily indexer is helpful for indexing a large dataset. Create a Lily Indexer configuration file. This step is the same as the one in live mode. Run HBaseMapReduceIndexer tool. We index the HBase table using MapReduce in batch mode. The following command helps us to output the indexes into HDFS. Key Command(s) hadoop --config /etc/hadoop/conf jar /usr/lib/hbase-solr/tools/hbase-indexer-mr-*-job.jar /etc/hbase/conf/hbase-site.xml -D 'mapred.child.java.opts=-Xmx500m' [LOCAL_DIR]/morphline-hbase-mapper.xml --zk-host 127.0.0.1/solr --log4j [LOCAL_DIR]/log4j.properties --collection [COLLECTION_NAME] --verbose --output-dir hdfs://quickstart.cloudera/user/cloudera/cs5604f16-small-index --overwrite-output-dir 1 Figure 28 Generate the Index Files with Batch Mode in Virtual Cloudera the files in HDFS. If the MapReduce job goes successfully, then multiple indexing files have already been generated in HDFS. Key Command(s) hadoop fs -ls /solr/cs5604f16-small/core_node1/data/index Figure 29 the Index Files in HDFS in Virtual Cloudera
SOLR_Report.docx