Introduction
Amazon EMR is a web service which can be used to easily and efficiently process enormous amounts of data. It uses a hosted Hadoop framework running on the web-scale infrastructure of Amazon EC2 and Amazon S3.
Amazon EMR removes most of the cumbersome details of Hadoop while taking care of provisioning of Hadoop, running the job flow, terminating the job flow, moving the data between Amazon EC2 and Amazon S3, and optimizing Hadoop.
In this tutorial, we will use a developed WordCount Java example using Hadoop and thereafter, we execute our program on Amazon Elastic MapReduce.
Prerequisites
You must have valid AWS account credentials. You should also have a general familiarity with using the Eclipse IDE before you begin. The reader can also use any other IDE of their choice.
Step 1 – Develop MapReduce WordCount Java Program
In this section, we are first going to develop a WordCount application. A WordCount program will determine how many times different words appear in a set of files.
- In Eclipse (or whatever the IDE you are using), Create simple Java Project with the name "WordCount".
- Create a java class name Map and override the map method as follow,
public class Map extends Mapper<longwritable, text,="" intwritable=""> {); } } }
- Create a java class named Reduce and override the reduce method as shown below,
public class Reduce extends Reducer<text, intwritable,="" text,="" intwritable=""> { @Override protected void reduce(Text key, java.lang.Iterable<intwritable> values, org.apache.hadoop.mapreduce.Reducer<text, intwritable,="" text,="" intwritable="">.Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable value : values) { sum += value.get(); } context.write(key, new IntWritable(sum)); } }
- Create a java class named WordCount); }
- Export the WordCount program in a jar using eclipse and save it to some location on disk. Make sure that you have provided the Main Class (WordCount.jar) during extraction ofu8u the jar file as shown below.
Our jar is ready!!!
Step 2 – Upload the WordCount JAR and Input Files to Amazon S3
Now we are going to upload the WordCount jar
Now )
- Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at
- Click Create New Job Flow.
- In the DEFINE JOB FLOW page, enter the following details,
- a) Job Flow Name = WordCountJob
b) Select Run your own applications) Select Custom JAR in the drop-down list) Click Continue
- In the SPECIFY PARAMETERS page, enter values in the boxes using the following table as a guide, and then click Continue.JAR Location = bucketName/jarFileLocationJAR Arguments =s3n://bucketName/inputFileLocations3n://bucketName/outputpath
Please note that the output path must be unique each time we execute the job. The Hadoop always create a folder with the same name specified here.
After executing the job, just wait and monitor your job that runs through the Hadoop flow. You can also look for errors by using the Debug button. The job should be complete within 10 to 15 minutes (can also depend on the size of the input). After completing the,
See more at
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}{{ parent.urlSource.name }}