title: Hadoop 2.2.0 on Windows: Job is Successful but just outputs the input file
body: <p>I downloaded the source code for Hadoop 2.2.0 and all of the relevant dependencies and built it/installed it/configured it. Note that I built it using MinGW from Git rather than Cygwin, if that matters. </p>
  
  <p>I am running a job but the only output that I get is the input file. Logging from my mapper or reducer is not working, either - in fact, I don't even know how to get Hadoop to tell me if it is running my specific class implementation. </p>
  
  <p>Mapper code:</p>
  
  <pre><code>package org.test.dummy;
  
  import java.io.IOException;
  
  import org.apache.hadoop.io.LongWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Mapper;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  
  public class DummyMapper extends Mapper&lt;LongWritable, Text, Text, LongWritable&gt;{
    private static final Logger log = LoggerFactory.getLogger(DummyMapper.class);
    private Text character = new Text();
    private LongWritable one = new LongWritable(1);
  
    /** @see org.apache.hadoop.mapreduce.Mapper#map(java.lang.Object, java.lang.Object, org.apache.hadoop.mapreduce.Mapper.Context) */
    @Override
    protected void map(
      LongWritable id, 
      Text line, 
      org.apache.hadoop.mapreduce.Mapper&lt;LongWritable, Text, Text, LongWritable&gt;.Context context) 
    throws IOException, InterruptedException 
    {
      log.info("Starting Dummy mapper");
      for(char c : line.toString().toCharArray()) {
        this.character.set(Character.toString(c));
        context.write(this.character, this.one);
      }
    }
  }
  </code></pre>
  
  <p>Reducer code:</p>
  
  <pre><code>package org.test.dummy;
  
  import java.io.IOException;
  
  import org.apache.hadoop.io.LongWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Reducer;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  
  public class DummyReducer extends Reducer&lt;Text, LongWritable, Text, LongWritable&gt; {
    private static final Logger log = LoggerFactory.getLogger(DummyReducer.class);
    private LongWritable count = new LongWritable(0);
  
    /** @see org.apache.hadoop.mapreduce.Reducer#reduce(java.lang.Object, java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer.Context) */
    @Override
    protected void reduce(
        Text character, 
        Iterable&lt;LongWritable&gt; tickMarks, 
        org.apache.hadoop.mapreduce.Reducer&lt;Text, LongWritable, Text, LongWritable&gt;.Context context) 
    throws IOException, InterruptedException 
    {
      log.info("Starting Dummy reducer");
      long count = 0;
      for(@SuppressWarnings("unused") LongWritable tick : tickMarks) count++;
      this.count.set(count);
      context.write(character, this.count);
    }
  }
  </code></pre>
  
  <p>Driver code:</p>
  
  <pre><code>package org.test.dummy;
  
  import org.apache.hadoop.conf.Configuration;
  import org.apache.hadoop.fs.Path;
  import org.apache.hadoop.io.LongWritable;
  import org.apache.hadoop.io.Text;
  import org.apache.hadoop.mapreduce.Job;
  import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  
  public class Dummy {
    private static final Logger log = LoggerFactory.getLogger(Dummy.class);
  
    /** @param args */
    public static void main(String[] args) {
      try {
        log.info("Starting Dummy driver");
        Job job = Job.getInstance(new Configuration(), "dummy-job");
        job.setJarByClass(Dummy.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);
        job.setMapperClass(DummyMapper.class);
        job.setReducerClass(DummyReducer.class);
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        boolean success = job.waitForCompletion(true) ;
        System.exit(success ? 0 : 1);
      } catch(Exception e) {
        e.printStackTrace(System.err);
        System.exit(1);
      }
    }
  }
  </code></pre>
  
  <p>Maven snippet in POM to set the driver class for Hadoop, so that the main class doesn't need to be set on the command line:</p>
  
  <pre><code>  &lt;plugin&gt;
      &lt;artifactId&gt;maven-jar-plugin&lt;/artifactId&gt;
      &lt;configuration&gt;
        &lt;archive&gt;
          &lt;manifest&gt;
            &lt;mainClass&gt;org.test.Dummy&lt;/mainClass&gt;
          &lt;/manifest&gt;
        &lt;/archive&gt;
      &lt;/configuration&gt;
    &lt;/plugin&gt;
  </code></pre>
  
  <p>Process to scrub clean and reset my Hadoop environment to guarantee clean execution (note that I am using MinGW that comes with Git for my Unix commands):</p>
  
  <pre><code>SET JAVA_HOME=c:\jdk7
  SET HADOOP_HOME=c:\hadoop
  rmdir c:\tmp /S /Q
  rmdir %HADOOP_HOME%\data /S/Q
  mkdir %HADOOP_HOME%\data\dfs
  mkdir %HADOOP_HOME%\data\dfs\namenode
  mkdir %HADOOP_HOME%\data\dfs\datanode
  call %HADOOP_HOME%\bin\hdfs namenode -format
  call %HADOOP_HOME%\sbin\start-dfs.cmd
  call %HADOOP_HOME%\sbin\start-yarn.cmd
  call hdfs dfs -mkdir /in
  call bash -c "hdfs dfs -copyFromLocal ./src/test/resources/my-file.txt hdfs://localhost:9000/in"
  call %HADOOP_HOME%\bin\yarn jar target\my-hadoop-job.jar /in/my-file.txt /out
  call %HADOOP_HOME%\bin\hdfs dfs -ls -R /out
  </code></pre>
  
  <p>Here is the relevant output of this process:</p>
  
  <pre><code>Formatting using clusterid: CID-9d811b80-42cb-4cc6-aa0b-e84e349701d5
  14/02/04 12:00:45 INFO namenode.HostFileManager: read includes:
  HostSet(
  )
  14/02/04 12:00:45 INFO namenode.HostFileManager: read excludes:
  HostSet(
  )
  14/02/04 12:00:45 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
  14/02/04 12:00:45 INFO util.GSet: Computing capacity for map BlocksMap
  14/02/04 12:00:45 INFO util.GSet: VM type       = 64-bit
  14/02/04 12:00:45 INFO util.GSet: 2.0% max memory = 888.9 MB
  14/02/04 12:00:45 INFO util.GSet: capacity      = 2^21 = 2097152 entries
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: defaultReplication         = 1
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: maxReplication             = 512
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: minReplication             = 1
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
  14/02/04 12:00:45 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
  14/02/04 12:00:45 INFO namenode.FSNamesystem: fsOwner             = some.one (auth:SIMPLE)
  14/02/04 12:00:45 INFO namenode.FSNamesystem: supergroup          = supergroup
  14/02/04 12:00:45 INFO namenode.FSNamesystem: isPermissionEnabled = true
  14/02/04 12:00:45 INFO namenode.FSNamesystem: HA Enabled: false
  14/02/04 12:00:45 INFO namenode.FSNamesystem: Append Enabled: true
  14/02/04 12:00:45 INFO util.GSet: Computing capacity for map INodeMap
  14/02/04 12:00:45 INFO util.GSet: VM type       = 64-bit
  14/02/04 12:00:45 INFO util.GSet: 1.0% max memory = 888.9 MB
  14/02/04 12:00:45 INFO util.GSet: capacity      = 2^20 = 1048576 entries
  14/02/04 12:00:45 INFO namenode.NameNode: Caching file names occuring more than 10 times
  14/02/04 12:00:45 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
  14/02/04 12:00:45 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
  14/02/04 12:00:45 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
  14/02/04 12:00:45 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
  14/02/04 12:00:45 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
  14/02/04 12:00:45 INFO util.GSet: Computing capacity for map Namenode Retry Cache
  14/02/04 12:00:45 INFO util.GSet: VM type       = 64-bit
  14/02/04 12:00:45 INFO util.GSet: 0.029999999329447746% max memory = 888.9 MB
  14/02/04 12:00:45 INFO util.GSet: capacity      = 2^15 = 32768 entries
  14/02/04 12:00:45 INFO common.Storage: Storage directory \hadoop\data\dfs\namenode has been successfully formatted.
  14/02/04 12:00:45 INFO namenode.FSImage: Saving image file \hadoop\data\dfs\namenode\current\fsimage.ckpt_0000000000000000000 using no compression
  14/02/04 12:00:45 INFO namenode.FSImage: Image file \hadoop\data\dfs\namenode\current\fsimage.ckpt_0000000000000000000 of size 208 bytes saved in 0 seconds.
  14/02/04 12:00:45 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid &gt;= 0
  14/02/04 12:00:45 INFO util.ExitUtil: Exiting with status 0
  14/02/04 12:00:45 INFO namenode.NameNode: SHUTDOWN_MSG:
  /************************************************************
  SHUTDOWN_MSG: Shutting down NameNode at xxxxxxxxxxxxx
  ************************************************************/
  ....
  14/02/04 12:31:31 INFO test.Dummy: Starting Dummy driver
  14/02/04 12:31:33 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
  14/02/04 12:31:33 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with
  ToolRunner to remedy this.
  14/02/04 12:31:34 INFO input.FileInputFormat: Total input paths to process : 1
  14/02/04 12:31:34 INFO mapreduce.JobSubmitter: number of splits:1
  14/02/04 12:31:34 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
  14/02/04 12:31:34 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  14/02/04 12:31:34 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
  14/02/04 12:31:34 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
  14/02/04 12:31:34 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
  14/02/04 12:31:34 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
  14/02/04 12:31:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1391509606870_0001
  14/02/04 12:31:34 INFO impl.YarnClientImpl: Submitted application application_1391509606870_0001 to ResourceManager at /127.0.0.1:8032
  14/02/04 12:31:34 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1391509606870_0001/
  14/02/04 12:31:34 INFO mapreduce.Job: Running job: job_1391509606870_0001
  14/02/04 12:31:43 INFO mapreduce.Job: Job job_1391509606870_0001 running in uber mode : false
  14/02/04 12:31:43 INFO mapreduce.Job:  map 0% reduce 0%
  14/02/04 12:31:50 INFO mapreduce.Job:  map 100% reduce 0%
  14/02/04 12:31:57 INFO mapreduce.Job:  map 100% reduce 100%
  14/02/04 12:31:58 INFO mapreduce.Job: Job job_1391509606870_0001 completed successfully
  14/02/04 12:31:58 INFO mapreduce.Job: Counters: 43
          File System Counters
                  FILE: Number of bytes read=1395
                  FILE: Number of bytes written=133855
                  FILE: Number of read operations=0
                  FILE: Number of large read operations=0
                  FILE: Number of write operations=0
                  HDFS: Number of bytes read=1275
                  HDFS: Number of bytes written=1243
                  HDFS: Number of read operations=6
                  HDFS: Number of large read operations=0
                  HDFS: Number of write operations=2
          Job Counters
                  Launched map tasks=1
                  Launched reduce tasks=1
                  Data-local map tasks=1
                  Total time spent by all maps in occupied slots (ms)=4357
                  Total time spent by all reduces in occupied slots (ms)=4717
          Map-Reduce Framework
                  Map input records=24
                  Map output records=24
                  Map output bytes=1340
                  Map output materialized bytes=1395
                  Input split bytes=104
                  Combine input records=0
                  Combine output records=0
                  Reduce input groups=24
                  Reduce shuffle bytes=1395
                  Reduce input records=24
                  Reduce output records=24
                  Spilled Records=48
                  Shuffled Maps =1
                  Failed Shuffles=0
                  Merged Map outputs=1
                  GC time elapsed (ms)=64
                  CPU time spent (ms)=1184
                  Physical memory (bytes) snapshot=397262848
                  Virtual memory (bytes) snapshot=641183744
                  Total committed heap usage (bytes)=334495744
          Shuffle Errors
                  BAD_ID=0
                  CONNECTION=0
                  IO_ERROR=0
                  WRONG_LENGTH=0
                  WRONG_MAP=0
                  WRONG_REDUCE=0
          File Input Format Counters
                  Bytes Read=1171
          File Output Format Counters
                  Bytes Written=1243
  ...
  -rw-r--r--   1 some.one supergroup          0 2014-02-04 12:31 /out/_SUCCESS
  -rw-r--r--   1 some.one supergroup       1243 2014-02-04 12:31 /out/part-r-00000    
  </code></pre>
  
  <p>When I call <code>hdfs dfs -cat /out/part-r-00000</code>, though, I only get a copy of my input file.  </p>
  
  <p>The Hadoop job logs tell me nothing: stdout and stderr logs are 0 bytes; the syslog is 37+kb but I don't see anything informative in it. The only logging from my code that gets recorded is in the driver. </p>
  
  <p>What am I doing wrong? (Besides relying on Hadoop in Windows....) My impression is that Hadoop is doing <em>something</em>, but it's not actually working. </p>
  
  <p>In case it helps, here are my Hadoop configuration properties:</p>
  
  <p>mapred-site.xml:</p>
  
  <pre><code>&lt;property&gt;
   &lt;name&gt;mapreduce.framework.name&lt;/name&gt;
   &lt;value&gt;yarn&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.child.java.opts&lt;/name&gt;
    &lt;value&gt;-Djava.awt.headless=true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;yarn.app.mapreduce.am.command-opts&lt;/name&gt;
    &lt;value&gt;-Djava.awt.headless=true -Xmx256m&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;yarn.app.mapreduce.am.admin-command-opts&lt;/name&gt;
    &lt;value&gt;-Djava.awt.headless=true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.job.tracker&lt;/name&gt;
    &lt;value&gt;local&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.mapper.new-api&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.reducer.new-api&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapreduce.job.user.classpath.first&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.tasktracker.map.tasks.maximum&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
    &lt;description&gt;The maximum number of map tasks that will be run simultaneously by a task tracker.&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.tasktracker.reduce.tasks.maximum&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
    &lt;description&gt;The maximum number of reduce tasks that will be run simultaneously by a task tracker.&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.map.tasks&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
    &lt;description&gt;The default number of map tasks per job. Ignored when mapred.job.tracker is "local".&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.reduce.tasks&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
    &lt;description&gt;The default number of reduce tasks per job.&lt;/description&gt;
  &lt;/property&gt;
  &lt;!-- This right here is an AWFUL hack to work around an even more awful bug --&gt;
  &lt;property&gt;
    &lt;name&gt;mapreduce.output.fileoutputformat.outputdir&lt;/name&gt;
    &lt;value&gt;/out&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>hdfs-site.xml:</p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;dfs.replication&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.name.dir&lt;/name&gt;
    &lt;value&gt;file:/hadoop/data/dfs/namenode&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.data.dir&lt;/name&gt;
    &lt;value&gt;file:/hadoop/data/dfs/datanode&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;
    &lt;value&gt;file:/hadoop/data/dfs/namenode&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;
    &lt;value&gt;file:/hadoop/data/dfs/datanode&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>yarn-site.xml:</p>
  
  <pre><code>&lt;property&gt;
   &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
   &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
   &lt;name&gt;yarn.nodemanager.aux-services.mapreduce.shuffle.class&lt;/name&gt;
   &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt;
    &lt;value&gt;127.0.0.1:8032&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;yarn.resourcemanager.scheduler.address&lt;/name&gt;
    &lt;value&gt;127.0.0.1:8030&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;yarn.resourcemanager.resource-tracker.address&lt;/name&gt;
    &lt;value&gt;127.0.0.1:8031&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;description&gt;CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries&lt;/description&gt;
    &lt;name&gt;yarn.application.classpath&lt;/name&gt;
    &lt;value&gt;
        %HADOOP_HOME%\etc\hadoop,
        %HADOOP_HOME%\share\hadoop\common\*,
        %HADOOP_HOME%\share\hadoop\common\lib\*,
        %HADOOP_HOME%\share\hadoop\hdfs\*,
        %HADOOP_HOME%\share\hadoop\hdfs\lib\*,
        %HADOOP_HOME%\share\hadoop\mapreduce\*,
        %HADOOP_HOME%\share\hadoop\mapreduce\lib\*,
        %HADOOP_HOME%\share\hadoop\yarn\*,
        %HADOOP_HOME%\share\hadoop\yarn\lib\*
    &lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>core-site.xml:</p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;fs.defaultFS&lt;/name&gt;
    &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>I have not been successful in finding anything online that helps. I've scoured StackOverflow, but haven't seen anything resembling my problem that I haven't already done. Any help would be great!</p>
  
  <p>EDIT: I should note that my Mapper/Reducer run correctly in MRUnit. </p>