title: Hadoop ArrayWritable giving me a ClassCastException
body: <p><strong>EDIT: PROBLEM SOLVED</strong> - I had a pretty silly error.</p>
  
  <p>I've got a MapReduce pipeline consisting of a map, reduce, map, and reduce.  I use SequenceFileOutputFormat for the first reduce, and SequenceFileInputFormat for the second map.   I've looked at usage for it, and it seems like I'm using it right.  The types that I'm putting into this are IntWritable and IntPairArrayWritable (a custom ArrayWritable subclass that uses IntPairWritable from mahout).  The problem is that when reading the IntPairArrayWritable in the second map, I get a ClassCastException when I try to get individual IntPairWritables out.  I'm not sure if this is due to an error in how I'm using the ArrayWritable class or if it's something wrong with my use of SequenceFile{Input,Output}Format.  I've looked at a bunch of examples here and elsewhere, and it looks to me like I'm doing both of them right, but I'm still getting an error.  Any help?</p>
  
  <p>The specifics:</p>
  
  <p>Here's my first reducer class:</p>
  
  <pre><code>public static class WalkIdReducer extends MapReduceBase implements
          Reducer&lt;IntWritable, IntPairWritable, IntWritable, IntPairArrayWritable&gt; {
  
      @Override
      public void reduce(IntWritable walk_id, Iterator&lt;IntPairWritable&gt; values,
              OutputCollector&lt;IntWritable, IntPairArrayWritable&gt; output,
              Reporter reporter) throws IOException {
          ArrayList&lt;IntPairWritable&gt; value_array = new ArrayList&lt;IntPairWritable&gt;();
          while (values.hasNext()) {
              value_array.add(values.next());
          }
          output.collect(walk_id, IntPairArrayWritable.fromArrayList(value_array));
      }
  }
  </code></pre>
  
  <p>And the second mapper class:</p>
  
  <pre><code>public static class NodePairMapper extends MapReduceBase implements
          Mapper&lt;IntWritable, IntPairArrayWritable, IntPairWritable, Text&gt; {
  
      @Override
      public void map(IntWritable key, IntPairArrayWritable value,
              OutputCollector&lt;IntPairWritable, Text&gt; output,
              Reporter reporter) throws IOException {
          // The following line gives a ClassCastException;
          // See IntPairArrayWritable.toArrayList(), below
          ArrayList&lt;IntPairWritable&gt; values = value.toArrayList();
          // other unimportant stuff
      }
  }
  </code></pre>
  
  <p>Relevant parts of the job configuration for the first MapReduce:</p>
  
  <pre><code>    conf.setReducerClass(WalkIdReducer.class);
      conf.setOutputKeyClass(IntWritable.class);
      conf.setOutputValueClass(IntPairArrayWritable.class);
      conf.setOutputFormat(SequenceFileOutputFormat.class);
  </code></pre>
  
  <p>And for the second MapReduce:</p>
  
  <pre><code>    conf.setInputFormat(SequenceFileInputFormat.class);
      conf.setMapperClass(NodePairMapper.class);
  </code></pre>
  
  <p>And, finally, my ArrayWritable subclass:</p>
  
  <pre><code>public static class IntPairArrayWritable extends ArrayWritable
  {
      // These two methods are what people say is all you need for
      // creating an ArrayWritable subclass
      public IntPairArrayWritable() {
          super(IntPairArrayWritable.class);
      }
  
      public IntPairArrayWritable(IntPairWritable[] values) {
          super(IntPairArrayWritable.class, values);
      }
  
      // Some convenience methods, so I can use ArrayLists in
      // other parts of the code
      public static IntPairArrayWritable fromArrayList(
              ArrayList&lt;IntPairWritable&gt; array) {
          IntPairArrayWritable writable = new IntPairArrayWritable();
          IntPairWritable[] values = new IntPairWritable[array.size()];
          for (int i=0; i&lt;array.size(); i++) {
              values[i] = array.get(i);
          }
          writable.set(values);
          return writable;
      }
  
      public ArrayList&lt;IntPairWritable&gt; toArrayList() {
          ArrayList&lt;IntPairWritable&gt; array = new ArrayList&lt;IntPairWritable&gt;();
          for (Writable pair : this.get()) {
              // This line is what kills it.  I get a ClassCastException here.
              IntPairWritable int_pair = (IntPairWritable) pair;
              array.add(int_pair);
          }
          return array;
      }
  }
  </code></pre>
  
  <p>The specific error I get is the following:</p>
  
  <pre><code>java.lang.ClassCastException: WalkAnalyzer$IntPairArrayWritable cannot be cast to org.apache.mahout.common.IntPairWritable
  at WalkAnalyzer$IntPairArrayWritable.toArrayList(WalkAnalyzer.java:231)
  at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:84)
  at WalkAnalyzer$NodePairMapper.map(WalkAnalyzer.java:77)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
  at org.apache.hadoop.mapred.Child.main(Child.java:170)
  </code></pre>
  
  <p>I am pretty baffled as to why what is coming out of the get() method from ArrayWritable is an instance of <code>WalkAnalyzer$IntPairArrayWritable</code> - I'm expecting get() to return an array of the elements contained by <code>IntPairArrayWritable</code>, as stated in the API.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>I found the problem.  It was in how I wrote the constructors for IntPairArrayWritable.  I called <code>super(IntPairArrayWritable.class);</code> when I should have called <code>super(IntPairWritable.class);</code>.  The code should actually look like this:</p>
  
  <pre><code>public static class IntPairArrayWritable extends ArrayWritable
  {
      // These two methods are what people say is all you need for
      // creating an ArrayWritable subclass
      public IntPairArrayWritable() {
          super(IntPairWritable.class);
      }
  
      public IntPairArrayWritable(IntPairWritable[] values) {
          super(IntPairWritable.class, values);
      }
  }
  </code></pre>
  
  <p>I suppose it would have been a good idea to use a less obviously confused name for the ArrayWritable subclass, so the error would have been easier to spot.</p>