title: Pig's AvroStorage LOAD removes unicode chars from input
body: <p>I am using pig to read avro files and normalize/transform the data before writing back out. The avro files have records of the form:</p>
  
  <pre><code>{
    "type" : "record",
    "name" : "KeyValuePair",
    "namespace" : "org.apache.avro.mapreduce",
    "doc" : "A key/value pair",
    "fields" : [ {
      "name" : "key",
      "type" : "string",
      "doc" : "The key"
    }, {
      "name" : "value",
      "type" : {
        "type" : "map",
        "values" : "bytes"
      },
      "doc" : "The value"
    } ]
  }
  </code></pre>
  
  <p>I have used the AvroTools <a href="http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/" rel="nofollow noreferrer">command-line utility</a> in conjunction with <a href="https://stedolan.github.io/jq/" rel="nofollow noreferrer">jq</a> to dump the first record to JSON:</p>
  
  <pre><code>    $ java -jar avro-tools-1.8.1.jar tojson part-m-00000.avro | ./jq --compact-output 'select(.value.pf_v != null)' | head -n 1 | ./jq .
  {
    "key": "some-record-uuid",
    "value": {
      "pf_v": "v1\u0003Basic\u0001slcvdr1rw\u001a\u0004v2\u0003DayWatch\u0001slcva2omi\u001a\u0004v3\u0003Performance\u0001slc1vs1v1w1p1g1i\u0004v4\u0003Fundamentals\u0001snlj1erwi\u001a\u0004v5\u0003My Portfolio\u0001svr1dews1b2b3k1k2\u001a\u0004v0\u00035"
    }
  }
  </code></pre>
  
  <p>I run the following pig commands:</p>
  
  <pre><code>REGISTER avro-1.8.1.jar
  REGISTER json-simple-1.1.1.jar
  REGISTER piggybank-0.15.0.jar
  REGISTER jackson-core-2.8.6.jar
  REGISTER jackson-databind-2.8.6.jar
  
  DEFINE AvroLoader org.apache.pig.piggybank.storage.avro.AvroStorage();
  AllRecords = LOAD 'part-m-00000.avro'
      USING AvroLoader()
      AS (key: chararray, value: map[]);
  
  Records = FILTER AllRecords BY value#'pf_v' is not null;
  
  SmallRecords = LIMIT Records 10;
  DUMP SmallRecords;
  </code></pre>
  
  <p>The corresponding record for the last command above is as follows:</p>
  
  <pre><code>...
  (some-record-uuid,[pf_v#v03v1Basicslcviv2DayWatchslcva2omiv3Performanceslc1vs1v1w1p1g1i])
  ...
  </code></pre>
  
  <p>As you can see the unicode chars have been removed from the <code>pf_v</code> value. The unicode characters are actually being used as delimiters in these values so I will need them in order to fully parse the records into their desired normalized state. The unicode characters are clearly present in the encoded <code>.avro</code> file (as demonstrated by dumping the file to JSON). Is anybody aware of a way to get AvroStorage to <strong>not</strong> remove the unicode chars when loading records?</p>
  
  <p>Thank you!</p>
  
  <p><strong>Update:</strong>
  I have also performed the <strong>same</strong> operation using Avro's python DataFileReader:</p>
  
  <pre><code>import avro.schema
  from avro.datafile import DataFileReader, DataFileWriter
  from avro.io import DatumReader, DatumWriter
  
  reader = DataFileReader(open("part-m-00000.avro", "rb"), DatumReader())
  
  for rec in reader:
      if 'some-record-uuid' in rec['key']:
          print rec
          print '--------------------------------------------'
          break
  
  reader.close()
  </code></pre>
  
  <p>This prints a dict with what looks like hex chars substituted for the unicode chars (which is preferable to removing them entirely):</p>
  
  <pre><code>{u'value': {u'pf_v': 'v0\x033\x04v1\x03Basic\x01slcvi\x1a\x04v2\x03DayWatch\x01slcva2omi\x1a\x04v3\x03Performance\x01slc1vs1v1w1p1g1i\x1a'}, u'key': u'some-record-uuid'}
  </code></pre>