title: Accessing Maxmind Geo API in Hadoop using Distributed Cache
body: <p>I am writing a MapReduce job to analyze web logs. My code is intended to map ip addresses to geo locations and I am using Maxmind Geo API(<a href="https://github.com/maxmind/geoip-api-java" rel="nofollow">https://github.com/maxmind/geoip-api-java</a>) for that purpose. My code has a LookupService method that needs database file with ip to location matchings. I am trying to pass this database file using distributed cache. I tried doing this in 2 different ways</p>
  
  <p><strong>Case1:</strong></p>
  
  <p>Run the job passing the file from HDFS but it always throws an error saying "<strong>FILE NOT FOUND</strong>"</p>
  
  <pre><code>sudo -u hdfs hadoop jar \
   WebLogProcessing-0.0.1-SNAPSHOT-jar-with-dependencies.jar \
  GeoLocationDatasetDriver /user/hdfs/input /user/hdfs/out_put \
  /user/hdfs/GeoLiteCity.dat 
  </code></pre>
  
  <p>OR</p>
  
  <pre><code>sudo -u hdfs hadoop jar \
  WebLogProcessing-0.0.1-SNAPSHOT-jar-with-dependencies.jar \
  GeoLocationDatasetDriver /user/hdfs/input /user/hdfs/out_put \
  hdfs://sandbox.hortonworks.com:8020/user/hdfs/GeoLiteCity.dat
  </code></pre>
  
  <p>Driver Class Code:</p>
  
  <pre><code>Configuration conf = getConf();
  Job job = Job.getInstance(conf);
  job.addCacheFile(new Path(args[2]).toUri()); 
  </code></pre>
  
  <p>Mapper Class Code:</p>
  
  <pre><code>public void setup(Context context) throws IOException
  {
  URI[] uriList = context.getCacheFiles();
  Path database_path = new Path(uriList[0].toString());
  LookupService cl = new LookupService(database_path.toString(),
              LookupService.GEOIP_MEMORY_CACHE | LookupService.GEOIP_CHECK_CACHE);
  }
  </code></pre>
  
  <p><strong>CASE 2:</strong>
  Run the code by the passing the file from local file system through the -files option. <strong>Error: Null Pointer exception</strong> in the line LookupService cl = new LookupService(database_path)</p>
  
  <pre><code>sudo -u hdfs hadoop jar  \
  WebLogProcessing-0.0.1-SNAPSHOT-jar-with-dependencies.jar \
  com.prithvi.mapreduce.logprocessing.ipgeo.GeoLocationDatasetDriver \
  -files /tmp/jobs/GeoLiteCity.dat /user/hdfs/input /user/hdfs/out_put \
  GeoLiteCity.dat
  </code></pre>
  
  <p>Driver Code:</p>
  
  <pre><code>Configuration conf = getConf();
  Job job = Job.getInstance(conf);
  String dbfile = args[2];
  conf.set("maxmind.geo.database.file", dbfile);
  </code></pre>
  
  <p>Mapper Code:</p>
  
  <pre><code>public void setup(Context context) throws IOException
  {
    Configuration conf = context.getConfiguration();
    String database_path = conf.get("maxmind.geo.database.file");
    LookupService cl = new LookupService(database_path,
              LookupService.GEOIP_MEMORY_CACHE | LookupService.GEOIP_CHECK_CACHE);
  }
  </code></pre>
  
  <p>I need this database file in all my task trackers to accomplish the job. Can any one please suggest me the right way to do so?</p>