title: Faster way to process 1.2 million JSON geolocation queries from large dataframe
body: <p>I am working on the <a href="https://snap.stanford.edu/data/loc-gowalla.html" rel="nofollow noreferrer">Gowalla location-based checkin dataset</a> which has around 6.44 million checkins. Unique locations on these checkins are 1.28 million. But Gowalla only gives latitudes and longitudes. So I need to find city, state and country for each of those lats and longs. From another post on StackOverflow I was able to create the R query below which queries the open street maps and finds the relevant geographical details. </p>
  
  <p>Unfortunately it takes around 1 minute to process 125 rows, which means 1.28 million rows would take a couple of days. Is there a faster way to find these details? Maybe there is some package with builtin lats and longs of cities of the world to find the city name for the given lat and long so I do not have to do online querying.</p>
  
  <p>Venue table is a data frame with 3 columns: <code>1: vid(venueId), 2 lat(latitude), 3: long(longitude)</code></p>
  
  <pre><code>for(i in 1:nrow(venueTable)) {
   #this is just an indicator to display current value of i on screen
   cat(paste(".",i,".")) 
  
   #Below code composes the url query 
   url &lt;- paste("http://nominatim.openstreetmap.org/reverse.php? format=json&amp;lat=",
                venueTableTest3$lat[i],"&amp;lon=",venueTableTest3$long[i])
   url &lt;- gsub(' ','',url)
   url &lt;- paste(url)
   x &lt;- fromJSON(url)
   venueTableTest3$display_name[i] &lt;- x$display_name
   venueTableTest3$country[i] &lt;- x$address$country
  }
  </code></pre>
  
  <p>I am using the <code>jsonlite</code> package in R which makes <code>x</code> which is the result of the JSON query as a dataframe which stores various results returned. So using <code>x$display_name</code> or <code>x$address$city</code> i use my required field. </p>
  
  <p>My laptop is core i5 3230M with 8gb ram and 120gb SSD using Windows 8.</p>