title: Splunk to avoid duplication of data pulled by REST API
body: <p>I have splunk instance where i configure Data Inputs as "REST API input for polling data from RESTful endpoints".
  I have almost around 20+ endpoints and where i am pulling data in json format and loading in single index.</p>
  
  <p>However each time any reports or search query runs it will double same data again like very first fetch brings 5 values and subsequent fetch will bring another 5 and so and keep increasing.</p>
  
  <p>Now in my dashboards and reports i kind of landed into problem of duplicate data. How i should avoid it.
  So for very unusual work around i increased interval from 1 min to 1 months which helps me to avoid data duplication.
  However i cannot have stale data for month...i can still survive with 1 day interval but not with 1 month.</p>
  
  <p><strong>Is there any way in splunk where i can keep my REST API Call tidy(avoid duplicates) ... to make my dashboards and  reports on the fly.</strong></p>
  
  <p>Here is snippet of my inputs.conf file for REST API.</p>
  
  <pre><code>[rest://rst_sl_get_version]
  auth_password = ccccc
  auth_type = basic
  auth_user = vvvvvvv
  endpoint = https://api.xx.com/rest/v3/xx_version
  host = slrestdata
  http_method = GET
  index = sldata
  index_error_response_codes = 0
  response_type = json
  sequential_mode = 0
  sourcetype = _json
  streaming_request = 0
  polling_interval = 2592000
  </code></pre>