title: API: How can I remotely check if two data sets are identical, without sending all data across the API?
body: <p>I'm building an API to allow remote systems to synchronise <strong>millions</strong> of rows in a data table (one-way).  The systems run different databases at each end (my receiving system is SQL Server but the sending system is SAP/Hana) so they can only communicate via an API.</p>
  
  <p>Using only methods on an API, how can I determine the differences in data rows between the two systems and ensure they are always in sync? </p>
  
  <p>One idea I had is to provide a list of IDs (primary keys), followed by some kind of "check" value or LastUpdatedDate which the remote systems can then use to find out which records they need to add or update. However if there are 10 million rows+, is there any way to avoid sending all 10 million keys each time I want to synchronise the databases?  </p>
  
  <p>Are there any protocols I can copy ideas from which do a similar thing?  I know IMAP for example, uses an approach similar to that above, but is poor at handling "millions" of emails. </p>
  
  <p><strong>EDIT</strong> (additional info):</p>
  
  <ul>
  <li>No records can be deleted on the source system - so I don't need to handle that.</li>
  <li>There will be 20 million new rows each year, up to a max of circa 150 million.</li>
  <li>Old records can occasionally be updated and need to trigger a re-sync.</li>
  </ul>