Launch HN: Datafold (YC S20) – Diff Tool for SQL Databases
I see a lot of these things and I don&#x27;t understand them. I&#x27;ve done too much ETL so I&#x27;m not naive. Now either 1) people are making a mountain out of a molehill (not saying that&#x27;s happening here, but in other cases I think so) 2) there&#x27;s something my experience of ETL hasn&#x27;t taught me or 3) these tools are specialised for niches. This one talks about &#x27;large datasets&#x27; but I don&#x27;t know how large that is.<p>Some questions then<p>&gt; Often it’s important to see how data changes between every iteration, and particularly useful if you have 1M+ rows and 100+ columns where “SELECT *” becomes useless.<p>select is fine for diffing. You just do an either-way except , something like<p><pre><code>  (
  select f1, f2, f3 ... f100
  from t1
  except
  select f1, f2, f3 ... f100
  from t2
  )
  union 
  (
  select f1, f2, f3 ... f100
  from t2
  except
  select f1, f2, f3 ... f100
  from t1
  )
</code></pre>
used this and it&#x27;s fine on many rows (millions is fine but I do recommend and index and a DB with a halfway decent optimiser).<p>&gt; (2)<p>Interesting. OK.<p>&gt; (3) Data transfer validation: moving large volumes of data between databases is error-prone<p>Really? I never had a problem. What is &#x27;large&#x27;? what problems have you seen? There are easy solutions with checksums, error correction (comes free with networks) or round-tripping, is that a problem?<p>Edit, just done that with mssql tables, 8 cols, 38 bytes per row, ~776,000 rows (identical but for one row), diff as above takes 2 seconds without an index (with PK it takes 5 seconds. Sigh. Well done MS). The single row discrepancy shows up fine.  Totally trivial to extend it to 100 columns (did that too in previous job).