Llm, Machine Learning, Fine Tuning.
N-gram overlap works by seeing if any n-number of words repeat in both datasets. When running, they found a few false-positive cases, causing them to turn to embedding and syntax-based similarity. Embedding similarity requires using the same embedding agent on both data sets and then comparing the