I'm building a event detector for Twitter and it's being extremely affected by spam tweets, so I'm planning to filter tweets a little bit with a text similarity algorithm.
The approach I have thinked about is to have a set of tweets where I will store the different tweets. First of all I will clear links and mentions from the tweets, and check if the tweet I'm processing has a similarity value with any of the tweets of the set greater than a threshold (0.7-0.8 for example). If that's the case, I will continue the iteration and ignore that tweet; otherwise I will add that tweet to the set and work with it.
I have been reading different answers to related questions but they were only for small corpuses of text, while this will work with a dataset of at least 15.000 tweets more or less, so the algorithm will be comparing between every tweet and the set of tweets 15.000 times.
Also other questions are a bit old, and new algorithms may have been created or better implementations of older ones may have appeared.
In conclusion, what do you think is the best way to afront this spam problem? Would it be a Python native one or an extern one?
to find similarity you can use tf-idf vectors and then calculate cosine similarity between them, but it's a large number of vectors to compare so you can cluster your data and find a center vector for each cluster, so you just need to compare your new tweet with center vectors not all of them.
Get similarity between text and the texts of a set in Python