Please read the topic category description 8 to understand what this is all about


Description
Twitter classifies trending tweets according to predefined set of topics like “Data science”, “Hip hop”, “Sport” etc. However, their algorithm appears to often get confused by certain keywords in the tweet or the content of the image (see here 14 for some funny examples).
The goal of this project is to explore whether it’s possible to create a better topic extractor, or at least one that is more targeted for a smaller set of domains. There’s several ways to approach to task:

Frame it as a multiclass classification problem
Frame it as an unsupervised clustering problem, and combine this with techniques like UMAP 3 and/or HDBSCAN


Model(s)
Since Tweets are short in length, picking one of the sentence-transformers models on the Hub 4 is likely a good place to start.

Datasets
There are various Twitter datasets on the Hub and here’s a few examples to start with:

tweet_eval 10
sentiment140 8
emotion 8


Challenges
If you take the unsupervised learning approach, be warned that there’s no “correct” answer and you will have to experiment with the dimensionality reduction / cluster algorithms to get meaningful clusters.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that either can automatically classify a Tweet according to a topic, or visualises the 2D projection of the embeddings and colours them by topic.
Don’t forget to push all your models and datasets to the Hub so others can build on them!


Additional resources

https://towardsdatascience.com/how-exactly-umap-works-13e3040e1668 2

https://huggingface.co/spaces/edugp/embedding-lenses 7 (useful for UMAP inspiration)


Discord channel
To chat and organise with other people interested in this project, head over to our Discord 1 and:


Follow the instructions on the #join-course channel


Join the #twitter-topic-extractor channel


Just make sure you comment here to indicate that you’ll be contributing to this project