Project Description
The aim of the project is to improve online safety for vulnerable populations (e.g. adolescents). We would like to train a NLP and CV model to detect hate speech and memes, toxic comments, and potentially identify the presence of online predators as users engage in conversations or browse through a website.
Language
The model will be trained in English.
Model
We will finetune pre-trained GPT2 and ViT models for text and image classification respectively, but are also open to testing other models like ELECTRA and RoBERTa.
Datasets
Possible links to publicly available datasets include:

https://hatespeechdata.com/ 4
Toxic Comment Classification Challenge | Kaggle 3
Hateful Memes Challenge and dataset 2

Training scripts
We can start with existing Flax scripts for sequence classification (transformers/run_flax_glue.py at master · huggingface/transformers · GitHub 1) but will likely have to create our own for ViT.
(Optional) Challenges
One challenge will be to appropriately identify the intent of a given statement within the broader content of the visited site. For example, the inclusion of profanities can massively impact the evaluation of a sentence even if the overall sentence is meant to convey a positive sentiment.
A second challenge will be the ability of the model to correctly identify forms of toxicity that it has yet to encounter, and also circumvent biases (e.g. ethnicity) that could arise as a result of the data.
(Optional) Desired project outcome
We would like to have a demo that can run in near real-time. Future work can similarly adopt a multimodal approach to tackle videos and other forms of media.
(Optional) Reads
The following links can be useful to better understand the project and what has previously been done.

Detecting predatory conversations in social media by deep Convolutional Neural Networks: (PDF) Detecting predatory conversations in social media by deep Convolutional Neural Networks 1

Contextualizing Hate Speech Classifiers with Post-hoc Explanation: [2005.02439] Contextualizing Hate Speech Classifiers with Post-hoc Explanation 1

Detecting Online Hate Speech Using Context Aware Models: https://arxiv.org/pdf/1710.07395.pdf


Team
@diffindiff 2
@krecceg 1
@draziert