add Aleph-Alpha-GermanWeb-Quality-Classifier-fastText model

README.md

@@ -0,0 +1,31 @@
+---
+language:
+- de
+library_name: fasttext
+---
+# Aleph-Alpha-GermanWeb-Quality-Classifier-fastText
+
+## Example Snippet
+
+```python
+import fasttext
+from huggingface_hub import hf_hub_download
+
+
+model_path = hf_hub_download(repo_id="Aleph-Alpha/Aleph-Alpha-Quality-Classifier-fastText", filename="model.bin")
+model = fasttext.load_model(model_path)
+
+text = "Das ist ein Beispieltext, um die Qualität zu überprüfen."
+
+pre_processed_document = text.replace("\n", " ")
+
+predicted_class, prob = model.predict(pre_processed_document)
+predicted_label = predicted_class[0].replace("__label__", "")
+document_score = prob[0]
+# similar to https://github.com/NVIDIA/NeMo-Curator/blob/31c8171434205e62f6a7d38565ffd9cb4c2806b7/nemo_curator/filters/classifier_filter.py#L47 , the document score is defined as the probability of the predicted class is the predicted label is 'high quality', otherwise it is 1 - document_score
+
+if predicted_label != "high_quality":
+    document_score = 1 - document_score
+
+print(predicted_label, document_score)
+```

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5983edf3f6988b199654a6f39489b72df65800eabe135986ed35cd24f758523f
+size 2471945221