Improved models

README.md

@@ -211,8 +211,8 @@ graph LR
 The following pre-trained models are available for use:
 | Dataset | Vectorizer | Classifier | Features | Accuracy on test | Accuracy on self | Model |
 | --- | --- | --- | --- | --- | --- | --- |
-| `imdb50k` | `tfidf` | `LinearRegression` | 20 000 | 59.59% ± 0.73% | 74.08% ± 0.14% | [Here](models/imdb50k_tfidf_ft20000.pkl) |
-| `sentiment140` | `tfidf` | `LinearRegression` | 20 000 | 52.81% ± 0.03% | ❌ | [Here](models/sentiment140_tfidf_ft20000.pkl) |
+| `imdb50k` | `tfidf` | `LinearRegression` | 20 000 | 83.24% ± 0.99% | 89.24% ± 0.13% | [Here](models/imdb50k_tfidf_ft20000.pkl) |
+| `sentiment140` | `tfidf` | `LinearRegression` | 20 000 | 83.24% ± 0.99% | 77.32% ± 0.28% | [Here](models/sentiment140_tfidf_ft20000.pkl) |
 | `amazonreviews` | `tfidf` | `LinearRegression` | 20 000 | ❌ | ❌ | [Here](models/amazonreviews_tfidf_ft1048576.pkl) |
 
 

app/model.py

@@ -36,7 +36,6 @@ def _identity(x: list[str]) -> list[str]:
 def _get_vectorizer(
     name: Literal["tfidf", "count", "hashing"],
     n_features: int,
-    df: tuple[float, float] = (1.0, 1.0),
     ngram: tuple[int, int] = (1, 2),
 ) -> TransformerMixin:
     """Get the appropriate vectorizer.
@@ -44,7 +43,6 @@ def _get_vectorizer(
     Args:
         name: Type of vectorizer
         n_features: Maximum number of features
-        df: Document frequency range [min_df, max_df] (ignored for HashingVectorizer)
         ngram: N-gram range [min_n, max_n]
 
     Returns:
@@ -66,15 +64,11 @@ def _get_vectorizer(
         case "tfidf":
             return TfidfVectorizer(
                 max_features=n_features,
-                min_df=df[0],
-                max_df=df[1],
                 **shared_params,
             )
         case "count":
             return CountVectorizer(
                 max_features=n_features,
-                min_df=df[0],
-                max_df=df[1],
                 **shared_params,
             )
         case "hashing":

models/imdb50k_tfidf_ft20000.pkl

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f06582370d3fe31c878c0feec254d2a3fcd7f6f6140de10d1bd11e528853d66e
-size 3288
+oid sha256:0c916d380fc84a33f3cb5892cd10e4aaa29330cbbac4243860e91fe9392df897
+size 398706

models/sentiment140_tfidf_ft20000.pkl

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ef80fcab97f510984f6f75f4579b817b9d2d361f68fc8dd24a3e3e9957fccfcf
-size 1230
+oid sha256:1308cb96bbee2befeb585c99fb3ad78b4bbef0504fcb5070d8c738289c212431
+size 397501