Update README.md

README.md

@@ -8,13 +8,13 @@ license: apache-2.0
 
 This project provides a high-performance script for classifying the quality of Polish texts using a pre-trained XGBoost model. The classifier assigns one of three quality categories (`LOW`, `MEDIUM`, `HIGH`) to each text and provides a confidence score (probability).
 
-The classification is based on over 200 linguistic features extracted from each text, such as the count of nouns and verbs, NER entities, sentence length statistics, and the number of out-of-vocabulary words. These features are calculated by a companion module (`predictor.py`).
+The classification is based on over 200 linguistic features extracted from each text, such as the count of nouns and verbs, NER entities, sentence length statistics, and the number of out-of-vocabulary words. These features are calculated by a companion modules located in "features" folder.
 
-The script is designed for efficient, large-scale data processing. It leverages Python's `multiprocessing` to parallelize computations across all available CPU cores, significantly speeding up the analysis of large datasets. It supports processing files in both **Parquet** and **JSONL** formats.
+The script supports processing files in both **Parquet** and **JSONL** formats.
 
 ## 2. Features
 
-- **High-Performance Processing**: Utilizes `multiprocessing` to process texts in parallel, maximizing CPU usage.
+- **Efficient Batch Processing**: Processes all texts from a file at once, minimizing I/O and leveraging vectorized computations for high performance.
 - **Dual Format Support**: Ingests data from either `.parquet` or `.jsonl` files.
 - **Robust Feature Extraction**: Relies on a sophisticated feature engineering module (`predictor.py`) to generate over 200 linguistic metrics for accurate classification.
 - **Scalable**: Capable of handling millions of documents by processing files sequentially and texts in parallel.
@@ -33,9 +33,9 @@ The script follows a simple yet powerful workflow:
     -   For **JSONL files**, it streams the file line by line, parsing each line as a separate JSON object.
 4.  **Parallel Processing**: The core task of text analysis is distributed across a pool of worker processes.
     -   A list of texts is extracted from the input file.
-    -   Each text is sent to a worker process.
-    -   The worker process uses the `process_text_file` function to compute the 200+ linguistic features.
-    -   These features are then scaled and fed into the XGBoost model to predict the quality category and confidence score.
+    -   The `text` column is extracted into a list, forming a complete batch.
+    -   This batch of texts is passed to the `predict_batch` function.
+    -   Inside the function, the `TextAnalyzer` calculates features for all texts. This step may itself use mini-batches for memory efficiency.
 5.  **Output Generation**:
     -   The results (category and confidence) are collected from all worker processes.
     -   The script appends two new fields to each original record:
@@ -82,11 +82,9 @@ Ensure your project follows this structure:
 ```
 .
 ├── input_parquet/
-│   └── docs1.parquet
-│   └── docs2.parquet
+│   └── test.parquet
 ├── input_jsonl/
-│   └── data1.jsonl
-│   └── data2.jsonl
+│   └── test.jsonl
 ├── models/
 │   ├── model.joblib    # The trained XGBoost model
 │   └── scaler.pkl      # The scikit-learn scaler
@@ -94,12 +92,24 @@ Ensure your project follows this structure:
 ├── dummy.py            # The interactive testing script
 ├── main_jsonl.py       # The main processing script jsonl
 ├── main_parquet.py     # The main processing script parquet
-└── predictor.py        # The feature extraction module
+└── text_analyzer/      # The feature extraction module
+    ├── __init__.py
+    ├── analyzer.py
+    ├── utils.py
+    ├── constants.py
+    └── features/
+        ├── base_features.py
+        ├── linguistic_features.py
+        ├── regex_features.py
+        ├── spacy_features.py
+        └── structural_features.py
+
+  
 ```
 
 ## 5. Usage
 
-The script is configured to run out-of-the-box. Simply place your data files in the appropriate input directory and run the script.
+The script is configured to run out-of-the-box. Simply place your data files in the input directory and execute the main script.
 
 ### Step 1: Place Your Data Files
 
@@ -129,30 +139,21 @@ The script automatically skips files that have already been processed and exist
 ```
 
 python -W ignore main_parquet.py
-Znaleziono 1 plików w folderze wejściowym
-input_parquet\docs.parquet
-
-Podsumowanie:
-- Plików do przetworzenia: 1
-- Plików już przetworzonych: 0
-
-Rozpoczynanie przetwarzania 1 plików...
-Dodawanie do kolejki: input_parquet\docs.parquet
-
+Analiza cech: 100%|███████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 12.47it/s]
                                                 text quality_ai  confidence
-0  Pierwszy kamienny kościół w stylu romańskim po...       HIGH   99.98
-1  FJM.B.ZP \n cykl kształcenia 2019-2024\nKARTA ...        LOW   99.98
-2  Sztuka romańska (styl romański, romanizm, roma...       HIGH   99.89
-3  Przypisy\n Jerzy Z. Łoziński: Pomniki sztuki w...        LOW   92.28
-4  Na temat historii sakramentarza w wiekach XII–...       HIGH   97.99
-5  Przednia okładka\nPrzednia okładka\n \nMiniatu...        LOW   87.70
-6  Uchwała Nr 19\nZarządu Związku Rzemiosła Polsk...     MEDIUM   48.32
-7  Alternatywy 4 to jeden z najważniejszych i naj...       HIGH   99.98
-8  Akslop, może to jakieś duńskie miasto\njestem ...       HIGH   74.70
-9  Bielik - orzeł, czy nie orzeł?\nBielik, birkut...       HIGH   99.89
-(dataframe for parquets)
-
-Processing time: 47.61986541748047 seconds
+0  Pierwszy kamienny kościół w stylu romańskim po...       HIGH       99.97
+1  FJM.B.ZP \n cykl kształcenia 2019-2024\nKARTA ...        LOW       99.97
+2  Sztuka romańska (styl romański, romanizm, roma...       HIGH       99.92
+3  Przypisy\n Jerzy Z. Łoziński: Pomniki sztuki w...        LOW       92.54
+4  Na temat historii sakramentarza w wiekach XII–...       HIGH       96.03
+5  Przednia okładka\nPrzednia okładka\n \nMiniatu...        LOW       92.64
+6  Uchwała Nr 19\nZarządu Związku Rzemiosła Polsk...     MEDIUM       62.49
+7  Alternatywy 4 to jeden z najważniejszych i naj...       HIGH       99.98
+8  Akslop, może to jakieś duńskie miasto\njestem ...       HIGH       73.60
+9  Bielik - orzeł, czy nie orzeł?\nBielik, birkut...       HIGH       99.92
+Pomyślnie zapisano przetworzone dane do pliku output\test.parquet
+Processing time: 0.8603 seconds
+
 Wszystkie pliki zostały przetworzone!
 ```