Update README.md

README.md

@@ -1,6 +1,7 @@
 ---
 language:
 - en
+- kn
 tags:
 - language
 - kannada
@@ -8,6 +9,7 @@ license: mit
 base_model:
 - text2font/ByteLevelBPETokenizer_default
 ---
+
 # Kannada ByteLevel BPE Tokenizer
 
 This repository contains a **Byte-Level BPE (Byte Pair Encoding) tokenizer** for the **Kannada** language, designed using the Hugging Face `tokenizers` library. The tokenizer is optimized for handling Kannada text, including **pure Kannada, mixed-language text, numbers, punctuation, and special cases like URLs and emojis**.
@@ -34,223 +36,54 @@ The tokenizer has been tested on multiple text categories:
 
 ## Tokenizer Test Results
 
-
-
-Category: Pure Kannada
-
-
-Test Case 1: Basic sentence
-Original text: ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ
-Encoded tokens: ['<s>', 'ನಮಸ', 'à³į', 'à²ķ', 'ಾ', 'ರ', 'Ġà²ķನ', 'à³į', 'ನಡ', 'Ġà²Ń', 'ಾ', 'ಷ', 'à³Ĩ', '</s>']
-Token IDs: [0, 1461, 264, 278, 270, 272, 738, 264, 407, 386, 270, 323, 268, 1]
-Decoded text: ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ
-Analysis:
-- Number of tokens: 14
-- Average token length: 1.29 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 2: Complex sentence
-Original text: ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ
-Encoded tokens: ['<s>', 'à²ķನ', 'à³į', 'ನಡ', 'Ġನ', 'ಾ', 'ಡ', 'ಿ', 'ನ', 'Ġಸ', 'à²Ĥ', 'ಸ', 'à³į', 'à²ķ', 'à³ĥ', 'ತ', 'ಿ', 'Ġಮತ', 'à³į', 'ತ', 'à³ģ', 'Ġಪರ', 'à²Ĥ', 'ಪರ', 'à³Ĩ', '</s>']
-Token IDs: [0, 754, 264, 407, 298, 270, 280, 267, 266, 300, 275, 281, 264, 278, 412, 271, 267, 382, 264, 271, 265, 360, 275, 524, 268, 1]
-Decoded text: ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ
-Analysis:
-- Number of tokens: 26
-- Average token length: 1.27 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-
-Category: Mixed Language
-
-
-Test Case 1: Kannada with English
-Original text: ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ
-Encoded tokens: ['<s>', 'ನನ', 'à³į', 'ನ', 'Ġ', 'e', 'm', 'a', 'i', 'l', 'Ġ', 'I', 'D', 'Ġà²ĩದ', 'à³ģ', 'Ġ', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '@', 'e', 'm', 'a', 'i', 'l', '.', 'com', 'Ġà²Ĩà²Ĺ', 'ಿ', 'ದ', 'à³Ĩ', '</s>']
-Token IDs: [0, 306, 264, 266, 225, 73, 81, 69, 77, 80, 225, 45, 40, 493, 265, 225, 73, 92, 69, 81, 84, 80, 73, 36, 73, 81, 69, 77, 80, 18, 469, 408, 267, 269, 268, 1]
-Decoded text: ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ
-Analysis:
-- Number of tokens: 36
-- Average token length: 1.14 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 2: Technical terms
-Original text: ಈ mobile phone ನಲ್ಲಿ 4G network ಇದೆ
-Encoded tokens: ['<s>', 'à²Ī', 'Ġ', 'm', 'o', 'b', 'i', 'l', 'e', 'Ġ', 'p', 'h', 'o', 'n', 'e', 'Ġನಲ', 'à³į', 'ಲ', 'ಿ', 'Ġ', '4', 'G', 'Ġ', 'n', 'e', 't', 'w', 'o', 'r', 'k', 'Ġà²ĩದ', 'à³Ĩ', '</s>']
-Token IDs: [0, 725, 225, 81, 83, 70, 77, 80, 73, 225, 84, 76, 83, 82, 73, 684, 264, 274, 267, 225, 24, 43, 225, 82, 73, 88, 91, 83, 86, 79, 493, 268, 1]
-Decoded text: ಈ mobile phone ನಲ್ಲಿ 4G network ಇದೆ
-Analysis:
-- Number of tokens: 33
-- Average token length: 1.06 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 3: Social media
-Original text: ಅವರ Instagram profile ತುಂಬಾ popular ಆಗಿದೆ
-Encoded tokens: ['<s>', 'à²ħವರ', 'Ġ', 'I', 'n', 's', 't', 'a', 'g', 'r', 'a', 'm', 'Ġ', 'p', 'r', 'o', 'f', 'i', 'l', 'e', 'Ġತ', 'à³ģà²Ĥ', 'ಬ', 'ಾ', 'Ġ', 'p', 'o', 'p', 'u', 'l', 'a', 'r', 'Ġà²Ĩà²Ĺ', 'ಿ', 'ದ', 'à³Ĩ', '</s>']
-Token IDs: [0, 744, 225, 45, 82, 87, 88, 69, 75, 86, 69, 81, 225, 84, 86, 83, 74, 77, 80, 73, 299, 336, 293, 270, 225, 84, 83, 84, 89, 80, 69, 86, 408, 267, 269, 268, 1]
-Decoded text: ಅವರ Instagram profile ತುಂಬಾ popular ಆಗಿದೆ
-Analysis:
-- Number of tokens: 37
-- Average token length: 1.11 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-
-Category: Numbers and Dates
-
-
-Test Case 1: Pure numbers
-Original text: ೧೨೩೪೫ 12345
-Encoded tokens: ['<s>', '೧', 'à³', '¨', 'à³', '©', 'à³', 'ª', 'à³', '«', 'Ġ', '1', '2', '3', '4', '5', '</s>']
-Token IDs: [0, 2891, 262, 106, 262, 107, 262, 108, 262, 109, 225, 21, 22, 23, 24, 25, 1]
-Decoded text: ೧೨೩೪೫ 12345
-Analysis:
-- Number of tokens: 17
-- Average token length: 0.65 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 2: Mixed numbers
-Original text: ನನ್ನ ಫೋನ್ ನಂಬರ್ +91 9876543210
-Encoded tokens: ['<s>', 'ನನ', 'à³į', 'ನ', 'Ġಫ', 'à³', 'ĭ', 'ನ', 'à³į', 'Ġನ', 'à²Ĥ', 'ಬರ', 'à³į', 'Ġ', '+', '9', '1', 'Ġ', '9', '8', '7', '6', '5', '4', '3', '2', '1', '0', '</s>']
-Token IDs: [0, 306, 264, 266, 1055, 262, 238, 266, 264, 298, 275, 505, 264, 225, 15, 29, 21, 225, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 1]
-Decoded text: ನನ್ನ ಫೋನ್ ನಂಬರ್ +91 9876543210
-Analysis:
-- Number of tokens: 29
-- Average token length: 1.03 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 3: Date formats
-Original text: ೨೦೨೪ ಫೆಬ್ರವರಿ ೧೪ / 14-02-2024
-Encoded tokens: ['<s>', 'à³', '¨', 'à³', '¦', 'à³', '¨', 'à³', 'ª', 'Ġಫ', 'à³Ĩ', 'ಬ', 'à³į', 'ರ', 'ವರ', 'ಿ', 'Ġ', '೧', 'à³', 'ª', 'Ġ', '/', 'Ġ', '1', '4', '-', '0', '2', '-', '2', '0', '2', '4', '</s>']
-Token IDs: [0, 262, 106, 262, 104, 262, 106, 262, 108, 1055, 268, 293, 264, 272, 328, 267, 225, 2891, 262, 108, 225, 19, 225, 21, 24, 17, 20, 22, 17, 22, 20, 22, 24, 1]
-Decoded text: ೨೦೨೪ ಫೆಬ್ರವರಿ ೧೪ / 14-02-2024
-Analysis:
-- Number of tokens: 34
-- Average token length: 0.85 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 4: Currency
-Original text: ₹೧೦೦೦.೫೦ ರೂಪಾಯಿ / Rs. 1000.50
-Encoded tokens: ['<s>', 'â', 'Ĥ', '¹', '೧', 'à³', '¦', 'à³', '¦', 'à³', '¦', '.', 'à³', '«', 'à³', '¦', 'Ġರ', 'à³Ĥ', 'ಪ', 'ಾ', 'ಯ', 'ಿ', 'Ġ', '/', 'Ġ', 'R', 's', '.', 'Ġ', '1', '0', '0', '0', '.', '5', '0', '</s>']
-Token IDs: [0, 163, 229, 122, 2891, 262, 104, 262, 104, 262, 104, 18, 262, 109, 262, 104, 371, 289, 291, 270, 277, 267, 225, 19, 225, 54, 87, 18, 225, 21, 20, 20, 20, 18, 25, 20, 1]
-Decoded text: ₹೧೦೦೦.೫೦ ರೂಪಾಯಿ / Rs. 1000.50
-Analysis:
-- Number of tokens: 37
-- Average token length: 0.78 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-
-Category: Punctuation
-
-
-Test Case 1: Basic punctuation
-Original text: ನೀವು ಹೇಗಿದ್ದೀರಿ? ನಾನು ಚೆನ್ನಾಗಿದ್ದೇನೆ!
-Encoded tokens: ['<s>', 'ನ', 'à³Ģ', 'ವ', 'à³ģ', 'Ġಹ', 'à³', 'ĩ', 'à²Ĺ', 'ಿ', 'ದ', 'à³į', 'ದ', 'à³Ģ', 'ರ', 'ಿ?', 'Ġನ', 'ಾ', 'ನ', 'à³ģ', 'Ġà²ļ', 'à³Ĩ', 'ನ', 'à³į', 'ನ', 'ಾ', 'à²Ĺ', 'ಿ', 'ದ', 'à³į', 'ದ', 'à³', 'ĩ', 'ನ', 'à³Ĩ!', '</s>']
-Token IDs: [0, 266, 276, 279, 265, 284, 262, 234, 273, 267, 269, 264, 269, 276, 272, 813, 298, 270, 266, 265, 339, 268, 266, 264, 266, 270, 273, 267, 269, 264, 269, 262, 234, 266, 532, 1]
-Decoded text: ನೀವು ಹೇಗಿದ್ದೀರಿ? ನಾನು ಚೆನ್ನಾಗಿದ್ದೇನೆ!
-Analysis:
-- Number of tokens: 36
-- Average token length: 1.03 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 2: Special characters
-Original text: ಇಂದು @bangalore #Karnataka (ಕರ್ನಾಟಕ) ರಾಜ್ಯ
-Encoded tokens: ['<s>', 'à²ĩ', 'à²Ĥ', 'ದ', 'à³ģ', 'Ġ', '@', 'b', 'an', 'g', 'a', 'l', 'o', 'r', 'e', 'Ġ', '#', 'K', 'a', 'r', 'n', 'a', 't', 'a', 'k', 'a', 'Ġ(', 'à²ķರ', 'à³į', 'ನ', 'ಾ', 'à²Łà²ķ', ')', 'Ġರ', 'ಾ', 'à²ľ', 'à³į', 'ಯ', '</s>']
-Token IDs: [0, 447, 275, 269, 265, 225, 36, 70, 380, 75, 69, 80, 83, 86, 73, 225, 7, 47, 69, 86, 82, 69, 88, 69, 79, 69, 443, 368, 264, 266, 270, 482, 13, 371, 270, 330, 264, 277, 1]
-Decoded text: ಇಂದು @bangalore #Karnataka (ಕರ್ನಾಟಕ) ರಾಜ್ಯ
-Analysis:
-- Number of tokens: 39
-- Average token length: 1.08 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 3: Mixed punctuation
-Original text: "ಕನ್ನಡ" - 'ನಾಡಿನ' & ಸಂಸ್ಕೃತಿ...
-Encoded tokens: ['<s>', '"', 'à²ķನ', 'à³į', 'ನಡ', '"', 'Ġ-', 'Ġ', "'", 'ನ', 'ಾ', 'ಡ', 'ಿ', 'ನ', "'", 'Ġ', '&', 'Ġಸ', 'à²Ĥ', 'ಸ', 'à³į', 'à²ķ', 'à³ĥ', 'ತ', 'ಿ.', '.', '.', '</s>']
-Token IDs: [0, 6, 754, 264, 407, 6, 438, 225, 11, 266, 270, 280, 267, 266, 11, 225, 10, 300, 275, 281, 264, 278, 412, 271, 517, 18, 18, 1]
-Decoded text: "ಕನ್ನಡ" - 'ನಾಡಿನ' & ಸಂಸ್ಕೃತಿ...
-Analysis:
-- Number of tokens: 28
-- Average token length: 1.11 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-
-Category: Special Cases
-
-
-Test Case 1: URLs
-Original text: ನಮ್ಮ ವೆಬ್‌ಸೈಟ್: https://kannada.example.com
-Encoded tokens: ['<s>', 'ನಮ', 'à³į', 'ಮ', 'Ġವ', 'à³Ĩ', 'ಬ', 'à³į', 'âĢ', 'Į', 'ಸ', 'à³Ī', 'à²Ł', 'à³į:', 'Ġhttps', '://', 'kannad', 'a', '.', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '.', 'com', '</s>']
-Token IDs: [0, 765, 264, 285, 332, 268, 293, 264, 297, 239, 281, 335, 286, 3786, 473, 452, 471, 69, 18, 73, 92, 69, 81, 84, 80, 73, 18, 469, 1]
-Decoded text: ನಮ್ಮ ವೆಬ್‌ಸೈಟ್: https://kannada.example.com
-Analysis:
-- Number of tokens: 29
-- Average token length: 1.48 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 2: Hashtags
-Original text: #ಕನ್ನಡ_ರಾಜ್ಯೋತ್ಸವ #Kannada
-Encoded tokens: ['<s>', '#', 'à²ķನ', 'à³į', 'ನಡ', '_', 'ರ', 'ಾ', 'à²ľ', 'à³į', 'ಯ', 'à³', 'ĭ', 'ತ', 'à³į', 'ಸವ', 'Ġ', '#', 'K', 'an', 'nad', 'a', '</s>']
-Token IDs: [0, 7, 754, 264, 407, 67, 272, 270, 330, 264, 277, 262, 238, 271, 264, 595, 225, 7, 47, 380, 461, 69, 1]
-Decoded text: #ಕನ್ನಡ_ರಾಜ್ಯೋತ್ಸವ #Kannada
-Analysis:
-- Number of tokens: 23
-- Average token length: 1.13 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 3: Emojis
-Original text: ಕನ್ನಡ ನಾಡು 🚩 ಕನ್ನಡ ಭಾಷೆ ❤️
-Encoded tokens: ['<s>', 'à²ķನ', 'à³į', 'ನಡ', 'Ġನ', 'ಾ', 'ಡ', 'à³ģ', 'Ġ', 'ð', 'Ł', 'ļ', '©', 'Ġà²ķನ', 'à³į', 'ನಡ', 'Ġà²Ń', 'ಾ', 'ಷ', 'à³Ĩ', 'Ġ', 'â', 'Ŀ', '¤', 'ï', '¸', 'ı', '</s>']
-Token IDs: [0, 754, 264, 407, 298, 270, 280, 265, 225, 177, 258, 253, 107, 738, 264, 407, 386, 270, 323, 268, 225, 163, 256, 102, 176, 121, 242, 1]
-Decoded text: ಕನ್ನಡ ನಾಡು 🚩 ಕನ್ನಡ ಭಾಷೆ ❤️
-Analysis:
-- Number of tokens: 28
-- Average token length: 0.93 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-Test Case 4: File paths
-Original text: C:\Users\ಬಳಕೆದಾರ\Documents\ಕನ್ನಡ.txt
-Encoded tokens: ['<s>', 'C', ':', '\\', 'U', 's', 'e', 'r', 's', '\\', 'ಬಳà²ķ', 'à³Ĩ', 'ದ', 'ಾ', 'ರ', '\\', 'D', 'o', 'c', 'u', 'm', 'e', 'n', 't', 's', '\\', 'à²ķನ', 'à³į', 'ನಡ', '.', 't', 'x', 't', '</s>']
-Token IDs: [0, 39, 30, 64, 57, 87, 73, 86, 87, 64, 4023, 268, 269, 270, 272, 64, 40, 83, 71, 89, 81, 73, 82, 88, 87, 64, 754, 264, 407, 18, 88, 92, 88, 1]
-Decoded text: C:\Users\ಬಳಕೆದಾರ\Documents\ಕನ್ನಡ.txt
-Analysis:
-- Number of tokens: 34
-- Average token length: 1.06 characters
-- Reconstruction: Perfect
-
-----------------------------------------
-
-
+### Category: Pure Kannada
+
+#### Test Case 1: Basic sentence
+**Original text:** ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ  
+**Encoded tokens:** ['<s>', 'ನಮಸ', 'à³į', 'à²ķ', 'ಾ', 'ರ', 'Ġà²ķನ', 'à³į', 'ನಡ', 'Ġà²Ń', 'ಾ', 'ಷ', 'à³Ĩ', '</s>']  
+**Token IDs:** [0, 1461, 264, 278, 270, 272, 738, 264, 407, 386, 270, 323, 268, 1]  
+**Decoded text:** ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ  
+**Analysis:**
+- Number of tokens: 14  
+- Average token length: 1.29 characters  
+- Reconstruction: Perfect  
+
+#### Test Case 2: Complex sentence
+**Original text:** ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ  
+**Encoded tokens:** ['<s>', 'à²ķನ', 'à³į', 'ನಡ', 'Ġನ', 'ಾ', 'ಡ', 'ಿ', 'ನ', 'Ġಸ', 'à²Ĥ', 'ಸ', 'à³į', 'à²ķ', 'à³ĥ', 'ತ', 'ಿ', 'Ġಮತ', 'à³į', 'ತ', 'à³ģ', 'Ġಪರ', 'à²Ĥ', 'ಪರ', 'à³Ĩ', '</s>']  
+**Token IDs:** [0, 754, 264, 407, 298, 270, 280, 267, 266, 300, 275, 281, 264, 278, 412, 271, 267, 382, 264, 271, 265, 360, 275, 524, 268, 1]  
+**Decoded text:** ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ  
+**Analysis:**
+- Number of tokens: 26  
+- Average token length: 1.27 characters  
+- Reconstruction: Perfect  
+
+### Category: Mixed Language
+
+#### Test Case 1: Kannada with English
+**Original text:** ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ  
+**Encoded tokens:** ['<s>', 'ನನ', 'à³į', 'ನ', 'Ġ', 'e', 'm', 'a', 'i', 'l', 'Ġ', 'I', 'D', 'Ġà²ĩದ', 'à³ģ', 'Ġ', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '@', 'e', 'm', 'a', 'i', 'l', '.', 'com', 'Ġà²Ĩà²Ĺ', 'ಿ', 'ದ', 'à³Ĩ', '</s>']  
+**Token IDs:** [0, 306, 264, 266, 225, 73, 81, 69, 77, 80, 225, 45, 40, 493, 265, 225, 73, 92, 69, 81, 84, 80, 73, 36, 73, 81, 69, 77, 80, 18, 469, 408, 267, 269, 268, 1]  
+**Decoded text:** ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ  
+**Analysis:**
+- Number of tokens: 36  
+- Average token length: 1.14 characters  
+- Reconstruction: Perfect  
+
+(Additional test cases can be added following the same format)
 
 ## Repository Structure
-The repository consists of tokenizer files, configuration files, and documentation.
+The repository consists of tokenizer files, configuration files, and documentation:
+```
+├── kannada_tokenizer/
+│   ├── vocab.json
+│   ├── merges.txt
+│   ├── tokenizer.json
+│   ├── tokenizer_config.json
+│   ├── special_tokens_map.json
+│   ├── config.json
+├── README.md
+```
 
 ## License
 This tokenizer is released under the **MIT License**.
@@ -258,9 +91,4 @@ This tokenizer is released under the **MIT License**.
 ## Citation
 If you use this tokenizer, please cite this repository.
 
----
-license: mit
-language:
-- en
-- kn
----
+For improvements and contributions, feel free to submit a **Pull Request** or open an **Issue**. 🚀