Update README.md

README.md

@@ -32,6 +32,223 @@ The tokenizer has been tested on multiple text categories:
 - **Punctuation Handling**: Sentences with punctuation and special characters.
 - **Special Cases**: URLs, hashtags, emojis, and file paths.
 
+## Tokenizer Test Results
+
+
+
+Category: Pure Kannada
+
+
+Test Case 1: Basic sentence
+Original text: ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ
+Encoded tokens: ['<s>', 'ನಮಸ', 'à³į', 'à²ķ', 'ಾ', 'ರ', 'Ġà²ķನ', 'à³į', 'ನಡ', 'Ġà²Ń', 'ಾ', 'ಷ', 'à³Ĩ', '</s>']
+Token IDs: [0, 1461, 264, 278, 270, 272, 738, 264, 407, 386, 270, 323, 268, 1]
+Decoded text: ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ
+Analysis:
+- Number of tokens: 14
+- Average token length: 1.29 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 2: Complex sentence
+Original text: ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ
+Encoded tokens: ['<s>', 'à²ķನ', 'à³į', 'ನಡ', 'Ġನ', 'ಾ', 'ಡ', 'ಿ', 'ನ', 'Ġಸ', 'à²Ĥ', 'ಸ', 'à³į', 'à²ķ', 'à³ĥ', 'ತ', 'ಿ', 'Ġಮತ', 'à³į', 'ತ', 'à³ģ', 'Ġಪರ', 'à²Ĥ', 'ಪರ', 'à³Ĩ', '</s>']
+Token IDs: [0, 754, 264, 407, 298, 270, 280, 267, 266, 300, 275, 281, 264, 278, 412, 271, 267, 382, 264, 271, 265, 360, 275, 524, 268, 1]
+Decoded text: ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ
+Analysis:
+- Number of tokens: 26
+- Average token length: 1.27 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+
+Category: Mixed Language
+
+
+Test Case 1: Kannada with English
+Original text: ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ
+Encoded tokens: ['<s>', 'ನನ', 'à³į', 'ನ', 'Ġ', 'e', 'm', 'a', 'i', 'l', 'Ġ', 'I', 'D', 'Ġà²ĩದ', 'à³ģ', 'Ġ', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '@', 'e', 'm', 'a', 'i', 'l', '.', 'com', 'Ġà²Ĩà²Ĺ', 'ಿ', 'ದ', 'à³Ĩ', '</s>']
+Token IDs: [0, 306, 264, 266, 225, 73, 81, 69, 77, 80, 225, 45, 40, 493, 265, 225, 73, 92, 69, 81, 84, 80, 73, 36, 73, 81, 69, 77, 80, 18, 469, 408, 267, 269, 268, 1]
+Decoded text: ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ
+Analysis:
+- Number of tokens: 36
+- Average token length: 1.14 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 2: Technical terms
+Original text: ಈ mobile phone ನಲ್ಲಿ 4G network ಇದೆ
+Encoded tokens: ['<s>', 'à²Ī', 'Ġ', 'm', 'o', 'b', 'i', 'l', 'e', 'Ġ', 'p', 'h', 'o', 'n', 'e', 'Ġನಲ', 'à³į', 'ಲ', 'ಿ', 'Ġ', '4', 'G', 'Ġ', 'n', 'e', 't', 'w', 'o', 'r', 'k', 'Ġà²ĩದ', 'à³Ĩ', '</s>']
+Token IDs: [0, 725, 225, 81, 83, 70, 77, 80, 73, 225, 84, 76, 83, 82, 73, 684, 264, 274, 267, 225, 24, 43, 225, 82, 73, 88, 91, 83, 86, 79, 493, 268, 1]
+Decoded text: ಈ mobile phone ನಲ್ಲಿ 4G network ಇದೆ
+Analysis:
+- Number of tokens: 33
+- Average token length: 1.06 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 3: Social media
+Original text: ಅವರ Instagram profile ತುಂಬಾ popular ಆಗಿದೆ
+Encoded tokens: ['<s>', 'à²ħವರ', 'Ġ', 'I', 'n', 's', 't', 'a', 'g', 'r', 'a', 'm', 'Ġ', 'p', 'r', 'o', 'f', 'i', 'l', 'e', 'Ġತ', 'à³ģà²Ĥ', 'ಬ', 'ಾ', 'Ġ', 'p', 'o', 'p', 'u', 'l', 'a', 'r', 'Ġà²Ĩà²Ĺ', 'ಿ', 'ದ', 'à³Ĩ', '</s>']
+Token IDs: [0, 744, 225, 45, 82, 87, 88, 69, 75, 86, 69, 81, 225, 84, 86, 83, 74, 77, 80, 73, 299, 336, 293, 270, 225, 84, 83, 84, 89, 80, 69, 86, 408, 267, 269, 268, 1]
+Decoded text: ಅವರ Instagram profile ತುಂಬಾ popular ಆಗಿದೆ
+Analysis:
+- Number of tokens: 37
+- Average token length: 1.11 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+
+Category: Numbers and Dates
+
+
+Test Case 1: Pure numbers
+Original text: ೧೨೩೪೫ 12345
+Encoded tokens: ['<s>', '೧', 'à³', '¨', 'à³', '©', 'à³', 'ª', 'à³', '«', 'Ġ', '1', '2', '3', '4', '5', '</s>']
+Token IDs: [0, 2891, 262, 106, 262, 107, 262, 108, 262, 109, 225, 21, 22, 23, 24, 25, 1]
+Decoded text: ೧೨೩೪೫ 12345
+Analysis:
+- Number of tokens: 17
+- Average token length: 0.65 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 2: Mixed numbers
+Original text: ನನ್ನ ಫೋನ್ ನಂಬರ್ +91 9876543210
+Encoded tokens: ['<s>', 'ನನ', 'à³į', 'ನ', 'Ġಫ', 'à³', 'ĭ', 'ನ', 'à³į', 'Ġನ', 'à²Ĥ', 'ಬರ', 'à³į', 'Ġ', '+', '9', '1', 'Ġ', '9', '8', '7', '6', '5', '4', '3', '2', '1', '0', '</s>']
+Token IDs: [0, 306, 264, 266, 1055, 262, 238, 266, 264, 298, 275, 505, 264, 225, 15, 29, 21, 225, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 1]
+Decoded text: ನನ್ನ ಫೋನ್ ನಂಬರ್ +91 9876543210
+Analysis:
+- Number of tokens: 29
+- Average token length: 1.03 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 3: Date formats
+Original text: ೨೦೨೪ ಫೆಬ್ರವರಿ ೧೪ / 14-02-2024
+Encoded tokens: ['<s>', 'à³', '¨', 'à³', '¦', 'à³', '¨', 'à³', 'ª', 'Ġಫ', 'à³Ĩ', 'ಬ', 'à³į', 'ರ', 'ವರ', 'ಿ', 'Ġ', '೧', 'à³', 'ª', 'Ġ', '/', 'Ġ', '1', '4', '-', '0', '2', '-', '2', '0', '2', '4', '</s>']
+Token IDs: [0, 262, 106, 262, 104, 262, 106, 262, 108, 1055, 268, 293, 264, 272, 328, 267, 225, 2891, 262, 108, 225, 19, 225, 21, 24, 17, 20, 22, 17, 22, 20, 22, 24, 1]
+Decoded text: ೨೦೨೪ ಫೆಬ್ರವರಿ ೧೪ / 14-02-2024
+Analysis:
+- Number of tokens: 34
+- Average token length: 0.85 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 4: Currency
+Original text: ₹೧೦೦೦.೫೦ ರೂಪಾಯಿ / Rs. 1000.50
+Encoded tokens: ['<s>', 'â', 'Ĥ', '¹', '೧', 'à³', '¦', 'à³', '¦', 'à³', '¦', '.', 'à³', '«', 'à³', '¦', 'Ġರ', 'à³Ĥ', 'ಪ', 'ಾ', 'ಯ', 'ಿ', 'Ġ', '/', 'Ġ', 'R', 's', '.', 'Ġ', '1', '0', '0', '0', '.', '5', '0', '</s>']
+Token IDs: [0, 163, 229, 122, 2891, 262, 104, 262, 104, 262, 104, 18, 262, 109, 262, 104, 371, 289, 291, 270, 277, 267, 225, 19, 225, 54, 87, 18, 225, 21, 20, 20, 20, 18, 25, 20, 1]
+Decoded text: ₹೧೦೦೦.೫೦ ರೂಪಾಯಿ / Rs. 1000.50
+Analysis:
+- Number of tokens: 37
+- Average token length: 0.78 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+
+Category: Punctuation
+
+
+Test Case 1: Basic punctuation
+Original text: ನೀವು ಹೇಗಿದ್ದೀರಿ? ನಾನು ಚೆನ್ನಾಗಿದ್ದೇನೆ!
+Encoded tokens: ['<s>', 'ನ', 'à³Ģ', 'ವ', 'à³ģ', 'Ġಹ', 'à³', 'ĩ', 'à²Ĺ', 'ಿ', 'ದ', 'à³į', 'ದ', 'à³Ģ', 'ರ', 'ಿ?', 'Ġನ', 'ಾ', 'ನ', 'à³ģ', 'Ġà²ļ', 'à³Ĩ', 'ನ', 'à³į', 'ನ', 'ಾ', 'à²Ĺ', 'ಿ', 'ದ', 'à³į', 'ದ', 'à³', 'ĩ', 'ನ', 'à³Ĩ!', '</s>']
+Token IDs: [0, 266, 276, 279, 265, 284, 262, 234, 273, 267, 269, 264, 269, 276, 272, 813, 298, 270, 266, 265, 339, 268, 266, 264, 266, 270, 273, 267, 269, 264, 269, 262, 234, 266, 532, 1]
+Decoded text: ನೀವು ಹೇಗಿದ್ದೀರಿ? ನಾನು ಚೆನ್ನಾಗಿದ್ದೇನೆ!
+Analysis:
+- Number of tokens: 36
+- Average token length: 1.03 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 2: Special characters
+Original text: ಇಂದು @bangalore #Karnataka (ಕರ್ನಾಟಕ) ರಾಜ್ಯ
+Encoded tokens: ['<s>', 'à²ĩ', 'à²Ĥ', 'ದ', 'à³ģ', 'Ġ', '@', 'b', 'an', 'g', 'a', 'l', 'o', 'r', 'e', 'Ġ', '#', 'K', 'a', 'r', 'n', 'a', 't', 'a', 'k', 'a', 'Ġ(', 'à²ķರ', 'à³į', 'ನ', 'ಾ', 'à²Łà²ķ', ')', 'Ġರ', 'ಾ', 'à²ľ', 'à³į', 'ಯ', '</s>']
+Token IDs: [0, 447, 275, 269, 265, 225, 36, 70, 380, 75, 69, 80, 83, 86, 73, 225, 7, 47, 69, 86, 82, 69, 88, 69, 79, 69, 443, 368, 264, 266, 270, 482, 13, 371, 270, 330, 264, 277, 1]
+Decoded text: ಇಂದು @bangalore #Karnataka (ಕರ್ನಾಟಕ) ರಾಜ್ಯ
+Analysis:
+- Number of tokens: 39
+- Average token length: 1.08 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 3: Mixed punctuation
+Original text: "ಕನ್ನಡ" - 'ನಾಡಿನ' & ಸಂಸ್ಕೃತಿ...
+Encoded tokens: ['<s>', '"', 'à²ķನ', 'à³į', 'ನಡ', '"', 'Ġ-', 'Ġ', "'", 'ನ', 'ಾ', 'ಡ', 'ಿ', 'ನ', "'", 'Ġ', '&', 'Ġಸ', 'à²Ĥ', 'ಸ', 'à³į', 'à²ķ', 'à³ĥ', 'ತ', 'ಿ.', '.', '.', '</s>']
+Token IDs: [0, 6, 754, 264, 407, 6, 438, 225, 11, 266, 270, 280, 267, 266, 11, 225, 10, 300, 275, 281, 264, 278, 412, 271, 517, 18, 18, 1]
+Decoded text: "ಕನ್ನಡ" - 'ನಾಡಿನ' & ಸಂಸ್ಕೃತಿ...
+Analysis:
+- Number of tokens: 28
+- Average token length: 1.11 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+
+Category: Special Cases
+
+
+Test Case 1: URLs
+Original text: ನಮ್ಮ ವೆಬ್‌ಸೈಟ್: https://kannada.example.com
+Encoded tokens: ['<s>', 'ನಮ', 'à³į', 'ಮ', 'Ġವ', 'à³Ĩ', 'ಬ', 'à³į', 'âĢ', 'Į', 'ಸ', 'à³Ī', 'à²Ł', 'à³į:', 'Ġhttps', '://', 'kannad', 'a', '.', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '.', 'com', '</s>']
+Token IDs: [0, 765, 264, 285, 332, 268, 293, 264, 297, 239, 281, 335, 286, 3786, 473, 452, 471, 69, 18, 73, 92, 69, 81, 84, 80, 73, 18, 469, 1]
+Decoded text: ನಮ್ಮ ವೆಬ್‌ಸೈಟ್: https://kannada.example.com
+Analysis:
+- Number of tokens: 29
+- Average token length: 1.48 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 2: Hashtags
+Original text: #ಕನ್ನಡ_ರಾಜ್ಯೋತ್ಸವ #Kannada
+Encoded tokens: ['<s>', '#', 'à²ķನ', 'à³į', 'ನಡ', '_', 'ರ', 'ಾ', 'à²ľ', 'à³į', 'ಯ', 'à³', 'ĭ', 'ತ', 'à³į', 'ಸವ', 'Ġ', '#', 'K', 'an', 'nad', 'a', '</s>']
+Token IDs: [0, 7, 754, 264, 407, 67, 272, 270, 330, 264, 277, 262, 238, 271, 264, 595, 225, 7, 47, 380, 461, 69, 1]
+Decoded text: #ಕನ್ನಡ_ರಾಜ್ಯೋತ್ಸವ #Kannada
+Analysis:
+- Number of tokens: 23
+- Average token length: 1.13 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 3: Emojis
+Original text: ಕನ್ನಡ ನಾಡು 🚩 ಕನ್ನಡ ಭಾಷೆ ❤️
+Encoded tokens: ['<s>', 'à²ķನ', 'à³į', 'ನಡ', 'Ġನ', 'ಾ', 'ಡ', 'à³ģ', 'Ġ', 'ð', 'Ł', 'ļ', '©', 'Ġà²ķನ', 'à³į', 'ನಡ', 'Ġà²Ń', 'ಾ', 'ಷ', 'à³Ĩ', 'Ġ', 'â', 'Ŀ', '¤', 'ï', '¸', 'ı', '</s>']
+Token IDs: [0, 754, 264, 407, 298, 270, 280, 265, 225, 177, 258, 253, 107, 738, 264, 407, 386, 270, 323, 268, 225, 163, 256, 102, 176, 121, 242, 1]
+Decoded text: ಕನ್ನಡ ನಾಡು 🚩 ಕನ್ನಡ ಭಾಷೆ ❤️
+Analysis:
+- Number of tokens: 28
+- Average token length: 0.93 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+Test Case 4: File paths
+Original text: C:\Users\ಬಳಕೆದಾರ\Documents\ಕನ್ನಡ.txt
+Encoded tokens: ['<s>', 'C', ':', '\\', 'U', 's', 'e', 'r', 's', '\\', 'ಬಳà²ķ', 'à³Ĩ', 'ದ', 'ಾ', 'ರ', '\\', 'D', 'o', 'c', 'u', 'm', 'e', 'n', 't', 's', '\\', 'à²ķನ', 'à³į', 'ನಡ', '.', 't', 'x', 't', '</s>']
+Token IDs: [0, 39, 30, 64, 57, 87, 73, 86, 87, 64, 4023, 268, 269, 270, 272, 64, 40, 83, 71, 89, 81, 73, 82, 88, 87, 64, 754, 264, 407, 18, 88, 92, 88, 1]
+Decoded text: C:\Users\ಬಳಕೆದಾರ\Documents\ಕನ್ನಡ.txt
+Analysis:
+- Number of tokens: 34
+- Average token length: 1.06 characters
+- Reconstruction: Perfect
+
+----------------------------------------
+
+
+
 ## Repository Structure
 The repository consists of tokenizer files, configuration files, and documentation.