Spaces:

getGO007
/

croakenizer

Sleeping

getGO007 commited on Apr 9

Commit

8fe0ddf

verified ·

1 Parent(s): 30b283c

Update app.py

Files changed (1) hide show

app.py CHANGED Viewed

@@ -22,6 +22,7 @@ class Tokenizer:
         ('NUMBER', r'\d+'),
         ('WORD',   r'[A-Za-z]+'),
         ('SPACE',  r'\s+'),
     ]
     combined_pattern = '|'.join(

         ('NUMBER', r'\d+'),
         ('WORD',   r'[A-Za-z]+'),
         ('SPACE',  r'\s+'),
+        ('PUNCT',  r'[^\w\s]'),  # <--- Added punctuation pattern
     ]
     combined_pattern = '|'.join(