Title: Hi HN! I&#x27;ve found this visualization tool immensely helpful over the years for getting an intuition for how an LLM &quot;sees&quot; some piece of text, and with a bit of elbow grease decided to move all compute to client side so I could make it publicly available.<p>I&#x27;ve found it particularly useful for<p>- Understanding exactly how repetition and patterns affect a small LM&#x27;s ability to predict correctly<p>- Understanding different tokenization patterns and how it affects model output<p>- Getting a general sense of how &quot;hard&quot; different prediction tasks are for GPT-style models<p>Known problems (that I probably won&#x27;t fix, since this was a kind of one-off project)<p>- Doesn&#x27;t work well with Unicode grapheme clusters that are multiple GPT-2 tokens (e.g. emoji, smart quotes)<p>- Support for other models (maybe later?)
Upvote: