tokenizer-eval/tokenizer-data
Viewer • Updated • 24.8k • 25
Tokenizer Eval
non-profit
AI & ML interests
Tokenizers
Recent Activity
View all activity
[bot] Conversion to Parquet
#1 opened about 16 hours ago
by 
parquet-converter
christopher
updated 2
datasets about 1 month ago
benjamin
authored a
paper 3 months ago
benjamin
submitted a
paper to Daily Papers 3 months ago
christopher
authored a
paper 4 months ago
sanderland
authored a
paper 5 months ago
christopher
authored a
paper 5 months ago
christopher
posted an update 6 months ago
sanderland
authored a
paper 10 months ago
sanderland
authored a
paper 11 months ago
sanderland
authored a
paper 12 months ago
benjamin
authored 2
papers 12 months ago
christopher
posted an update over 1 year ago
Post
2113
The folks at Foursquare released a dataset of 104.5 million places of interest ( foursquare/fsq-os-places) and here's all of them on a plot
- 4 replies
christopher
posted an update over 1 year ago
Post
2453
The Lichess database of games, puzzles, and engine evaluations is now on the Hub: 
Lichess
Billions of chess data points to download, query, and stream and we're excited to see what you'll build with it! ♟️ 🤗
- https://huggingface.co/collections/Lichess/positions-datasets-66f50837db5cd3287d60d489
- https://huggingface.co/collections/Lichess/games-datasets-66f508df78f4b43e1bb2d353
Billions of chess data points to download, query, and stream and we're excited to see what you'll build with it! ♟️ 🤗
- https://huggingface.co/collections/Lichess/positions-datasets-66f50837db5cd3287d60d489
- https://huggingface.co/collections/Lichess/games-datasets-66f508df78f4b43e1bb2d353
christopher
posted an update over 1 year ago
Post
1347
4 million chess puzzles
benjamin
authored 3
papers almost 2 years ago
Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation
Paper • 2406.16678 • Published • 16
Zero-Shot Tokenizer Transfer
Paper • 2405.07883 • Published • 5
Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation
Paper • 2305.18893 • Published • 2