Segment Any Text: A Universal Approach for Robust, Efficient and
  Adaptable Sentence Segmentation
Segmenting text into sentences plays an early and crucial role in many NLP
systems. This is commonly achieved by using rule-based or statistical methods
relying on lexical features such as punctuation. Although some recent works no
longer exclusively rely on punctuation, we find that no prior method achieves
all of (i) robustness to missing punctuation, (ii) effective adaptability to
new domains, and (iii) high efficiency. We introduce a new model - Segment any
Text (SaT) - to solve this problem. To enhance robustness, we propose a new
pretraining scheme that ensures less reliance on punctuation. To address
adaptability, we introduce an extra stage of parameter-efficient fine-tuning,
establishing state-of-the-art performance in distinct domains such as verses
from lyrics and legal documents. Along the way, we introduce architectural
modifications that result in a threefold gain in speed over the previous state
of the art and solve spurious reliance on context far in the future. Finally,
we introduce a variant of our model with fine-tuning on a diverse, multilingual
mixture of sentence-segmented data, acting as a drop-in replacement and
enhancement for existing segmentation tools. Overall, our contributions provide
a universal approach for segmenting any text. Our method outperforms all
baselines - including strong LLMs - across 8 corpora spanning diverse domains
and languages, especially in practically relevant situations where text is
poorly formatted. Our models and code, including documentation, are available
at https://huggingface.co/segment-any-text under the MIT license.
667a937567be4d7b18f989c2
Markus Frohmann
claimed_verified
2024-06-26T08:36:06.767Z
61a0b6254e926abc0de81839
/avatars/28ba4fbb780ad4192f6fae980bba2c0d.svg
Markus Frohmann
user
markus583
667a937567be4d7b18f989c3
Igor Sterner
admin_assigned
2024-06-26T12:52:39.150Z
639da44a7270667011143fab
/avatars/dacad16055c175f1044a8140c532d84a.svg
Igor Sterner
user
igorsterner
667a937567be4d7b18f989c4
Ivan Vulić
admin_assigned
2024-06-26T12:52:45.019Z
6273e70dc8d55dd434bd8e52
/avatars/3483eeda218e95b1eb00c3dc63c7d000.svg
Ivan Vulić
user
ivulic
667a937567be4d7b18f989c5
Benjamin Minixhofer
admin_assigned
2024-06-26T12:52:50.766Z
5fbe9404dd6e953c5a521e13
https://cdn-avatars.huggingface.co/v1/production/uploads/1648291574509-5fbe9404dd6e953c5a521e13.jpeg
Benjamin Minixhofer
user
benjamin
667a937567be4d7b18f989c6
Markus Schedl
Segment Any Text: A Universal Approach for Robust, Efficient and
  Adaptable Sentence Segmentation
Segmenting text into sentences plays an early and crucial role in many NLP
systems. This is commonly achieved by using rule-based or statistical methods
relying on lexical features such as punctuation. Although some recent works no
longer exclusively rely on punctuation, we find that no prior method achieves
all of (i) robustness to missing punctuation, (ii) effective adaptability to
new domains, and (iii) high efficiency. We introduce a new model - Segment any
Text (SaT) - to solve this problem. To enhance robustness, we propose a new
pretraining scheme that ensures less reliance on punctuation. To address
adaptability, we introduce an extra stage of parameter-efficient fine-tuning,
establishing state-of-the-art performance in distinct domains such as verses
from lyrics and legal documents. Along the way, we introduce architectural
modifications that result in a threefold gain in speed over the previous state
of the art and solve spurious reliance on context far in the future. Finally,
we introduce a variant of our model with fine-tuning on a diverse, multilingual
mixture of sentence-segmented data, acting as a drop-in replacement and
enhancement for existing segmentation tools. Overall, our contributions provide
a universal approach for segmenting any text. Our method outperforms all
baselines - including strong LLMs - across 8 corpora spanning diverse domains
and languages, especially in practically relevant situations where text is
poorly formatted. Our models and code, including documentation, are available
at https://huggingface.co/segment-any-text under the MIT license.
A new model called Segment Any Text (SaT) is introduced for robust and efficient text segmentation across various domains, outperforming existing methods, especially in poorly formatted text.
Segment Any Text (SaT) pretraining scheme parameter-efficient fine-tuning state-of-the-art multilingual fine-tuning drop-in replacement sentence-segmented data