title: Segment Any Text: A Universal Approach for Robust, Efficient and
    Adaptable Sentence Segmentation
summary: Segmenting text into sentences plays an early and crucial role in many NLP
  systems. This is commonly achieved by using rule-based or statistical methods
  relying on lexical features such as punctuation. Although some recent works no
  longer exclusively rely on punctuation, we find that no prior method achieves
  all of (i) robustness to missing punctuation, (ii) effective adaptability to
  new domains, and (iii) high efficiency. We introduce a new model - Segment any
  Text (SaT) - to solve this problem. To enhance robustness, we propose a new
  pretraining scheme that ensures less reliance on punctuation. To address
  adaptability, we introduce an extra stage of parameter-efficient fine-tuning,
  establishing state-of-the-art performance in distinct domains such as verses
  from lyrics and legal documents. Along the way, we introduce architectural
  modifications that result in a threefold gain in speed over the previous state
  of the art and solve spurious reliance on context far in the future. Finally,
  we introduce a variant of our model with fine-tuning on a diverse, multilingual
  mixture of sentence-segmented data, acting as a drop-in replacement and
  enhancement for existing segmentation tools. Overall, our contributions provide
  a universal approach for segmenting any text. Our method outperforms all
  baselines - including strong LLMs - across 8 corpora spanning diverse domains
  and languages, especially in practically relevant situations where text is
  poorly formatted. Our models and code, including documentation, are available
  at https://huggingface.co/segment-any-text under the MIT license.
paper_authors._id: 667a937567be4d7b18f989c2
paper_authors.name: Markus Frohmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:36:06.767Z
paper_authors._id: 61a0b6254e926abc0de81839
paper_authors.avatarUrl: /avatars/28ba4fbb780ad4192f6fae980bba2c0d.svg
paper_authors.fullname: Markus Frohmann
paper_authors.type: user
paper_authors.user: markus583
paper_authors._id: 667a937567be4d7b18f989c3
paper_authors.name: Igor Sterner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:52:39.150Z
paper_authors._id: 639da44a7270667011143fab
paper_authors.avatarUrl: /avatars/dacad16055c175f1044a8140c532d84a.svg
paper_authors.fullname: Igor Sterner
paper_authors.type: user
paper_authors.user: igorsterner
paper_authors._id: 667a937567be4d7b18f989c4
paper_authors.name: Ivan Vulić
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:52:45.019Z
paper_authors._id: 6273e70dc8d55dd434bd8e52
paper_authors.avatarUrl: /avatars/3483eeda218e95b1eb00c3dc63c7d000.svg
paper_authors.fullname: Ivan Vulić
paper_authors.type: user
paper_authors.user: ivulic
paper_authors._id: 667a937567be4d7b18f989c5
paper_authors.name: Benjamin Minixhofer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:52:50.766Z
paper_authors._id: 5fbe9404dd6e953c5a521e13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648291574509-5fbe9404dd6e953c5a521e13.jpeg
paper_authors.fullname: Benjamin Minixhofer
paper_authors.type: user
paper_authors.user: benjamin
paper_authors._id: 667a937567be4d7b18f989c6
paper_authors.name: Markus Schedl
paper_title: Segment Any Text: A Universal Approach for Robust, Efficient and
    Adaptable Sentence Segmentation
paper_summary: Segmenting text into sentences plays an early and crucial role in many NLP
  systems. This is commonly achieved by using rule-based or statistical methods
  relying on lexical features such as punctuation. Although some recent works no
  longer exclusively rely on punctuation, we find that no prior method achieves
  all of (i) robustness to missing punctuation, (ii) effective adaptability to
  new domains, and (iii) high efficiency. We introduce a new model - Segment any
  Text (SaT) - to solve this problem. To enhance robustness, we propose a new
  pretraining scheme that ensures less reliance on punctuation. To address
  adaptability, we introduce an extra stage of parameter-efficient fine-tuning,
  establishing state-of-the-art performance in distinct domains such as verses
  from lyrics and legal documents. Along the way, we introduce architectural
  modifications that result in a threefold gain in speed over the previous state
  of the art and solve spurious reliance on context far in the future. Finally,
  we introduce a variant of our model with fine-tuning on a diverse, multilingual
  mixture of sentence-segmented data, acting as a drop-in replacement and
  enhancement for existing segmentation tools. Overall, our contributions provide
  a universal approach for segmenting any text. Our method outperforms all
  baselines - including strong LLMs - across 8 corpora spanning diverse domains
  and languages, especially in practically relevant situations where text is
  poorly formatted. Our models and code, including documentation, are available
  at https://huggingface.co/segment-any-text under the MIT license.