I think the encoding with “break” makes it a bit hard.
If you encode sequence ids instead (can be done with comparison and cumsum if you don’t have it already), you can use index_add or some scatter_add implementation.
Last I looked the public implementations for PyTorch were not terribly optimized but what can you do.