I definitely would be interested in the results.
You might also want to try stemming and/or lemmatization. From a human perspective, it destroys proper sentences, of course. But again, the model might not (have to) care to identify named entities, and it certainly benefits greatly from the reduced vocabulary.
The model can only find patterns between the input and labels. But if that also means that the model “understands language” or at least to what degree is highly debatable. I recommend this fiery blog post 11 :).
In short, I wouldn’t over-emphasize to importance of proper sentences.