What about bigrams (pairs of letters) and trigrams? Are some more common than others? Does that depend on the position in the sentence?

Are you suggesting building the one hot encoded classes based on bigrams / trigrams / quadgrams instead of the individual letters that exist in the corpus alphabet?