Ok from what I have understood, you can have a separate classifier for consonant and separate classifier for vowels. To both this  classifier pass the same alphabet. The consonant classifier will classify the alphabet to a consonant and the vowel classifier will classify the alphabet to a vowel. Have some threshold for the confidence score in classifiers. So you can ignore values with least confidence.