  Lipreading is understanding speech from observed lip movements. An observed
series of lip motions is an ordered sequence of visual lip gestures. These
gestures are commonly known, but as yet are not formally defined, as `visemes'.
In this article, we describe a structured approach which allows us to create
speaker-dependent visemes with a fixed number of visemes within each set. We
create sets of visemes for sizes two to 45. Each set of visemes is based upon
clustering phonemes, thus each set has a unique phoneme-to-viseme mapping. We
first present an experiment using these maps and the Resource Management
Audio-Visual (RMAV) dataset which shows the effect of changing the viseme map
size in speaker-dependent machine lipreading and demonstrate that word
recognition with phoneme classifiers is possible. Furthermore, we show that
there are intermediate units between visemes and phonemes which are better
still. Second, we present a novel two-pass training scheme for phoneme
classifiers. This approach uses our new intermediary visual units from our
first experiment in the first pass as classifiers; before using the
phoneme-to-viseme maps, we retrain these into phoneme classifiers. This method
significantly improves on previous lipreading results with RMAV speakers.

Image and Video Processing;Audio and Speech Processing