what would future experiments entail ?[SEP] <s>grad e: ok , we 're on . professor b: ok . grad e: so , i mean , everyone who 's on the wireless check that they 're on . phd f: c we grad g: alright . postdoc c: i see . yeah . phd f: yeah . grad e: ok , our agenda was quite short . professor b: oh , could you close the door , maybe ? yeah . grad e: sure . two items , which was , uh , digits and possibly stuff on on , uh , forced alignment , which jane said that liz and andreas had in information on , professor b:  grad e: but they did n't , phd f: mm - hmm . professor b: i guess the only other thing , uh , for which i grad e: so . phd f: we should do that second , because liz might join us in time for that . grad e: ok . professor b: um . ok , so there 's digits , alignments , and , um , i guess the other thing , which i came unprepared for , uh , is , uh , to dis s s see if there 's anything anybody wants to discuss about the saturday meeting . grad e: right . professor b: so . any i mean , maybe not . grad e: digits and alignments . but professor b: uh . phd f: talk about aligning people 's schedules . professor b: yeah . grad e: yeah . postdoc c: mm - hmm . professor b: yeah . i mean right . yeah , i mean , it was grad e: yeah , it 's forced alignment of people 's schedules . phd f: yeah . phd d: forced align . phd f: if we 're very professor b: yeah . phd f: yeah . professor b: with with whatever it was , a month and a half or something ahead of time , the only time we could find in common roughly in common , was on a saturday . phd d: yeah . professor b: ugh . grad e: yep . phd f: it 's pretty sad . professor b: yeah . phd f: yeah . postdoc c: have have we thought about having a conference call to include him in more of in more of the meeting ? i i mean , i do n't know , if we had the if we had the telephone on the table professor b: no . but , h i mean , he probably has to go do something . phd f: no , actually i i have to i have to shuttle kids from various places to various other places . professor b: right ? postdoc c: i see . ok . professor b: yeah . phd f: so . and i do n't have and i do n't , um , have a cell phone phd d: a cell phone ? phd f: so i ca n't be having a conference call while driving . professor b: r r right . postdoc c: no . { comment } it 's not good . professor b: so we have to we postdoc c: that 's not good . phd f: plus , it would make for interesting noise background noise . professor b:  grad e: yep . phd f: uh professor b: so we have to equip him with a with a with a head - mounted , uh , cell phone grad e: ye - we and we 'd have to force you to read lots and lots of digits , professor b: and grad e: so it could get real real car noise . phd f: oh , yeah . phd d: yeah . phd f: oh , yeah . grad g: take advantage . phd d: and with the kids in the background . phd f: i 'll let i 'd let phd d: yeah . phd f: i let , uh , my five - year - old have a try at the digits , eh . professor b: yeah . grad e: so , anyway , i can talk about digits . um , did everyone get the results or shall i go over them again ? i mean that it was basically the only thing that was even slightly surprising was that the lapel did so well . um , and in retrospect that 's not as surprising as maybe i it should n't have been as surprising as i as as i felt it was . the lapel mike is a very high - quality microphone . and as morgan pointed out , that there are actually some advantages to it in terms of breath noises and clothes rustling if no one else is talking . phd d: yeah . phd f: exactly . grad e: um , so , uh grad g: mm - hmm . professor b: well , it 's yeah , sort of the bre the breath noises and the mouth clicks and so forth like that , the lapel 's gon na be better on . grad g: it 's g it phd d: or the cross - talk . yeah . professor b: the lapel is typically worse on the on clothes rustling , but if no one 's rustling their clothes , grad e: right . i mean , a lot of people are just sort of leaning over and reading the digits , professor b: it 's it 's grad e: so it 's it 's a very different task than sort of the natural . phd d: yeah . you do n't move much during reading digits , i think . professor b: yeah . grad e: so . professor b: yeah . grad e: right . grad g: probably the fact that it picks up other people 's speakers other people 's talking is an indication of that it the fact it is a good microphone . phd d: yeah . professor b: right . so in the digits , in most most cases , there were n't other people talking . grad e: right . right . grad g: so . professor b: so . phd f: d do the lapel mikes have any directionality to them ? professor b: there typically do n't , no . phd f: because i i suppose you could make some that have sort of that you have to orient towards your mouth , grad e: they have a little bit , phd f: and then it would grad e: but they 're not noise - cancelling . so , uh professor b: they 're they 're intended to be omni - directional . grad e: right . professor b: and th it 's and because you do n't know how people are gon na put them on , you know . phd f: mm - hmm . grad e: right . so , also , andreas , on that one the the back part of it should be right against your head . and that will he keep it from flopping aro up and down as much . phd f: it is against my head . grad e: ok . professor b: yeah . um . yeah , we actually talked about this in the , uh , front - end meeting this morning , too . much the same thing , grad e: uh - huh . professor b: and and it was uh , i mean , there the point of interest to the group was primarily that , um , the , uh the system that we had that was based on h t k , that 's used by , you know , all the participants in aurora , was so much worse than the than the s r grad e: everybody . professor b: and the interesting thing is that even though , yes , it 's a digits task and that 's a relatively small number of words and there 's a bunch of digits that you train on , it 's just not as good as having a a l very large amount of data and training up a a a nice good big hmm . um , also you had the adaptation in the sri system , which we did n't have in this . um . so . um . phd f: and we know di - did i send you some results without adaptation ? grad e: no . professor b: i s i think stephane , uh , had seen them . grad e: or if you did , i did n't include them , cuz it was professor b: so phd f: yeah , i think i did , actually . so there was a significant loss from not doing the adaptation . professor b: yeah . phd f: um . a a a couple percent or some i mean well , i do n't know it overall uh , i i do n't remember , but there was there was a significant , um , loss or win { comment } from adaptation with with adaptation . and , um , that was the phone - loop adaptation . and then there was a very small like point one percent on the natives uh , win from doing , um , you know , adaptation to the recognition hypotheses . and i tried both means adaptation and means and variances , and the variances added another or subtracted another point one percent . so , it 's , um that 's the number there . point six , i believe , is what you get with both , uh , means and variance adaptation . grad e: right . professor b: but i think one thing is that , uh , i would presume hav - have you ever t have you ever tried this exact same recognizer out on the actual ti - digits test set ? phd f: this exact same recognizer ? no . professor b: it might be interesting to do that . cuz my my cuz my sense , um phd f: but but , i have i mean , people people at sri are actually working on digits . grad e: i bet it would do even slightly better . phd f: i could and they are using a system that 's , um you know , h is actually trained on digits , um , but h h otherwise uses the same , you know , decoder , the same , uh , training methods , and so forth , professor b: mm - hmm . phd f: and i could ask them what they get on ti - digits . professor b: yeah , bu although i 'd be i think it 'd be interesting to just take this exact actual system so that these numbers were comparable phd f: mm - hmm . professor b: and try it out on ti - digits . phd f: well , adam knows how to run it , professor b: yeah . grad e: yeah . no problem . phd f: so you just make a f professor b: yeah . yeah . cuz our sense from the other from the aurora , uh , task is that grad e: and try it with ti - digits ? phd f: mm - hmm . professor b: i mean , cuz we were getting sub one percent numbers on ti - digits also with the tandem thing . phd f: mm - hmm . professor b: so , one so there were a number of things we noted from this . phd f: mmm . professor b: one is , yeah , the sri system is a lot better than the htk phd f: hmm . professor b: this , you know , very limited training htk system . phd f: mm - hmm . professor b: uh , but the other is that , um , the digits recorded here in this room with these close mikes , i uh , are actually a lot harder than the studio - recording ti - digits . i think , you know , one reason for that , uh , might be that there 's still even though it 's close - talking , there still is some noise and some room acoustics . phd f: mm - hmm . mm - hmm . professor b: and another might be that , uh , i 'd i would presume that in the studio , uh , uh , situation recording read speech that if somebody did something a little funny or n pronounced something a little funny or made a little that they did n't include it , grad e: they did n't include it . professor b: they made them do it again . grad e: whereas , i took out the ones that i noticed that were blatant that were correctable . professor b: mmm . yeah . grad e: so that , if someone just read the wrong digit , i corrected it . professor b: yeah . grad e: and then there was another one where jose could n't tell whether i could n't tell whether he was saying zero or six . and i asked him and he could n't tell either . grad i: hmm . grad e: so i just cut it out . professor b: yeah . grad e: you know , so i just e edited out the first , i uh , word of the utterance . um , so there 's a little bit of correction but it 's definitely not as clean as ti - digits . so my expectations is ti - digits would , especially i think ti - digits is all american english . professor b: mm - hmm . grad e: right ? so it would probably do even a little better still on the sri system , but we could give it a try . phd f: well . but remember , we 're using a telephone bandwidth front - end here , uh , on this , uh on this sri system , so , um , i was i thought that maybe that 's actually a good thing because it it gets rid of some of the uh , the noises , um , you know , in the the below and above the um , the , you know , speech bandwidth professor b: mm - hmm . mm - hmm . phd f: and , um , i suspect that to get sort of the last bit out of these higher - quality recordings you would have to in fact , uh , use models that , uh , were trained on wider - band data . and of course we ca n't do that or grad e: wha - what 's ti - digits ? i thought t professor b: it 's wide - band , yeah . it 's in in fact , we looked it up grad e: it is wide - band . ok . professor b: and it was actually twenty kilohertz sampling . grad e: oh , that 's right . i i did look that up . phd f: mm - hmm . grad e: i could n't remember whether that was ti - digits or one of the other digit tasks . professor b: yeah . phd f: right . but but , i would yeah . it 's it 's easy enough to try , just run it on professor b: yeah . grad e: mm - hmm . professor b: see w grad e: so , morgan , you 're getting a little breath noise . phd f: now , eh , does grad e: you might wan na move the mike down a little bit . phd f: one one issue one issue with with that is that um , the system has this , uh , notion of a speaker to which is used in adaptation , variance norm uh , you know , both in , uh , mean and variance normalization and also in the vtl estimation . professor b: mm - hmm . phd f: so grad e: yeah , i noticed the script that extracted it . phd f: do y ? is ? so does so th so does does , um , the ti - digits database have speakers that are known ? grad e: yep . yep . phd f: and is there is there enough data or a comparable comparable amount of data to to what we have in our recordings here ? grad e: that i do n't know . i do n't know . i do n't know how many speakers there are , professor b: yeah . grad e: and and how many speakers per utterance . phd f: ok . professor b: well , the other thing would be to do it without the adaptation and compare to these numbers without the adaptation . that would phd f: right . uh , but i 'm not so much worried about the adaptation , actually , than than the , um , um the , uh , vtl estimation . grad e: right . phd f: if you have only one utterance per speaker you might actually screw up on estimating the the warping , uh , factor . so , um grad e: i strongly suspect that they have more speakers than we do . so , uh phd f: right . but it 's not the amount of speakers , it 's the num it 's the amount of data per speaker . grad e: right . so we we could probably do an extraction that was roughly equivalent . phd f: right . right . grad e: um . phd f: so grad e: so , although i i sort of know how to run it , there are a little a f few details here and there that i 'll have to dig out . phd f: ok . the key so th the system actually extracts the speaker id from the waveform names . grad e: right . i saw that . phd f: and there 's a there 's a script and that is actually all in one script . so there 's this one script that parses waveform names and extracts things like the , um , speaker , uh , id or something that can stand in as a speaker id . so , we might have to modify that script to recognize the , um , speakers , um , in the in the , uh , um , ti - digits database . grad e: right . right . and that , uh phd f: or you can fake you can fake names for these waveforms that resemble the names that we use here for the for the meetings . grad e: right . phd f: that would be the , sort of probably the safest way to do grad e: i might have to do that anyway to to do because we may have to do an extract to get the amount of data per speaker about right . phd f: uh - huh . grad e: the other thing is , is n't ti - digits isolated digits ? phd f: right . grad e: or is that another one ? i 'm i looked through a bunch of the digits t corp corpora , and now they 're all blurring . professor b: mm - hmm . grad e: cuz one of them was literally people reading a single digit . and then others were connected digits . professor b: yeah . most of ti - digits is connected digits , i think . grad e: ok . professor b: the i mean , we had a bellcore corpus that we were using . it was that 's that was isolated digits . grad e: maybe it 's the bell gram . bell digits . alright . professor b: um . phd f: by the way , i think we can improve these numbers if we care to compr improve them by , um , not starting with the switchboard models but by taking the switchboard models and doing supervised adaptation on a small amount of digit data collected in this setting . grad e: yep . phd f: because that would adapt your models to the room acoustics and f for the far - field microphones , you know , to the noise . and that should really improve things , um , further . and then you use those adapted models , which are not speaker adapted but sort of acous you know , channel adapted grad e: channel adapted . phd f: use that as the starting models for your speaker adaptation . professor b: yeah . but the thing is , uh i mean , w when you it depends whether you 're ju were just using this as a a starter task for you know , to get things going for conversational or if we 're really interested i in connected digits . and i i think the answer is both . and for for connected digits over the telephone you do n't actually want to put a whole lot of effort into adaptation phd f: well , i do n't know . professor b: because somebody gets on the phone and says a number and then you just want it . you do n't do n't , uh postdoc c: this is this that one 's better . phd f: right . postdoc c: mm - hmm . phd f: um , but , you know , i uh , my impression was that you were actually interested in the far - field microphone , uh , problem , i mean . so , you want to you want to that 's the obvious thing to try . postdoc c: oh . oh . professor b: right . phd f: right ? then , eh because you you do n't have any postdoc c: yeah . phd f: that 's where the most m acoustic mismatch is between the currently used models and the the r the set up here . professor b: right . phd f: so . professor b: yeah . so that 'd be anoth another interesting data point . phd f: mm - hmm . professor b: i mean , i i guess i 'm saying i do n't know if we 'd want to do that as the as phd d: other way . grad e: other way . liz phd a: now you 're all watching me . grad e: it f it clips over your ears . phd a: alright . this way . grad e: there you go . postdoc c: if you have a strong fe if you have a strong preference , you could use this . phd a: you 're all watching . this is terrible . postdoc c: it 's just we we think it has some spikes . so , uh , we we did n't use that one . phd a: i 'll get it . postdoc c: but you could if you want . professor b: yeah . at any rate , i do n't know if w postdoc c: i do n't know . and andre - andreas , your your microphone 's a little bit low . professor b: yeah . phd f: it is ? professor b: i do n't know if we wan na use that as the postdoc c: yeah . grad e: uh , it pivots . phd f: uh . postdoc c: so if you see the picture grad e: it it like this . phd f: i i postdoc c: and then you have to scr phd f: i i already adjusted this a number of times . grad e: eh . phd f: i i grad e: yeah , i think these mikes are not working as well as i would like . phd f: ca n't quite seem to yeah , i think this contraption around your head is not working so well . professor b: too many adju too many adjustments . yeah . anyway , what i was saying is that i i think i probably would n't want to see that as sort of like the norm , that we compared all things to . postdoc c: that looks good . yeah . professor b: to , uh , the to have have all this ad all this , uh , adaptation . but i think it 's an important data point , if you 're if yeah . phd f: right . professor b: um . the other thing that that , uh of course , what barry was looking at was was just that , the near versus far . and , yeah , the adaptation would get th some of that . phd f: mm - hmm . professor b: but , i think even even if there was , uh , only a factor of two or something , like i was saying in the email , i think that 's that 's a big factor . so phd f: mm - hmm . professor b: n grad e: liz , you could also just use the other mike if you 're having problems with that one . postdoc c: well . phd a: ok . postdoc c: yeah . this would be ok . we we we think that this has spikes on it , phd a: it 's this thing 's this is too big for my head . postdoc c: so it 's not as good acoustically , phd f: yeah , basically your ears are too big . postdoc c: but phd f: i mean , mine are too . e th everybody 's ears are too big for these things . phd a: no , my my but this is too big for my head . so , i mean , { comment } { comment } it does n't you know , it 's sit phd f: uh postdoc c: well , if you 'd rather have this one then it 's phd a: ok . professor b: yeah . grad e: oh , well . professor b: it 's great . grad e: so the to get that , uh , pivoted this way , it pivots like this . phd a: no this way . yeah . grad e: yeah . there you go . postdoc c: and there 's a screw that you can tighten . grad e: and then it phd a: right . grad e: right . phd a: i already tried to get it close . postdoc c: good . grad e: so if it does n't bounce around too much , that 's actually good placement . phd a: ok . postdoc c: that looks good . grad e: but it looks like it 's gon na bounce a lot . professor b: so , where were we ? uh yeah . postdoc c: yeah . grad e: digits . adaptation . professor b: uh , adaptation , non - adaptation , um , factor of two , um oh , yeah . i know what i was go w phd f: what k u by the way , wh what factor of two did you ? professor b: oh , no , no . phd f: i mean professor b: it 's tha that that we were saying , you know , well is how much worse is far than near , you know . phd f: oh , th ok . professor b: and i mean it depends on which one you 're looking at , phd f: that factor of two . professor b: but for the everybody , it 's little under a factor or two . phd f: mm - hmm . professor b: yeah . i i know what i was thinking was that maybe , uh , i i we could actually t t try at least looking at , uh , some of the the large vocabulary speech from a far microphone , at least from the good one . phd f: mm - hmm . professor b: i mean , before i thought we 'd get , you know , a hundred and fifty percent error or something , but if if , uh if we 're getting thirty - five , forty percent or something , u um phd f: mm - hmm . phd a: actually if you run , though , on a close - talking mike over the whole meeting , during all those silences , you get , like , four hundred percent word error . professor b: mm - hmm . right . i understand . but doing the same kind of limited thing phd a: or or some high number . professor b: yeah , sure . get all these insertions . but i 'm saying if you do the same kind of limited thing as people have done in switchboard evaluations or as a phd a: yeah . where you know who the speaker is and there 's no overlap ? and you do just the far - field for those regions ? professor b: yeah . yeah . the same sort of numbers that we got those graphs from . right ? grad e: could we do exactly the same thing that we 're doing now , but do it with a far - field mike ? professor b: yeah , do it with one of on grad e: cuz we extract the times from the near - field mike , but you use the acoustics from the far - field mike . phd a: right . i understand that . i just meant that so you have three choices . there 's , um you can use times where that person is talking only from the transcripts but the segmentations were were synchronized . or you can do a forced alignment on the close - talking to determine that , the you know , within this segment , these really were the times that this person was talking and elsewhere in the segment other people are overlapping and just front - end those pieces . or you can run it on the whole data , which is which is , you know , a professor b: but but but how did we get the how did we determine the links , uh , that we 're testing on in the stuff we reported ? phd a: in the h l t paper we took segments that are channel time - aligned , which is now h being changed in the transcription process , which is good , and we took cases where the transcribers said there was only one person talking here , because no one else had time any words in that segment and called that `` non - overlap `` . professor b: and tha and that 's what we were getting those numbers from . phd a: yes . tho - good the good numbers . professor b: right . phd a: the bad numbers were from the segments where there was overlap . professor b: well , we could start with the good ones . phd a: yeah . professor b: but anyway so i think that we should try it once with the same conditions that were used to create those , and in those same segments just use one of the p z phd a: right . so we we can do that . yeah . professor b: and then , you know , i mean , the thing is if we were getting , uh what , thirty - five , forty percent , something like that on on that particular set , uh , does it go to seventy or eighty ? phd a: right . professor b: or , does it use up so much memory we ca n't decode it ? phd a: it might also depend on which speaker th it is and how close they are to the pzm ? professor b: uh phd a: i do n't know how different they are from each other . phd f: you want to probably choose the pzm channel that is closest to the speaker . phd a: to be best phd d: yeah . grad e: for this particular digit ones , i just picked that one . phd a: f professor b: well phd a: ok . so we would then use that one , too , grad e: so phd f: oh , ok . professor b: this is kind of central . phd a: or ? professor b: you know , it 's so i but i would i 'd pick that one . it 'll be less good for some people than for other , but i i 'd like to see it on the same exact same data set that that we did the other thing on . grad e: actually i sh actually should 've picked a different one , professor b: right ? grad e: because that could be why the pda is worse . because it 's further away from most of the people reading digits . phd d: it 's further away . yeah . yeah . professor b: that 's probably one of the reasons . postdoc c: hmm . mm - hmm . phd a: well , yeah . you could look at , i guess , that pzm or something . grad e: yep . professor b: but the other is , it 's very , uh i mean , even though there 's i 'm sure the f f the the sri , uh , front - end has some kind of pre - emphasis , it 's it 's , uh still , th it 's picking up lots of low - frequency energy . phd f: mm - hmm . professor b: so , even discriminating against it , i 'm sure some of it 's getting through . um . but , yeah , you 're right . prob - a part of it is just the distance . phd a: and are n't these pretty bad microphones ? grad e: yep . phd a: i mean professor b: well , they 're bad . but , i mean , if you listen to it , it sounds ok . you know ? u yeah . grad e: yeah . when you listen to it , uh , the pzm and the pda yeah , th the pda has higher sound floor but not by a lot . it 's really pretty uh , pretty much the same . phd a: i just remember you saying you got them to be cheap on purpose . cheap in terms of their quality . so . professor b: well , they 're twenty - five cents or so . grad e: th - we wanted them to be to be typical of what would be in a pda . professor b: yeah . phd a: mm - hmm . grad e: so they are they 're not the pzm three hundred dollar type . they 're the twenty - five cent , professor b: yeah . grad e: buy them in packs of thousand type . phd a: i see . professor b: but , i mean , the thing is people use those little mikes for everything because they 're really not bad . grad e: everything . phd a: mm - hmm . professor b: i mean , if you 're not doing something ridiculous like feeding it to a speech recognizer , they they they you know , you can hear the sou hear the sounds just fine . phd a: right . professor b: you know , it 's they i mean , i it 's more or less the same principles as these other mikes are built under , it 's just that there 's less quality control . they just , you know , churn them out and do n't check them . um . so . so that was yeah . so that was i interesting result . so like i said , the front - end guys are very much interested in in this is as as well and phd f: so so , but where is this now ? i mean , what 's where do we go from here ? grad e: yeah . that was gon na be my question . phd f: i mean , we so we have a we have a a system that works pretty well but it 's not , you know , the system that people here are used to using to working with . professor b: well , i think what we wan na do is we want to eh , phd f: so what what do we do now ? professor b: and we 've talked about this in other contexts we want to have the ability to feed it different features . phd f: mm - hmm . professor b: and then , um , from the point of view of the front - end research , it would be s uh , substituting for htk . phd f: ok . ok . professor b: i think that 's the key thing . and then if we can feed it different features , then we can try all the different things that we 're trying there . phd f: ok . alright . professor b: and then , um , uh , also dave is is thinking about using the data in different ways , uh , to um , uh , explicitly work on reverberation phd f: mm - hmm . professor b: starting with some techniques that some other people have found somewhat useful , and yeah . phd f: ok . so so the key thing that 's missing here is basically the ability to feed , you know , other features i into the recognizer professor b: right . phd f: and also then to train the system . professor b: right . phd f: ok . and , uh , es i do n't know when chuck will be back but that 's exactly what he he 's gon na professor b: h h he 's he 's sort of back , but he drove for fourteen hours an and was n't gon na make it in today . phd f: oh , ok . so , i think that 's one of the things that he said he would be working on . um . grad e: yeah . phd f: just sort of t to make sure that we can do that professor b: yeah . phd f: and um . professor b: right . phd f: it 's uh , i mean , the the front - end is f i tha that 's in the sri recognizer is very nice in that it does a lot of things on the fly but it unfortunately is not designed and , um like the , uh , icsi system is , where you can feed it from a pipeline of of the command . so , the what that means probably for the foreseeable future is that you have to , uh , dump out , um you know , if you want to use some new features , you have to dump them into individual files and give those files to the recognizer . grad e: we do we tend to do that anyway . phd f: ok . grad e: oh . so , although you you can pipe it as well , we tend to do it that way because that way you can concentrate on one block and not keep re - doing it over and over . phd f: oh , ok . professor b: yeah . phd f: alright . professor b: yeah . so i 've i grad e: so tha that 's exactly what the p - file is for . professor b: yeah . phd f: yeah , the the the cumbersome thing is is , um is that you actually have to dump out little little files . phd a: uh phd f: so for each segment that you want to recognize you have to dump out a separate file . grad e: uh - huh . phd f: just like i th like th as if there were these waveform segments , but instead you have sort of feature file segments . but , you know so . professor b: cool . ok . so the s the the next thing we had on the agenda was something about alignments ? phd a: oh . yes , we have i do n't know , did you wan na talk about it , or ? i can give a i was just telling this to jane and and w we we were able to get some definite improvement on the forced alignments by looking at them first and then realizing the kinds of errors that were occurring and um , some of the errors occurring very frequently are just things like the first word being moved to as early as possible in the recognition , which is a um , i think was both a a pruning problem and possibly a problem with needing constraints on word locations . and so we tried both of these st things . we tried saying i do n't know , i got this whacky idea that just from looking at the data , that when people talk their words are usually chunked together . it 's not that they say one word and then there 's a bunch of words together . they 're { comment } might say one word and then another word far away if they were doing just backchannels ? but in general , if there 's , like , five or six words and one word 's far away from it , that 's probably wrong on average . so , um and then also , ca the pruning , of course , was too too severe . phd f: so that 's actually interesting . the pruning was the same value that we used for recognition . and we had lowered that we had used tighter pruning after liz ran some experiments showing that , you know , it runs slower and there 's no real difference in phd a: actually it was better with slightly better or about th grad e: no gain . phd a: it was the same with tighter pruning . phd f: right . so for free recognition , this the lower pruning value is better . phd a: it 's probably cuz the recognition 's just bad en at a point where it 's bad enough that that you do n't lose anything . phd f: you correct . right . um , but it turned out for for to get accurate alignments it was really important to open up the pruning significantly . phd a: right . professor b: hmm . phd f: um because otherwise it would sort of do greedy alignment , um , in regions where there was no real speech yet from the foreground speaker . professor b: mm - hmm . phd f: um , so that was one big factor that helped improve things and then the other thing was that , you know , as liz said the we f enforce the fact that , uh , the foreground speech has to be continuous . it can not be you can not have a background speech hypothesis in the middle of the foreground speech . you can only have background speech at the beginning and the end . phd a: yeah . i mean , yeah , it is n't always true , and i think what we really want is some clever way to do this , where , um , you know , from the data or from maybe some hand - corrected alignments from transcribers that things like words that do occur just by themselves a alone , like backchannels or something that we did allow to have background speech around it phd d: yeah . phd a: those would be able to do that , postdoc c: sorry . phd a: but the rest would be constrained . so , i think we have a version that 's pretty good for the native speakers . i do n't know yet about the non - native speakers . and , um , we basically also made noise models for the different sort of grouped some of the mouth noises together . um , so , and then there 's a background speech model . and we also there was some neat or , interesting cases , like there 's one meeting where , um , jose 's giving a presentation and he 's talking about , um , the word `` mixed signal `` and someone did n't understand , uh , that you were saying `` mixed `` i think , morgan . and so your speech - ch was s saying something about mixed signal . phd h: yeah , yeah . phd a: and the next turn was a lot of people saying `` mixed `` , like `` he means mixed signal `` or `` i think it 's mixed `` . and the word `` mixed `` in this segment occurs , like , a bunch of times . phd h: sh phd a: and chuck 's on the lapel here , and he also says `` mixed `` but it 's at the last one , and of course the aligner th aligns it everywhere else to everybody else 's `` mixed `` , phd h: yeah . phd a: cuz there 's no adaptation yet . so there 's i think there 's some issues about u we probably want to adapt at least the foreground speaker . but , i guess andreas tried adapting both the foreground and a background generic speaker , and that 's actually a little bit of a f funky model . like , it gives you some weird alignments , just because often the background speakers match better to the foreground than the foreground speaker . phd f: oh phd d: yeah . phd a: so there 's some things there , phd h: oh . phd a: especially when you get lots of the same words , uh , occurring in the phd f: well , the i i think you can do better by uh , cloning so we have a reject phone . and you and what we wanted to try with you know , once we have this paper written and have a little more time , uh , t cloning that reject model and then one copy of it would be adapted to the foreground speaker to capture the rejects in the foreground , like fragments and stuff , and the other copy would be adapted to the background speaker . phd a: right . i mean , in general we actually phd f: and phd a: right now the words like partial words are reject models and you normally allow those to match to any word . phd f: mm - hmm . phd a: but then the background speech was also a reject model , and so this constraint of not allowing rejects in between you know , it needs to differentiate between the two . so just sort of working through a bunch of debugging kinds of issues . phd f: right . phd a: and another one is turns , like people starting with `` well i think `` and someone else is `` well how about `` . so the word `` well `` is in this in this segment multiple times , and as soon as it occurs usually the aligner will try to align it to the first person who says it . but then that constraint of sort of uh , proximity constraint will push it over to the person who really said it in general . grad e: is the proximity constraint a hard constraint , or did you do some sort of probabilistic weighting distance , or ? phd f: we we did n't phd a: right now it 's a kluge . phd f: no . we w ok . we it 's straightforward to actually just have a a penalty that does n't completely disallows it but discourages it . but , um , we just did n't have time to play with , you know , tuning yet another yet another parameter . grad e: the ve level . yeah . phd a: yeah . phd f: and really the reason we ca n't do it is just that we do n't have a we do n't have ground truth for these . so , we would need a hand - marked , um , word - level alignments or at least sort of the boundaries of the speech betw you know , between the speakers . um , and then use that as a reference and tune the parameters of the of the model , uh , to op to get the best performance . phd a: yeah . professor b: g given i i mean , i wa i wa i was gon na ask you anyway , uh , how you assessed that things were better . phd f: mm - hmm . phd a: i looked at them . i spent two days um , in waves professor b: ok . phd a: oh , it was painful because the thing is , you know the alignments share a lot in common , so and you 're yo you 're looking at these segments where there 's a lot of speech . i mean , a lot of them have a lot of words . not by every speaker professor b: yeah . phd a: but by some speaker there 's a lot of words . no , not professor b: yeah . phd a: i mean that if you look at the individual segments from just one person you do n't see a lot of words , phd h: ju professor b: yeah . phd a: but altogether you 'll see a lot of words up there . professor b: yeah . phd f: mm - hmm . phd d: yeah . phd a: and so the reject is also mapping and pauses so i looked at them all in waves and just lined up all the alignments , and , at first it sort of looked like a mess and then the more i looked at it , i thought `` ok , well it 's moving these words leftward and `` you know , it was n't that bad . it was just doing certain things wrong . so but , i do n't , you know , have time to l { comment } to look at all of them and it would be really useful to have , like , a a transcriber who could use waves , um , just mark , like , the beginning and end of the foreground speaker 's real words like , the beginning of the first word , the end of the last word and then we could , you know , do some adjustments . postdoc c: yeah . i ok . i have to ask you something , is i does it have to be waves ? because if we could benefit from what you did , incorporate that into the present transcripts , { comment } that would help . phd f: no . postdoc c: and then , um , the other thing is , i believe that i did hand so . one of these transcripts was gone over by a transcriber and then i hand - marked it myself so that we do have , uh , the beginning and ending of individual utterances . um , i did n't do it word level , phd f: mm - hmm . postdoc c: but but in terms phd a: mm - hmm . postdoc c: so i so for for one of the n s a groups . and also i went back to the original one that i first transcribed and and did it w uh , w uh , utterance by utterance for that particular one . so i think you do have if that 's a sufficient unit , i think that you do have hand - marking for that . but it 'd be wonderful to be able to benefit from your waves stuff . phd a: mm - hmm . phd f: we do n't care what what tool you use . phd a: yeah . i mean , if if you can , um if you wan na postdoc c: ok . i used it in transcriber phd f: u uh postdoc c: and it 's it 's in the phd a: well , jane and i were just in terms of the tool , talking about this . i guess sue had had some reactions . you know , interface - wise if you 're looking at speech , you wan na be able to know really where the words are . and so , we can give you some examples of sort of what this output looks like , postdoc c: yeah , that 's right . middle of the word , or phd a: um , and see if you can in maybe incorporate it into the transcriber tool some way , or postdoc c: well , i th i 'm thinking just ch e e incorporating it into the representation . phd a: um . postdoc c: i mean , if it 's if it 's phd a: you mean like yeah , word start insights . postdoc c: if you have start points , if you have , like , time tags , phd a: right . postdoc c: which is what i assume . is n't that what what you ? well , see , adam would be phd f: yeah , whatever you use . phd a: yeah . phd f: i mean , we convert it to this format that the , um , nist scoring tool unders uh , ctm . conversation time - marked file . and and then that 's the that 's what the grad e: i think transcriber , uh , outputs ctm . postdoc c: if it ? ok . phd a: yeah . postdoc c: so you would know this more than i would . grad e: i think so . phd a: so , i mean postdoc c: it seems like she if she 's g if she 's moving time marks around , phd f: right . postdoc c: since our representation in transcriber uses time marks , it seems like there should be some way of of using that benefitting from that . grad e: right . phd a: yeah , it wou the advantage would just be that when you brought up a bin you would be able if you were zoomed in enough in transcriber to see all the words , professor b: mm - hmm . phd a: you would be able to , like , have the words sort of located in time , if you wanted to do that . professor b: so so if we e e even just had a a it sounds like w we we almost do . phd a: so . professor b: uh , if we we have two . postdoc c: we have two . professor b: yeah . just ha uh , trying out the alignment procedure that you have on that phd a: mm - hmm . professor b: you could actually get something , um uh , uh , get an objective measure . uh phd f: mm - hmm . phd a: you mean on on the hand - marked , um so we we only r hav i only looked at actually alignments from one meeting that we chose , professor b: yeah . phd a: i think mr four , just randomly , um and phd f: actually , not randomly . phd a: not randomly phd f: we knew we knew that it had these insertion errors from phd a: it had sort of average recognition performance in a bunch of speakers phd f: yeah . yeah . phd a: and it was a meeting recorder meeting . um . but , yeah , we should try to use what you have . i did re - run recognition on your new version of mr one . postdoc c: oh , good . phd a: i i mean the the one with dan ellis in it and eric . postdoc c: good ! uh - huh . yeah , exactly . yeah . yeah . grad g: i do n't think that was the new version . phd a: um that yeah , actually it was n't the new new , it was the medium new . postdoc c: ok . phd a: but but we would we should do the the latest version . postdoc c: ok . grad g: yeah . phd a: it was the one from last week . grad g: you did you adjust the the utterance times , um , for each channel ? postdoc c: yes . yes , i did . and furthermore , i found that there were a certain number where not not a lot , but several times i actually moved an utterance from adam 's channel to dan 's or from dan 's to adam 's . so there was some speaker identif and the reason was because i transcribed that at a point before uh , before we had the multiple audio available f so i could n't switch between the audio . i i transcribed it off of the mixed channel entirely , which meant in overlaps , i was at a at a terrific disadvantage . phd a: right . right . postdoc c: in addition it was before the channelized , uh , possibility was there . and finally i did it using the speakers of my , um of you know , off the cpu on my on my machine cuz i did n't have a headphone . phd a: right . postdoc c: so it @ @ , like , i mean yeah , i i mean , i in retrospect it would 've been good to ha have got i should 've gotten a headphone . but in any case , um , thi this is this was transcribed in a in a , uh , less optimal way than than the ones that came after it , and i was able to you know , an and this meant that there were some speaker identif identifications which were changes . grad g: well , i know there were some speaker labelling problems , um , after interruptions . postdoc c: yeah . fixed that . grad g: is that what you 're referring to ? i mean , cuz there 's this one instance when , for example , you 're running down the stairs . postdoc c: oh , well grad g: i remember this meeting really well . phd d: yeah . phd a: don don has had he knows he can just read it like a play . grad g: right . it 's a yeah , i 've i 've i 'm very well acquainted with this meeting . phd d: yeah . grad g: yeah , i can s phd a: `` and then she said , and then he said . `` grad g: yeah , i know it by heart . so , um , there 's one point when you 're running down the stairs . postdoc c: uh - oh . grad g: right ? and , like , there 's an interruption . you interrupt somebody , but then there 's no line after that . for example , there 's no speaker identification after that line . postdoc c: uh - huh . grad g: is that what you 're talking about ? or were there mislabellings as far as , like , the a adam was ? postdoc c: that was fixed , um , before i i i i think i i think i understood that pretty grad g: yeah . cuz i thought i let you know about that . postdoc c: thank you for mentioning . yeah , no , tha that that i think went away a couple of versions ago , grad g: yeah . ok . postdoc c: but it 's good to know . grad g: but you 're actually saying that certain , uh , speakers were mis mis - identified . postdoc c: yeah . so , with under um , uh , listening to the mixed channel , there were times when , as surprising as that is , i got adam 's voice confused with dan 's and vice versa grad g: ok . postdoc c: not for long utterances , grad g: ok . phd a: yeah . postdoc c: but jus just a couple of places , professor b: mm - hmm . postdoc c: and embedde embedded in overlaps . the other thing that was w interesting to me was that i picked up a lot of , um , backchannels which were hidden in the mixed signal , phd a: right . postdoc c: which , you know , i mean , you c not not too surprising . but the other thing that i i had n't thought about this , but i thou i wanted to raise this when you were uh , with respect to also a strategy which might help with the alignments potentially , but that 's when i was looking at these backchannels , they were turning up usually very often in w well , i wo n't say `` usually `` but anyway , very often , i picked them up in a channel w which was the person who had asked a question . s so , like , someone says `` an and have you done the so - and - so ? `` and then there would be backchannels , but it would be the person who asked the question . other people were n't really doing much backchannelling . and , you know , sometimes you have the yeah , uh - huh . phd a: well , that 's interesting . yeah . postdoc c: i mean , i it would n't be perfect , but but it does seem more natural to give a backchannel when when you 're somehow involved in the topic , phd a: no , that 's really interesting . professor b: mm - hmm . postdoc c: and the most natural way is for you to have initiated the topic by asking a question . phd f: well , phd a: that 's interesting . phd f: i think no . i think it 's actually i think what 's going on is backchannelling is something that happens in two - party conversations . postdoc c: mm - hmm . phd f: and if you ask someone a question , you essentially initiating a little two - party conversation . postdoc c: yeah . phd a: well , actu yeah , when we looked at this postdoc c: exactly . phd f: so then you 're so and then you 're expected to backchannel because the person is addressing you directly and not everybody . postdoc c: exactly . exactly my point . an - and so this is the expectation thing that uh , uh , phd f: yeah . yeah . phd a: mm - hmm . phd f: right . postdoc c: just the dyadic phd f: right . postdoc c: but in addition , you know , if someone has done this analysis himself and is n't involved in the dyad , but they might also give backchannels to verify what what the answer is that this that the the answerer 's given professor b: h phd a: right . professor b: i tell you , i say i say `` uh - huh `` a lot , phd a: it 's postdoc c: there you go . phd a: well , but it 's interesting cuz , uh professor b: while people are talking to each other . phd a: but there are fewer i think there are fewer `` uh - huhs `` . postdoc c: there you go . yeah . yeah . phd a: i mean , just from we were looking at word frequency lists to try to find the cases that we would allow to have the reject words in between in doing the alignment . you know the ones we would n't constrain to be next to the other words . postdoc c: oh , yeah . phd a: and `` uh - huh `` is not as frequent as it sort of would be in switchboard , if you looked at just a word frequency list of one - word short utterances . and `` yeah `` is way up there , but not `` uh - huh `` . and so i was thinking thi it 's not like you 're being encouraged by everybody else to keep talking in the meeting . and uh , that 's all , i i 'll stop there , cuz i i think what you say makes a lot of sense . postdoc c: well , that 's right . and that would phd a: but it was sort of postdoc c: well , an and what you say is the is the re uh , o other side of this , which is that , you know , so th there are lots of channels where you do n't have these backchannels , w when a question has been asked and and these phd a: right . there 's just probably less backchannelling in general , postdoc c: mm - hmm . so that 's good news , really . phd a: even if you consider every other person altogether one person in the meeting , but we 'll find out anyway . we were i guess the other thing we 're we 're i should say is that we 're gon na , um try compare this type of overlap analysis to switchboard , where phd f: and phd a: and callhome , where we have both sides , so that we can try to answer this question of , you know , is there really more overlap in meetings or is it just because we do n't have the other channel in switchboard professor b: mm - hmm . grad e: mm - hmm . phd a: and we do n't know what people are doing . try to create a paper out of that . professor b: yeah . i mean , y y you folks have probably already told me , but were were you intending to do a eurospeech submission , or ? phd a: um , you mean the one due tomorrow ? professor b: yeah . phd a: yeah . well , we 're still , like , writing the scripts for doing the research , and we will yes , we 're gon na try . postdoc c: mm - hmm . phd a: and i was telling don , do not take this as an example of how people should work . professor b: do as i say , grad g: that 's r phd a: so , { comment } we will try . professor b: do n't do as i do . yeah . phd a: it 'll probably be a little late , grad e: well phd a: but i 'm gon na try it . grad e: it is different . in previous years , eurospeech only had the abstract due by now , not the full paper . phd a: right . grad g: right . grad e: and so all our timing was off . i 've given up on trying to do digits . i just do n't think that what i have so far makes a eurospeech paper . phd a: well , i 'm no we may be in the same position , and i figured we 'll try , because that 'll at least get us to the point where we have we have this really nice database format that andreas and i were working out that it it 's not very fancy . it 's just a ascii line by line format , but it does give you information phd f: it 's the it 's the spurt format . phd a: it yeah , we 're calling these `` spurts `` after chafe . i was trying to find what 's a word for a continuous region with pauses around it ? postdoc c: hmm . professor b: yeah . i know that th the telecom people use use `` spurt `` for that . postdoc c: good . phd a: they do ? oh ! professor b: yes . phd f: oh . phd a: oh . professor b: and that 's i mean , i i was using that for a while when i was doing the rate of speech stuff , phd a: i would jus professor b: because i because i looked up in some books and i found ok , i wan na find a spurt in which phd a: ah , right ! it 's just , like , defined by the acoustics . professor b: and an because cuz it 's another question about how many pauses they put in between them . grad e: horrible . phd a: right . professor b: but how fast do they do the words within the spurt ? phd a: right . professor b: yeah . phd a: well , that 's what we were calling spurt , grad e: it 's gon na grad g: you know `` burst `` also ? grad e: burst . grad g: is n't `` burst `` is used also ? phd a: so grad e: spurt has the horrible name overloading with other with hardware at icsi . professor b: here . just very locally , yeah . phd a: well , well , chafe had this wor i think it was chafe , or somebody had a the word `` spurt `` originally , professor b: but but that just phd h: here @ @ phd a: and so i but tha that 's good to know . postdoc c: actually phd a: was thi it 's chafe ? postdoc c: well , see , i know s sue wrote about spurts of development . phd f: so maybe we should talk phd a: maybe it was sue ? y postdoc c: but , in any case , i think it 's a good term , phd a: so we have spurts and we have spurt - ify dot shell and spurt - ify professor b: yeah . postdoc c: and , uh grad e: hmm ! professor b: yeah . postdoc c: and ma maybe maybe chafe did . phd f: uh . phd a: and then it 's got all it 's a verb now . postdoc c: i know i know ch - chafe dealt with phd f: so s grad g: that 's cool . phd f: w uh , w postdoc c: chafe speaks about intonation units . phd a: yes . right . postdoc c: but maybe he speaks about spurts as well phd f: we postdoc c: and i just do n't know . yeah , go ahead . grad e: i 've heard `` burst `` also . phd f: so what we 're doing uh , this this is just maybe someone has s some some ideas about how to do it better , grad g: mmm . phd f: but we so we 're taking these , uh , alignments from the individual channels . we 're from each alignment we 're producing , uh , one of these ctm files , postdoc c: great . phd f: which essentially has it 's just a linear sequence of words with the begin times for every word and the duration . phd a: it looks like a waves label file almost . right ? phd f: and and and of course phd a: it 's just phd f: right . but it has one the first column has the meeting name , so it could actually contain several meetings . um . and the second column is the channel . third column is the , um , start times of the words and the fourth column is the duration of the words . and then we 're , um ok . then we have a messy alignment process where we actually insert into the sequence of words the , uh , tags for , like , where where sentence ends of sentence , question marks , um , various other things . phd a: yeah . these are things that we had don phd f: uh . phd a: so , don sort of , um , propagated the punctuation from the original transcriber phd f: right . phd a: so whether it was , like , question mark or period or , um , you know , comma and things like that , and we kept the and disfluency dashes uh , kept those in because we sort of wan na know where those are relative to the spurt overlaps phd f: mm - hmm . right . phd a: sp overlaps , phd f: so so those are actually sort of retro - fitted into the time alignment . phd a: or phd f: and then we merge all the alignments from the various channels and we sort them by time . and then there 's a then there 's a process where you now determine the spurts . that is actually , no , you do that before you merge the various channels . so you you id identify by some criterion , which is pause length you identify the beginnings and ends of these spurts , and you put another set of tags in there to keep those straight . professor b: mm - hmm . phd f: and then you merge everything in terms of , you know , linearizing the sequence based on the time marks . and then you extract the individual channels again , but this time you know where the other people start and end talking you know , where their spurts start and end . and so you extract the individual channels , uh , one sp spurt by spurt as it were . um , and inside the words or between the words you now have begin and end tags for overlaps . so , you you basically have everything sort of lined up and in a form where you can look at the individual speakers and how their speech relates to the other speakers ' speech . grad e: right . phd a: uh , i mean , i think that 's actually really u useful also phd f: and phd a: because even if you were n't studying overlaps , if you wan na get a transcription for the far - field mikes , how are you gon na know which words from which speakers occurred at which times relative to each other ? you have to be able to get a transcript like like this anyway , just for doing far - field recognition . so , you know , it 's it 's sort of phd f: yeah . phd a: i thi it 's just an issue we have n't dealt with before , how you time - align things that are overlapping anyway . postdoc c: that 's wonderful . phd f: so phd a: i mean , i i never thought about it before , grad e: well phd f: and and we phd a: but grad e: y yes . phd f: in grad e: i mean , s when i came up with the original data suggested data format based on the transcription graph , there 's capability of doing that sort of thing in there . phd a: right . but you ca n't get it directly from the transcription . postdoc c: mm - hmm . yeah , that 's right . phd f: right . well , this is this is just phd a: yeah , this is like a poor man 's ver formatting version . but it 's , you know it 's clean , it 's just not fancy . grad e: right . phd a: um . phd f: well , there 's lots of little things . it 's like there 're twelve different scripts which you run and then at the end you have what you want . but , um , at the very last stage we throw away the actual time information . all we care about is whether that there 's a certain word was overlapped by someone else 's word . so you sort of at that point , you discretize things into just having overlap or no overlap . because we figure that 's about the level of analysis that we want to do for this paper . grad e: mm - hmm . phd f: but if you wanted to do a more fine - grained analysis and say , you know , how far into the word is the overlap , you could do that . phd a: yeah . phd f: it 's just it 'll just require more phd a: just sort of huge . phd f: you know , slightly different postdoc c: what 's interesting is it 's exactly what , um , i in discussing with , um , sue about this , phd a: yeah . postdoc c: um , she , um , i i i indicated that that you know , that 's very important for overlap analysis . phd a: yeah . it 's it 's nice to know , phd f: right . phd a: and also i think as a human , like , i do n't always hear these in the actual order that they occur . so i can have two foreground speakers , you know , morgan an and um , adam and jane could all be talking , and i could align each of them to be starting their utterance at the correct time , and then look where they are relative to each other , and that 's not really what i heard . postdoc c: and that 's another thing she said . phd a: cuz it 's just hard to do . postdoc c: this is this is bever 's bever 's effect , phd a: y yeah . postdoc c: when where in psy ps psycho - linguistics you have these experiments where people have perceptual biases a as to what they hear , phd a: it 's sort of yeah , you sort of move things around until you get to a low information point postdoc c: that that not the best phd a: and yo then you can bring in the other person . so it 's actually not even possible , i think , for any person to listen to a mixed signal , even equalize , and make sure that they have all the words in the right order . so , i guess , we 'll try to write this eurospeech paper . postdoc c: mm - hmm . superb . phd a: i mean , we will write it . whether they accept it late or not , i do n't know . um , and the good thing is that we have it 's sort of a beginning of what don can use to link the prosodic features from each file to each other . phd f: yeah . professor b: yeah . that 's the good thing about these pape phd a: so . i you know , might as well . phd f: plus , mayb phd h: hmm ? phd a: we - i ju otherwise we wo n't get the work done { comment } on our deadline . phd f: i do n't know , m professor b: yeah . phd f: i mean , u u jane likes to look at data . maybe , you know , you could you could look at this format and see if you find anything interesting . professor b: yeah . phd f: i do n't know . phd a: yeah . professor b: no , it 's that 's the good thing about these pape paper deadlines and , uh , you know , class projects , and and things like that , postdoc c: well , what i 'm thinking is phd f: yeah . postdoc c: yeah . phd a: right . phd f: mm - hmm . postdoc c: well , my phd f: well th th the other thing that that that yo that you usually do n't tell your graduate students is that these deadlines are actually not that , um , you know , strictly enforced , professor b: because you you really get g phd a: forces you to do the work . postdoc c: yeah . professor b: yeah . phd a: exactly . grad e: strict . phd f: because the professor b: oh , now it 's out in the public , this this this secret information . phd f: because phd a: right . professor b: yeah . postdoc c: i think we can ha phd f: bec b nah phd a: so grad e: no . professor b: no . postdoc c: nah . phd f: i because these the conference organizers actually have an interest in getting lots of submissions . phd a: right . grad e: right . phd f: i mean , a a monetary interest . professor b: yeah . phd f: so um . professor b: th - that 's that 's true . postdoc c: and good ones , good ones , which sometimes means a little extra time . phd f: and good submission professor b: that 's phd f: right . professor b: that 's true . phd f: well that 's another issue , professor b: by th by the way , this is totally unfair , you may you may feel , phd f: but professor b: but the the , uh the morning meeting folks actually have an an extra month or so . phd f: mm - hmm . phd d: yep . grad e: yep . the aurora there 's a special aurora phd a: uh phd f: when professor b: there 's a special aurora session phd a: oh . professor b: and the aurora pe people involved in aurora have till ma - uh , early may or something to turn in their paper . phd f: mmm . phd a: oh . phd f: mmm . phd a: oh , well maybe we 'll submit to s { comment } actually phd f: well , then you can just maybe you can submit the digits paper on e for the aurora session . phd h: yeah . phd a: yeah . phd d: yeah . grad e: oh , i could ! phd a: yeah . professor b: i if it w grad e: i could submit that to aurora . professor b: well grad e: that would be pretty pretty phd f: yeah . professor b: i it has phd a: yeah . professor b:  grad e: s that would n't work . professor b: no , it would n't work . grad e: it 's not aurora . professor b: it 's it 's not the aurora i mean , it it 's it 's actually the aurora task . phd a: maybe they 'll get s grad e: aurora 's very specific . professor b: it phd a: well , maybe it wo n't be after this deadline extension . phd f: but but the people i mean , a a paper that is not on aurora would probably be more interesting at that point phd a: maybe they 'll phd f: because everybody 's so sick and tired of the aurora task . phd d: yeah . grad e: oh , i thought you meant this was just the digits section . i did n't know you meant it was aurora digits . professor b: yeah . phd f: well , no . if you if you have it 's to if you discuss some relation to the aurora task , like if you use the same professor b: this is not the aurora task . so they just do a little grep for phd a: do uh , d d do not do not we are not setting a good example . phd f: um . well , a relation other than negation , maybe , phd a: this is not a phd f: um . so . phd a: anyway . phd f: i do n't know . phd a: but the good thing is this does grad e: well , i i do n't know . i mean , you could you could do a paper on what 's wrong with the aurora task by comparing it to other ways of doing it . phd f: how well does an aurora system do on on you know , on digits collected in a in this environment ? phd h:  grad e: different way . yeah . phd f: yeah . professor b: maybe . phd f: maybe . grad e: pretty hokey . professor b: i think it 's a littl little far - fetched . nah , i mean , the thing is aurora 's pretty closed community . grad e: yep . professor b: i mean , you know , the people who were involved in the the only people who are allowed to test on that are people who who made it above a certain threshold in the first round , phd f: mm - hmm . grad e: it 's very specific . professor b: uh w in ninety - nine and it 's it 's sort of a it 's not like a phd f: well , that 's maybe why they do n't f know that they have a crummy system . i mean , a crummy back - end . no , i mean i mean , seriously , if you if you have a very no , i 'm sorry . phd a: uh , { comment } `` beep `` `` bee `` grad e: i mean , th phd f: no . i did n't mean anybody any particular system . i meant this h t k back - end . professor b: oh , you do n't like htk ? phd f: if they phd h: yeah . phd f: i do n't h i do n't have any stock in htk or entropic or anything . professor b: no . i mean , this it it 's the htk that is trained on a very limited amount of data . grad e: it 's d it 's very specific . phd f: right . professor b: yeah . phd f: but so , if you but maybe you should , you know , consider more using more data , or i mean professor b: oh , yeah . i i really think that that 's true . and they i i phd f: if yo if you sort of hermetically stay within one task and do n't look left and right , then you 're gon na grad e: but they they had professor b: i but grad e: they had something very specific in mind when they designed it . right ? professor b: well , u i phd f: right . grad e: and so so you can you can argue about maybe that was n't the right thing to do , but , you know , they they they had something specific . professor b: but , one of the reasons i have chuck 's messing around with with the back - end that you 're not supposed to touch i mean , for the evaluations , yes , we 'll run a version that has n't been touched . phd f: mm - hmm . mm - hmm . professor b: but , uh , one of the reasons i have him messing around with that , because i think it 's sort of an open question that we do n't know the answer to . people always say very glibly that i if you s show improvement on a bad system , that does n't mean anything , cuz it may not be show uh , because , you know , it does n't tell you anything about the good system . phd f: mm - hmm . professor b: and i i 've always sort of felt that that depends . you know , that if some peopl if you 're actually are getting at something that has some conceptual substance to it , it will port . phd f: mm - hmm . professor b: and in fact , most methods that people now use were originally tried with something that was not their absolute best system at some level . but of course , sometimes it does n't , uh , port . so i think that 's that 's an interesting question . if we 're getting three percent error on , uh , u uh , english , uh , nati native speakers , um , using the aurora system , and we do some improvements and bring it from three to two , do those same improvements bring , uh , th you know , the sri system from one point three to you know , to point eight ? phd f: hmm . mm - hmm . grad e: zero . professor b: well . you know , so that 's that 's something we can test . phd f: mmm . right . professor b: so . anyway . phd f: ok . professor b: i think we 've we 've covered that one up extremely well . postdoc c: mm - hmm . phd f: whew ! professor b: ok . so , um yeah . so tha so we 'll you know , maybe you guys 'll have have one . uh , you you and , uh and dan have have a paper that that 's going in . phd d: yeah . professor b: you know , that 's that 's pretty solid , on the segmentation stuff . phd d: yeah . yeah . i will send you the the final version , professor b: yeah . and the aurora folks here will will definitely get something in on aurora , phd d: which is not phd f: actually this this , um so , there 's another paper . professor b: so . phd f: it 's a eurospeech paper but not related to meetings . but it 's on digits . so , um , uh , a colleague at sri developed a improved version of mmie training . professor b: uh - huh . phd f: and he tested it mostly on digits because it 's sort of a you know , it does n't take weeks to train it . professor b: right . phd f: um . and got some very impressive results , um , with , you know , discriminative , uh , gaussian training . um , you know , like , um , error rates go from i do n't know , in very noisy environment , like from , uh , uh i for now i ok , now i have the order of magnit i 'm not sure about the order of magnitude . was it like from ten percent to eight percent or from e e you know , point you know , from one percent to point eight percent ? professor b: h i it got it got better . phd f: i mean , it 's a professor b: yeah , yeah . phd d: yeah . phd f: it got better . that 's the important thing . grad e: hey , that 's the same percent relative , phd f: yeah . but it 's grad e: so phd f: yeah . right . professor b: yeah . phd f: it 's , uh , something in professor b: yeah . grad e: twenty percent relative gain . phd f: right . professor b: yeah . phd f: yeah . professor b: yeah . um , let 's see . i think the only thing we had left was unless somebody else well , there 's a couple things . uh , one is anything that , um , anybody has to say about saturday ? anything we should do in prep for saturday ? um i guess everybody knows about i mean , u um , mari was asking was trying to come up with something like an agenda and we 're sort of fitting around people 's times a bit . but , um , clearly when we actually get here we 'll move things around this , as we need to , but so you ca n't absolutely count on it . phd d: ok . professor b: but but , uh phd d: yeah . phd a: are we meeting in here probably or ? ok . professor b: yeah . that was my thought . phd a: yeah . professor b: i think this is phd f: are we recording it ? phd a: we wo n't have enough microphones , professor b:  phd a: but professor b: u no . i i had n't in intended to . phd a: there 's no way . professor b: we won we wan na i mean , they 're there 's gon na be , uh , jeff , katrin , mari and two students . phd f: ok . professor b: so there 's five from there . grad e: and brian . professor b: and brian 's coming , phd f: but you know th professor b: so that 's six . grad e: and plus all of us . phd f: mm - hmm . professor b: uh phd f: can use the oprah mike . phd a: depends how fast you can throw it . grad e: it seems like too many too much coming and going . phd a: it 's just yeah . phd f: mm - hmm . phd a: we do n't even have enough channel professor b: well phd f: because it would be a different kind of meeting , phd d: yeah . phd f: that 's what i 'm professor b: well phd f: but phd h: yeah . professor b: i had n't really thought of it , phd f: maybe just maybe not the whole day professor b: but phd f: but just , you know , maybe some i mean , professor b: maybe part of it . phd f: part of it ? professor b: maybe part of it . grad e: make everyone read digits . professor b: at the same time . phd a: at the same time . grad e: at the same time . phd f: please . phd h:  professor b: yeah . phd a: we c professor b: i do n't know . phd a: that 's their initiation into our professor b: any phd a: w grad e: into our our our cult . phd a: yeah , our yeah , our phd f: maybe the sections that are not right afte you know , after lunch when everybody 's still munching and phd a: so can you send out a schedule once you know it , jus ? professor b: ok . well phd a: is is there a r ? professor b: ok . yeah . i guess i sent it around a little bit . phd a: there 's a res is it changed now , or ? professor b: but i had n't heard back from mari after i i u u uh , brought up the point abou about andreas 's schedule . so , um , maybe when i get back there 'll be some some mail from her . phd a: ok . professor b: so , i 'll make a postdoc c: i 'm looking forward to seeing your representation . that 'd be , uh phd a: and w we should get the two meetings from y postdoc c: i 'd like to see that . yeah . phd a: i mean , i know about the first meeting , um , but the other one that you did , the nsa one , which we had n't done cuz we were n't running recognition on it , because the non - native speaker postdoc c: mm - hmm . phd a: there were five non - native speakers . postdoc c: mm - hmm . i see . mm - hmm . phd a: but , it would be useful for the to see what we get with that one . so . postdoc c: great . ok . it 's , uh , two thousand eleven twenty - one one thousand . phd a: yeah , three . right . so postdoc c: great . i sent email when i finished the that one . phd a: n s a three , i think . postdoc c: that was sort of son yeah , that 's right . that 's right . that 's much simpler . phd a: i do n't know what they said but i know the number . professor b: th - that part 's definitely gon na confuse somebody who looks at these later . phd f: right . professor b: i mean , this is we we 're recording secret nsa meetings ? phd f: um . not the professor b: i mean , it 's phd f: yeah . postdoc c: yeah . not that nsa . phd f: uh . the th the phd a: they are hard to understand . professor b: it 's network services and applications . phd f: wait . phd a: they 're very , uh , out there . phd f: the phd a: i have no idea what they 're talking about . professor b: yeah . phd f: the , um th the other good thing about the alignments is that , um , it 's not always the machine 's fault if it does n't work . so , you can actually find , um , phd a: it 's the person 's fault . phd f: problem uh , proble phd a: it 's morgan 's fault . phd f: you can find professor b: it 's always morgan 's fault . phd f: you can find , uh , problems with with the transcripts , um , you know , grad e: oh . phd a: yeah . phd f: and go back and fix them . phd a: tha - there are some cases like where the the wrong speaker uh , these ca not a lot , but where the the wrong person the the speech is addre attached to the wrong speaker phd f: but phd a: and you can tell that when you run it . or at least you can get clues to it . postdoc c: interesting . phd a: so these are from the early transcriptions that people did on the mixed signals , like what you have . postdoc c: i guess it does w mm - hmm . it also raises the possibility of , um , using that kind of representation i mean , i do n't know , this 'd be something we 'd wan na check , { comment } but maybe using that representation for data entry and then displaying it on the channelized , uh , representation , cuz it i think that the i mean , my my preference in terms of , like , looking at the data is to see it in this kind of musical score format . phd a: mm - hmm . postdoc c: and also , s you know , sue 's preference as well . phd a: yeah , if you can get it to postdoc c: and and but , i mean , this if this is a better interface for making these kinds of , uh , you know , lo clos local changes , then that 'd be fine , too . i do n't i have no idea . i think this is something that would need to be checked . yeah . professor b: ok . th - the other thing i had actually was , i i did n't realize this till today , but , uh , this is , uh , jose 's last day . grad e: yeah . phd h: is my last my last day . phd a: oh ! postdoc c: oh . phd f: oh ! grad e: you 're not gon na be here tomorrow ? phd h: my my last meeting about meetings . grad e: oh , that 's right . tomorrow phd h: yeah . phd d: the last meeting meeting ? phd h: because , eh , i leave , eh , the next sunday . grad e: it 's off . phd a: oh . phd f: mm - hmm . phd h: i will come back to home to spain . professor b: yeah . phd a: oh . professor b: i d so i i jus phd f: mm - hmm . phd h: and i i would like to to to say thank you very much , eh , to all people in the group and at icsi , phd f: mm - hmm . grad e: yeah . it was good having you . phd f: mmm . phd a: yeah . phd h: because i i enjoyed @ @ very much , phd f: mmm . phd h: uh . and i 'm sorry by the result of overlapping , because , eh , i have n't good results , eh , yet but , eh , i i pretend { comment } to to continuing out to spain , eh , during the the following months , professor b: uh - huh . phd h: eh , because i have , eh , another ideas but , eh , i have n't enough time to to with six months it 's not enough to to to research , grad e: yep . professor b: yeah . phd h: eh , and e i i mean , if , eh , the topic is , eh , so difficult , uh , in my opinion , there is n't professor b: yeah . maybe somebody else will come along and will be , uh , interested in working on it and could start off from where you are also , you know . they 'd make use of of what you 've done . phd h: yeah . professor b: yeah . phd h: yeah . but , eh , i i will try to recommend , eh , at , eh , the spanish government but , eh , the following @ @ scholarship , eh , eh , eh , will be here more time , because eh , i in my opinion is is better , eh , for us to to spend more time here and to work more time i i in a topic . professor b: yeah , it 's a very short time . phd h: no ? but , uh professor b: yeah . yeah . grad e: yeah , six months is hard . phd h: yeah . it is . grad e: i think a year is a lot better . phd h: yeah . professor b: yeah . phd h: it 's difficult . you e you have , eh you are lucky , and you you find a solution { comment } in in in some few tim uh , months , eh ? ok . but , eh , i think it 's not , eh , common . but , eh , anyway , thank you . thank you very much . eh , i i bring the chocolate , eh , to to tear , uh , with with you , phd a: oh . postdoc c: ah . phd f: mmm . postdoc c: nice . phd h: uh . i i hope if you need , eh , something , eh , from us in the future , i i will be at spain , to you help , uh . professor b: well . grad e: great . postdoc c: great . phd a: right . professor b: thank you , jose . postdoc c: thank you . phd h: and , thank you very much . phd f: have a good trip . professor b: yeah . postdoc c: yeah . phd f: keep in touch . phd h: thank you . professor b: yeah . ok . i guess , uh , unless somebody has something else , we 'll read read our digits grad e: digits ? professor b: and we 'll get our phd d: uh . professor b: get our last bit of , uh , jose 's jose jose 's digit phd d: oops . grad e: are we gon na do them simultaneously or ? phd h: you eh professor b: uh , i 'm sorry ? phd h: ye - ye you prefer , eh , to eat , eh , chocolate , eh , at the coffee break , eh , at the ? or you prefer now , before after ? postdoc c: well , we have a time phd f: no , we prefer to keep it for ourselves . phd d: during postdoc c: well , we have a s a time time constraint . phd f: yeah , yeah . phd d: during digits . professor b: so keep it away from that end of the table . postdoc c: yeah . phd f: yeah . phd h: yeah . phd a: why is it that i can read your mind ? postdoc c: yeah . grad e: well , we 've got ta wait until after di after we take the mikes off . phd d: no , no . grad e: so are we gon na do digits simultaneously phd a: you this is our reward if we do our digi professor b: well ? yeah . postdoc c: ok . phd d: yeah . grad e: or what ? phd d: simultaneous digit chocolate task . phd h: i i think , eh , it 's enough , eh , for more peopl for more people after . professor b: we 're gon na we 're gon na do digits at the same phd a: oh . phd f: mmm ! postdoc c: that 's nice . phd h: but , eh phd f: mm - hmm . phd a: oh , thanks , jose . professor b: um . postdoc c: wow . phd h: to andreas , the idea is is good . s to eat here . professor b: well phd f: mmm . postdoc c: wow . very nice . phd f: oh . phd a: oh , wow . professor b: tha - that 's that looks great . phd f: oh , yeah . th - it does n't it wo n't leave this room . professor b: alright , so in the interest of getting to the phd a: we could do digits while other people eat . phd d: yeah . phd a: so it 's background crunching . phd d: yeah . phd h: yeah . phd f: mmm . phd a: we do n't have background chewing . postdoc c: nice . phd h: is , eh , a another acoustic event . phd d: background crunch . yeah . phd a: no , we do n't have any data with background eating . phd f: mmm . phd d: yeah . phd a: i 'm serious . you professor b: she 's she 's serious . phd a: i am serious . grad e: it 's just the rest of the digits the rest of the digits are very clean , professor b: she is serious . phd f: mmm . phd a: well ? phd h: are you ? oh , they 're clean . phd d: yeah ! grad e: um , without a lot of background noise , phd a: and it you have to write down , like , while y what you 're what ch chocolate you 're eating grad e: so i 'm just not sure phd a: cuz they might make different sounds , like n nuts chocolate with nuts , chocolate without nuts . postdoc c: oh . professor b: um phd d: crunchy frogs . phd f: chocolate adaptation . professor b: actually actually kind of careful cuz i have a strong allergy to nuts , so i have to sort of figure out one without th phd a: that w oh , yeah , they they might . professor b: it 's hard to hard to say . phd a: maybe those ? they 're so i do n't know . professor b: i do n't know . um phd a: this is you know , this is a different kind of speech , professor b: well phd h: take take several . phd a: looking at chocolates , deciding phd f: mmm . phd a: you know , it 's another style . professor b: yeah . i may i may hold off . phd f: mmm . professor b: but if i was eh , but maybe i 'll get some later . thanks . phd f: mmm . professor b: well well , why do n't we ? he he 's worried about a ticket . why do n't we do a simultaneous one ? phd a: ok . professor b: simultaneous one ? postdoc c: ok . grad e: ok . phd f: mmm . phd a: and you laughed at me , too , f the first time i said that . professor b: ok . grad e: remember to read the transcript number , please . phd f: right . phd h: ok . professor b: i have to what ? phd d: oops . phd h: yeah . phd a: you laughed at me , too , the first time i sa said professor b: i did , phd a: you really should n't , uh , te professor b: and now i love it so much . grad e: ok , everyone ready ? phd a: you have to sort of , um jose , if you have n't done this , you have to plug your ears while you 're t talking professor b: w wait wait a minute wait a minute . w we want we want phd a: so that you do n't get confused , i guess . professor b: we want it synchronized . phd a: yeah . oh , you 've done this one before ? postdoc c: hey , you 've done this before . have n't you ? phd h: yeah . phd d: that 's phd a: together ? postdoc c: you 've read digits together with us , have n't you i mean , at the same time ? phd a: i 'm not we we oh , and you have n't done this either . professor b: ok . postdoc c: oh , you have n't ! phd h: no . postdoc c: oh , ok . phd d: oh , yeah . phd a: i the first time is traumatic , professor b: we phd a: but professor b: y yeah , bu postdoc c: oh , and the groupings are important , phd h: mmm . postdoc c: so yo you 're supposed to pause between the groupings . phd h: the grouping . professor b: yeah . phd h: yeah . professor b: ok . so , uh phd f: you mean that the the grouping is supposed to be synchronized ? professor b: no , no . postdoc c: no . grad e: yeah , sure . phd f: no ? phd a: that 'd be good . professor b: synchronized digits . postdoc c: no . phd f: no ? phd a: we - we 'll give everybody the same sheet phd f: it 's like a like a greek like a greek choir ? phd a: but they say different phd f: you know ? professor b: yes . grad e: hey , what a good idea . phd f: like grad e: we could do the same sheet for everyone . phd f: yeah . grad e: have them all read them at once . phd a: well , different digits phd d: eh phd a: but same groupings . grad e: or or just same digits . phd a: so they would all be yeah . postdoc c: yeah . that 'd be good . grad e: see if anyone notices . professor b: there 's so many possibilities . postdoc c: and then then we can sing them next time . professor b: uh . ok , why do n't we go ? uh , one two three go ! postdoc c: ok . mmm ! professor b: and andreas has the last word . grad e: did you read it twice or what ? phd a: he 's try no , he 's trying to get good recognition performance . postdoc c: he had the h phd h: yeah . postdoc c: he had the the long form . phd h: yeah . grad e: and we 're off . phd f: no . </s> [SEP]what would future experiments entail ?
subsequent recognition experiments will look at large vocabulary speech from a far-field microphone ( as performed in switchboard evaluations ) .