“Active Larynx”: Preliminary Evaluation of the Reliability of Visual Assessments of Laryngeal Inflammation
Study Design This study was a cross‐sectional evaluation of direct laryngoscopy and bronchoscopy (DLB) images using 3 judges via an electronic survey. The study was approved by the Institutional Review Board of Nationwide Children's Hospital (STUDY00000032). The study was designed after reviewing the literature and consulting with airway reconstruction experts for content validity. Judges Three fellowship‐trained pediatric otolaryngologists served as judges for the DLB image library. All 3 were blinded to all aspects of construction of the survey instrument including selection of images for the library and any associated patient information. The judges all have a demonstrated LTR practice at Nationwide Children's Hospital, a tertiary pediatric care center in Columbus, Ohio. Survey Ratings For the pilot study, deidentified DLB images from 15 procedures were used to create a library. All procedures were performed at Nationwide Children's Hospital, and each included a supraglottic and glottic view of the larynx ( and ). The images were selected by the authors P.P. and A.M.M. to represent a diverse range of activeness levels and photos that were excessively blurry, not centered, or otherwise difficult to evaluate were excluded. The images were primarily selected from surgical logs in the medical record, as well as newly created tracheostomy and open airway reconstruction databases. Notably, image suggestions were not solicited from colleagues for fear of biasing those colleagues regarding those images in the questionnaire. Both views of each larynx were presented twice (Presentation A and Presentation B), with specific questions for each presentation. The 30 presentations (15 larynges, 2 presentations each) were randomized throughout the survey. Presentation A ( ) asked the respondent to analyze the larynx and rate several characteristics on a 5‐point Likert scale. The following characteristics were surveyed for edema, erythema, cobblestone mucosa, and eversion of laryngeal ventricles. Other features of inflammation, such as crusting, exudate, and the presence of nodules or polyps were initially included but removed after insufficient representation was found in our image library. Presentation B ( ) asked the respondent to analyze the overall “activeness” of the larynx on a scale of 0.0 to 10.0, indicating how active the larynx appeared to be. The respondent was also asked a binary question (yes/no) if they would delay performing airway reconstruction surgery based on how active the larynx was. Statistical Analysis Data were assessed for internal consistency, and a preliminary scale based on the tested characteristics was presented and tested. In total, there were 4 inflammation characteristic variables (edema, erythema, cobblestoning, and eversion of laryngeal ventricles), 1 variable for overall “activeness,” and 1 for whether the surgeon would delay airway reconstruction. Finally, 1 variable for a tentative active larynx scale was created summing the 4 characteristic variables (with 1 point for each step on the Likert scale), with a minimum value of 4 and a maximum value of 20. Interrater agreement was evaluated using intraclass correlation coefficient (ICC) and 95% confidence interval (CI) using a 2‐way random‐effects model with both judges and pictures as random‐effects testing for absolute agreement. ICC is comparable to Cohen's κ for assessing interrater reliability with multiple raters and categorical data and is in line with our previous work in scale development for airway pathology. The ICCs were interpreted with standard cutoffs for levels of agreement , : below 0.50 for poor agreement, between 0.50 and 0.75 for moderate agreement, between 0.75 and 0.90 for good agreement, and above 0.90 for excellent agreement. In addition to ICC, Fleiss's κ using quadratic weighting was also calculated to evaluate interrater reliability. The same cutoffs were used for κ , as this would be similar to or more stringent than other cutoffs in the literature. , , Pearson's R was used to evaluate interrater reliability for each pair of the 3 judges (judges 1 and 2, 2 and 3, and 1 and 3) for the overall activeness variable and the activeness scale. Preliminary measures of construct validity were measured. Convergent validity was measured via correlation of the tentative activeness scale to the overall activeness rating, again using Pearson's R . Next, discriminant validity was tested. A 2‐sample t ‐test was used to determine if activeness was significantly different for larynges that surgeons would delay surgery than those they would not. This was performed for both the tentative activeness scale and the overall activeness rating. All analyses were performed using Stata/IC 16 software (StataCorp LP). 

This study was a cross‐sectional evaluation of direct laryngoscopy and bronchoscopy (DLB) images using 3 judges via an electronic survey. The study was approved by the Institutional Review Board of Nationwide Children's Hospital (STUDY00000032). The study was designed after reviewing the literature and consulting with airway reconstruction experts for content validity. 

Three fellowship‐trained pediatric otolaryngologists served as judges for the DLB image library. All 3 were blinded to all aspects of construction of the survey instrument including selection of images for the library and any associated patient information. The judges all have a demonstrated LTR practice at Nationwide Children's Hospital, a tertiary pediatric care center in Columbus, Ohio. 

For the pilot study, deidentified DLB images from 15 procedures were used to create a library. All procedures were performed at Nationwide Children's Hospital, and each included a supraglottic and glottic view of the larynx ( and ). The images were selected by the authors P.P. and A.M.M. to represent a diverse range of activeness levels and photos that were excessively blurry, not centered, or otherwise difficult to evaluate were excluded. The images were primarily selected from surgical logs in the medical record, as well as newly created tracheostomy and open airway reconstruction databases. Notably, image suggestions were not solicited from colleagues for fear of biasing those colleagues regarding those images in the questionnaire. Both views of each larynx were presented twice (Presentation A and Presentation B), with specific questions for each presentation. The 30 presentations (15 larynges, 2 presentations each) were randomized throughout the survey. Presentation A ( ) asked the respondent to analyze the larynx and rate several characteristics on a 5‐point Likert scale. The following characteristics were surveyed for edema, erythema, cobblestone mucosa, and eversion of laryngeal ventricles. Other features of inflammation, such as crusting, exudate, and the presence of nodules or polyps were initially included but removed after insufficient representation was found in our image library. Presentation B ( ) asked the respondent to analyze the overall “activeness” of the larynx on a scale of 0.0 to 10.0, indicating how active the larynx appeared to be. The respondent was also asked a binary question (yes/no) if they would delay performing airway reconstruction surgery based on how active the larynx was. 

Data were assessed for internal consistency, and a preliminary scale based on the tested characteristics was presented and tested. In total, there were 4 inflammation characteristic variables (edema, erythema, cobblestoning, and eversion of laryngeal ventricles), 1 variable for overall “activeness,” and 1 for whether the surgeon would delay airway reconstruction. Finally, 1 variable for a tentative active larynx scale was created summing the 4 characteristic variables (with 1 point for each step on the Likert scale), with a minimum value of 4 and a maximum value of 20. Interrater agreement was evaluated using intraclass correlation coefficient (ICC) and 95% confidence interval (CI) using a 2‐way random‐effects model with both judges and pictures as random‐effects testing for absolute agreement. ICC is comparable to Cohen's κ for assessing interrater reliability with multiple raters and categorical data and is in line with our previous work in scale development for airway pathology. The ICCs were interpreted with standard cutoffs for levels of agreement , : below 0.50 for poor agreement, between 0.50 and 0.75 for moderate agreement, between 0.75 and 0.90 for good agreement, and above 0.90 for excellent agreement. In addition to ICC, Fleiss's κ using quadratic weighting was also calculated to evaluate interrater reliability. The same cutoffs were used for κ , as this would be similar to or more stringent than other cutoffs in the literature. , , Pearson's R was used to evaluate interrater reliability for each pair of the 3 judges (judges 1 and 2, 2 and 3, and 1 and 3) for the overall activeness variable and the activeness scale. Preliminary measures of construct validity were measured. Convergent validity was measured via correlation of the tentative activeness scale to the overall activeness rating, again using Pearson's R . Next, discriminant validity was tested. A 2‐sample t ‐test was used to determine if activeness was significantly different for larynges that surgeons would delay surgery than those they would not. This was performed for both the tentative activeness scale and the overall activeness rating. All analyses were performed using Stata/IC 16 software (StataCorp LP). 

All judges reviewed all 15 DLBs, yielding 45 image‐ratings. The overall distribution for all image findings is demonstrated in . Notably, responses were skewed, with limited responses indicating “moderate,” “very,” or “extreme,” especially for cobblestoning and ventricular eversion. demonstrates the larynx with the lowest rating by the judges (average rating 4.67 on the tentative scale). demonstrates the larynx with the highest rating for the edema item, with an average score of 3.33 from this item (between “moderate” and “very”). demonstrates the larynx with the highest rating for the erythema item (average score 4). demonstrates the larynx tied for the highest score for cobblestoning (average score 2; tied with ). Finally, the larynx in had the highest score for the ventricular eversion item (average score 2), tied for the highest score for cobblestoning, and had the highest overall rating (average rating 10). demonstrates the interrater reliability of the ratings, through ICC, Fleiss's κ , and Pearson's correlation. By ICC, edema and erythema had a substantial agreement, while ventricular eversion had a moderate agreement and cobblestoning had a poor agreement. ICC was poor for whether the surgeon would delay airway reconstruction, but substantial for the overall assessment of activeness and the tentative scale created by summing the 4 characteristics. By Fleiss's κ , edema and erythema had a moderate agreement, while eversion and cobblestoning had a poor agreement. The question of whether to delay surgery also had low agreement. Both the overall assessment of activeness and the tentative scale had a strong correlation when comparing each pair of judges. The preliminary assessment of construct validity demonstrated strong convergent validity ( R of 0.68). A significant difference was noted in the tentative activeness scale between larynges for which the surgeon would delay surgery and ones for which they would not ( P < .001). A significant difference was also noted in the overall activeness rating for both decisions ( P < .001). Convergent and discriminant validity measures are demonstrated in . 

The focus of this study was to measure the reliability of visual indicators of laryngeal inflammation, improve clinical communication and decision‐making, and provide a framework for developing a scale to define an “active” larynx. This study provides promising preliminary data toward the development of a scale for visually assessing the inflammation of a larynx when planning LTR. In particular, both ICC and Fleiss's κ indicated good interrater reliability for edema and erythema, but relatively poor agreement for ventricular eversion or cobblestone appearance of laryngeal mucosa. This is likely due to the poor representation of laryngeal eversion and cobblestoned mucosa in the image library, a limitation of the preliminary work that will be rectified in future inquiries. This limited image library was in part due to an effort to not use images from the respondents' own procedures in case of potential bias. In future multi‐center expansions, this concern will be less relevant. Not surprisingly, while the tentative scale has a range of possible scores from 4 to 20, the range of scores given by the raters was considerably less representative (4.67‐10), as shown in . Additionally, the agreement was substantial for the overall active larynx rating and the tentative scale formed by summing the inflammatory characteristics together. But agreement was low to moderate for the binary (yes/no) question of whether the rater would delay surgery. This indicates that although raters agreed on the total score, they did not consistently agree on whether surgery should be delayed. Given that the raters were blinded to individual patient factors, this disagreement demonstrates how subjectivity can lead to inconsistencies in clinical decision‐making among surgeons. It reinforces the need for a more standardized assessment such as with the use of a scale to assess latent factors that might otherwise influence treatment decision‐making. While the agreement was poor for the decision of whether to delay surgery due to laryngeal inflammation, there was a significant difference in the overall rating of activeness and in the sum of the characteristics measured between larynges for which surgeons would delay surgery and larynges for which surgeons would not. A formal analysis of the predictive capacity including exploratory factor analysis and evaluations of internal consistency is not possible with such a small study population. , The decision to proceed with surgery is multifactorial, and activeness may play a small role in the decision. As such, an activeness scale is unlikely to ever be predictive of surgery planning. However, this initial evaluation of convergent and discriminant validity is promising and warrants further investigation. Although the term active larynx is anecdotally discussed frequently among airway surgeons, it is rarely seen in the literature. A recent consensus statement on outcomes to report for LTR nearly reached consensus for including “active/reactive larynx.” Other guidance documents reference evaluating for inflammation whether inflammation is “active” or mature. Numerous other reports discuss prophylactic treatments of reflux medications, azithromycin, topical or systemic steroids, or other anti‐inflammatory treatments. , , This is not surprising given how inflammation is implicated in the formation and progression of stenosis in the context of airway reconstruction and otherwise. , , , , Despite these numerous causes of airway inflammation, it is not clear that these causes have different appearances endoscopically. For example, diagnosis of laryngopharyngeal reflux by laryngoscopy findings has been limited by low positive predictive value and competing diagnoses. , Follow‐up studies on the active larynx might categorize larynges in the image library according to etiology to determine if these causes can be differentiated visually. Despite airway inflammation being implicated in the success rate after LTR, literature on the active larynx has been limited. Multicenter outcome studies on LTR have primarily identified medical comorbidities and grade and anatomy of airway stenosis in predicting surgical success rate. A standardized assessment of laryngeal inflammation could allow for the assessment of this potential surgical risk factor. While the current study is preliminary, it sets a foundation for such a standardized assessment, reducing unnecessary variation in surgical decision‐making. In addition to the limited representation of more severe ventricular eversion or cobblestoned mucosa, there were several limitations in this study. The limited sample size of the preliminary study could lead to an increase in agreement due to overlapping management strategies and paradigms. Additionally, the use of static images for the survey may have affected assessments, given that a DLB is a dynamic evaluation. While video presentations may have improved the accuracy of ratings, lack of palpation with endoscopic instruments may have limited subtler assessments (eg, friability of mucosa). Additionally, the image library was created by selecting appropriate and clear images of laryngeal pathology, potentially biasing the results toward greater agreement among raters. Finally, subglottic and suprastomal pathology visible through the glottis may have biased raters toward specific management paradigms and similar scores. These concerns are being used to plan a larger, multi‐institutional survey project, evaluating the agreement of visual assessment of laryngeal inflammation for LTR decision‐making. 

The active larynx is an important consideration in planning laryngeal reconstructive surgery. While the assessment of the active larynx is subjective, some characteristics can be reliably ascertained. Hence, the creation of an active larynx scale is feasible. 

Prasanth Pattisapu , design, conduct, analysis, presentation, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Amy M. Manning , design, conduct, analysis, presentation, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Michael J. Boutros , design, conduct, analysis, presentation, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Megan McNutt , conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Tendy Chiang , conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Jonathan M. Grischkan , conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Meredith N. Lind , conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; John F.P. Bridges , design, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript. 

Competing interests The authors have no conflicts of interest to disclose. Funding source None. 

The authors have no conflicts of interest to disclose. 

None. 

