  An automated metric to evaluate dialogue quality is vital for optimizing data
driven dialogue management. The common approach of relying on explicit user
feedback during a conversation is intrusive and sparse. Current models to
estimate user satisfaction use limited feature sets and employ annotation
schemes with limited generalizability to conversations spanning multiple
domains. To address these gaps, we created a new Response Quality annotation
scheme, introduced five new domain-independent feature sets and experimented
with six machine learning models to estimate User Satisfaction at both turn and
dialogue level.
  Response Quality ratings achieved significantly high correlation (0.76) with
explicit turn-level user ratings. Using the new feature sets we introduced,
Gradient Boosting Regression model achieved best (rating [1-5]) prediction
performance on 26 seen (linear correlation ~0.79) and one new multi-turn domain
(linear correlation 0.67). We observed a 16% relative improvement (68% -> 79%)
in binary ("satisfactory/dissatisfactory") class prediction accuracy of a
domain-independent dialogue-level satisfaction estimation model after including
predicted turn-level satisfaction ratings as features.

Machine Learning;Human-Computer Interaction;Machine Learning