  Despite the valuable information contained in software chat messages,
disentangling them into distinct conversations is an essential prerequisite for
any in-depth analyses that utilize this information. To provide a better
understanding of the current state-of-the-art, we evaluate five popular dialog
disentanglement approaches on software-related chat. We find that existing
approaches do not perform well on disentangling software-related dialogs that
discuss technical and complex topics. Further investigation on how well the
existing disentanglement measures reflect human satisfaction shows that
existing measures cannot correctly indicate human satisfaction on
disentanglement results. Therefore, in this paper, we introduce and evaluate a
novel measure, named DLD. Using results of human satisfaction, we further
summarize four most frequently appeared bad disentanglement cases on
software-related chat to insight future improvements. These cases include (i)
ignoring interaction patterns; (ii) ignoring contextual information; (iii)
mixing up topics; and (iv) ignoring user relationships. We believe that our
findings provide valuable insights on the effectiveness of existing dialog
disentanglement approaches and these findings would promote a better
application of dialog disentanglement in software engineering.

Software Engineering