MESIA: Understanding and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generation
<|reference_start|>MESIA: Understanding and Leveraging Supplementary Nature of Method-level Comments for Automatic Comment Generation: Code comments are important for developers in program comprehension. In scenarios of comprehending and reusing a method, developers expect code comments to provide supplementary information beyond the method signature. However, the extent of such supplementary information varies a lot in different code comments. In this paper, we raise the awareness of the supplementary nature of method-level comments and propose a new metric named MESIA (Mean Supplementary Information Amount) to assess the extent of supplementary information that a code comment can provide. With the MESIA metric, we conduct experiments on a popular code-comment dataset and three common types of neural approaches to generate method-level comments. Our experimental results demonstrate the value of our proposed work with a number of findings. (1) Small-MESIA comments occupy around 20% of the dataset and mostly fall into only the WHAT comment category. (2) Being able to provide various kinds of essential information, large-MESIA comments in the dataset are difficult for existing neural approaches to generate. (3) We can improve the capability of existing neural approaches to generate large-MESIA comments by reducing the proportion of small-MESIA comments in the training set. (4) The retrained model can generate large-MESIA comments that convey essential meaningful supplementary information for methods in the small-MESIA test set, but will get a lower BLEU score in evaluation. These findings indicate that with good training data, auto-generated comments can sometimes even surpass human-written reference comments, and having no appropriate ground truth for evaluation is an issue that needs to be addressed by future work on automatic comment generation.<|reference_end|>
@article{pan2024mesia:,
    title={MESIA: Understanding and Leveraging Supplementary Nature of Method-level
  Comments for Automatic Comment Generation},
    author={Xinglu Pan, Chenxiao Liu, Yanzhen Zou, Tao Xie, Bing Xie},
    journal={arXiv preprint arXiv:2403.17357},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.17357},
    primaryClass={cs.SE cs.AI}
}