text:
\section{The Taxonomy of Noisy Data}

To develop a taxonomy of noisy data for code summarization tasks, we employ an \textit{open card sort} \cite{DBLP:journals/es/RuggM05} process by involving nine  participants.
{Participants include two PhD students, four master students, and three senior researchers.
All of them are fluent English speakers, and have done either intensive research work with software development or have been actively contributing to open-source projects.}
The sorting process is conducted in multiple rounds. In the first round,
all participants label the same sample of benchmark data, with an intensive discussion session to achieve conceptual coherence about noisy categories. The average Cohen's Kappa is 0.86.
Then, a shared pool of categories is utilized and carefully maintained, and
each participant could select existing categories from and/or add new category names into the shared pool. The sorting process ends when there is no new category added for two consecutive rounds. In total, we conducted 10 rounds and labeled 1600 pairs of code summary (400 pairs for each four benchmark datasets).
{Among them, we annotated XXX syntactic noises, XXX non-summary comments, and XXX summary comments.} \lin{@fangwen}\lin{updated @Ge}
The detail annotation results are shown in Table \ref{table:rq2_dataset}. {Next, we introduce 11 categories of syntactic noise and one category (non-summary) of semantic noise, as well as and their examples in the following.}

\input{sec/taxonomy-2}

\section{The Code-Summary Cleaning Tool}

In this section, we introduce our automated data cleaning tool for code summarization datasets. As shown in Figure \ref{fig:overview}, the tool consists of two subsequent cleaners, a rule-based syntactic cleaner and a model-based semantic cleaner. First, the rule-based syntactic cleaner is used for removing syntactic noises, and then the model-based semantic cleaner is used for classifying non-summary and summary comments from the rest (Note that, we refer to semantic noises as non-summary comments and refer to normal comments as summary comments in this study).\lin{updated @Ge}
In the following, we first elaborate on the design of the two subsequent cleaners, and then report the effectiveness evaluation of the tool.

\subsection{The Syntactic Cleaner}

To automatically detect each category of syntactic noises in the taxonomy, we develop a syntactic cleaner with a set of heuristic rules.
The heuristic rules comply with three criteria: (1) Any rule should define a unique and
specific construction category; (2) The rules should limit the preclusion of valid data within an acceptable range \jie{the reviewers might argue what is the `acceptable range?'}\lin{I follow previous study when state this, our inspection shows all above 90\% F1 score, any idea how can we update here @junjie}\jie{i am not sure}; and (3) Any rule is not a subrule of other rules in the set.
Table \ref{tab:taxonomy} demonstrates the syntax feature of heuristic rules and our actions.

 \begin{figure}[t]
\centering
 \includegraphics[width=0.9\columnwidth]{fig/alg.pdf}

 \caption{If-else rules of collecting the first sentence in the raw comments for partial and verbose sentence noises.}
 \label{fig:3-flow}
 \end{figure}

We develop heuristic rules for detecting each category of syntactic noises. Taking the partial and verbose sentence category as an example, first, we carefully identify their syntax features from part of the manually annotated noises. Then, we design a set of if-else rules to collect the correct first sentence from the raw data, and test their correctness on the rest of the manually annotated noises. \lin{updated @Ge}
(Note that, the raw data refers to the source data that has not been processed for use, and the origin data refers to the processed raw data by benchmark datasets).
Second, by comparing the first sentence we extracted with the data provided in the benchmark dataset, we can determine the verbose or partial sentence category.
Figure \ref{fig:3-flow}
illustrates the if-else rules of collecting the first sentence in the raw comments for partial and verbose sentence noises. The key idea is to sequentially determine whether each comment line in a raw comment contains a complete sentence, and if so, return the first complete sentence; if not, save the content of the line and continue to determine the next line. \lin{@updated @junjie}

Details of the heuristic rules for other categories can be found on our website \cite{website}.

\input{table/RQ1_v3}

\subsection{The Semantic Cleaner}

\lin{TBD: add rationale of using external data? @wangsong what do you think}
We utilize a manual-inspected external data provided by Chen et al.~\cite{DBLP:journals/tosem/ChenXHLL21} to train the semantic noise classifier. They classified comments into six categories: ``what'', ``why'', ``how-to-use'', ``how-it-is-done'', ``property'', and ``others''.  Following their work, we consider the first five categories as summary, and the ``others'' as non-summary noises.

\textbf{GBDT Model.}
To classify non-summary noises, we employ the Gradient Boosting Decision Tree (GBDT) \cite{DBLP:conf/nips/KeMFWCMYL17} as our classification model.
\jie{do we need such details of the model. it is a commonly used machine learning model. also i think using the `log-likelihood loss function' is a common practice.}\lin{TBD}
The GBDT is an ensemble model consisting of $k$ Decision Trees, and the latter algorithm has been proven to be effective in comment classification tasks \cite{DBLP:conf/iwpc/SteidlHJ13, DBLP:journals/ese/PascarellaBB19, DBLP:journals/tosem/ChenXHLL21}. \mfw{updated @Ge}
It can be formulated as:
\[\hat{y}_i=\sum\limits_{k=1}^K{f_k(x_i)},f_k\in F\]
where $f_k(x_i)$ is the $k-$th decision tree, $F$ is a functional space composed of all trees. It is trained in sequence, and each base model is added to correct the error produced by its previous tree models. For each step, the samples misclassified by previous models are more likely to be selected as the train data, producing more accurate prediction performance. We use the log-likelihood loss function to optimize the model:
\[L=\log(1+\exp(-y\hat{y}_i)\]
where $y$ is the ground-truth label, $\hat{y}_i$ is the predicted probability.

\input{table/RQ2_feature}

\textbf{Features.}
Based on our observation of the comment data, we find that non-summary data has statistical characteristics and content patterns that can be used to distinguish it from summary data. Thus, we define two categories features, which are statistical features and content features, as the input of the GBDT model.
\begin{figure*}[htpb]
\centering
\includegraphics[width=1.7\columnwidth]{fig/pie.pdf}
\caption{Percentage of syntactic (SYN) and semantic (SEM) noise in terms of removal and update.\lin{add overall}}
\label{fig:rq1-pie}
\end{figure*}

Table \ref{tab:rq2_feature} illustrates the detail information of these features.
For statistical features, after tokenization of the comment data, we extract TF-IDF \cite{DBLP:conf/sigir/SaltonY73} vector of each word, which could measure the mathematical significance of words in comments. This technique is a conventional feature extraction approach for the natural language text analysis in the software engineering literature \cite{behl2014bug}. We also extract the number of words and unique words, and calculate the percentage of punctuation and digits in comments.
For content features, we determine whether comments contain keywords such as ``todo", ``deprecate", ``copyright", ``FIXME:", and whether they contain code snippets or mathematical formulas. Such features are also used in prior software engineering literature \cite{DBLP:conf/iwpc/SteidlHJ13, DBLP:journals/ese/PascarellaBB19}.

\textbf{SMOTE.} The classification of summary and non-summary from the training
data is imbalanced in that the summary data are much more than the non-summary data. To alleviate the imbalance problem, we employ the synthetic minority over-sampling technique (SMOTE) \cite{DBLP:journals/jair/ChawlaBHK02}, which can avoid the over-fitting problem by generating new instances with the help of interpolation between the positive instances that lie together.

\subsection{Effectiveness Evaluation}
In this section, we evaluate the effectiveness our syntactic cleaner and semantic cleaner.

\subsubsection{Data Preparation.}
As introduced in Section 3.1, we labeled 11 categories of syntactic noises from 1600 code-summary pairs sampled from the four datasets.
Based on the data that are not labeled as syntactic noises, we further manually annotate them into summary and non-summary data. We use these manually annotated data to evaluate the performance of our cleaner.
Table \ref{table:rq2_dataset} shows the detail.
For syntactic cleaner, We build the heuristic rules based on observing the 80\% of the annotated syntactic noises, and evaluate on the rest 20\%.
For semantic cleaner, we utilize the external dataset provided by Chen et al. \cite{DBLP:journals/tosem/ChenXHLL21} to train our model, and test the model on the summary and non-summary data that we manually  annotated. \lin{updated @Ge}

\subsubsection{Experimental setup}
We implement the model-based semantic cleaner using scikit-learn \cite{scikitlearn} which is an open-source data analysis and machine learning library. We train the GBDT model with log-likelihood loss function.
{For the} hyper-parameters,
we use greedy search as the parameter tuning  method to
obtain the best performance.
{After the parameter tuning, we set the \textit{learning\_rate}=0.01, \textit{n\_estimators}=100, \textit{subsample}=1, and \textit{min\_samples\_leaf}=2.}

\subsubsection{Evaluation Metrics.}

We use three commonly-used metrics to evaluate the performance of our cleaning tool, i.e., \textit{Precision}, \textit{Recall}, and \textit{F1}. (1) \textit{Precision} refers to the ratio of correct predictions to the total number of predictions; (2) \textit{Recall} refers to the ratio of correct predictions to the total number of samples in the golden test set; and (3) \textit{F1} is the harmonic mean of precision and recall.

\input{table/RQ2_dataset_v4}
\subsubsection{Results}

Table \ref{table:rq2_dataset} demonstrates the performance of our two subsequent cleaners in terms of noisy data detection. We can see that, the syntactic cleaner can precisely detect syntactic noises on the four benchmark datasets. The Macro F1 scores are ranging from 97.61\% to 96.49\%, and 96.99\% on average, which is highly satisfactory \jie{highly satisfactory might not a good expression}\lin{@wangsong any suggestions?}.
For semantic cleaner, the F1 scores of non-summary classification are ranging from 86.67\% to 92.86\%, and 91.74\% on average. The results show that, the semantic cleaner can also achieve a highly satisfactory performance on filtering non-summary noises.
In summary, Our code-summary cleaner can precisely filter noisy data syntactically and semantically, with all the F1 scores of over 90\%, {which help build a high-quality dataset for the follow-up code comment summary.}

\section{The Code-Summary Cleaning Tool}

In this section, we introduce our automated data
cleaning tool for code summarization datasets, mainly to filter out
code-summary pairs with syntactic or semantic noises. The tool consists of two subsequent cleaners, a rule-based syntactic cleaner and a
model-based semantic cleaner.
An overview of the tool is shown in Figure \ref{fig:overview}.
The origin code-summary pairs in benchmark datasets are firstly cleaned
by the rule-based cleaner, where a set of heuristic rules are applied to detect the
existence of invalid syntax.
Next, we further filter semantic noise based on our model-based cleaner, which is trained on an external dataset.
In the following, we first elaborate on the design of the two subsequent cleaners, and then report the effectiveness evaluation of the tool.

\subsection{The Syntactic Cleaner}

Since raw data can hardly be directly used for code summarization tasks, nearly all the existing benchmark datasets are constructed under multiple preprocessing operations, e.g., using `newline' as the delimiter to collect the first sentence in the comments \cite{leclair2019neural}.
\jie{not only the preprocessing, but also the data itself, cause the noise in the data. i think this current statement might be misleading.}
However, we have a concern that about how well are these preprocessing operations performed in constructing benchmark datasets.
To address the above concern, we develop a syntactic cleaner with a set of heuristic rules that can precisely identify syntactic noises.

To develop a taxonomy of syntactic noises for both code data and summary data, we employ an \textit{open card sort} \cite{DBLP:journals/es/RuggM05} process by involving XXX participants. \lin{add demographical profiles@fangwen}
The sorting process is conducted in multiple rounds. In the first round,
all participants label the same sample of benchmark data, with an intensive discussion session to achieve conceptual coherence about noisy categories. The average Cohen's Kappa is 0.86.
Then, a shared pool of categories is utilized and carefully maintained, and
each participant could select existing categories from and/or add new category names into the shared pool. The sorting process ends when there is no new category added for two consecutive rounds. In total, we conducted XX rounds and labeled 1600 pairs of code summary (400 pairs for each four benchmark datasets). The detail annotation results are shown in XXX.

To automatically detect each category of syntactic noises in the taxonomy, we develop a syntactic cleaner with a set of heuristic rules.
The heuristic rules comply with three criteria: (1) Any rule should define a unique and
specific construction category; (2) The rules should limit the preclusion of valid data within an acceptable range; and (3) Any rule is not a subrule of other rules in the set.
Limited by the space, the details of the heuristic rules can be found on our website \cite{}.

Table \ref{tab:taxonomy} demonstrates the taxonomy of syntactical noises, syntax feature of heuristic rules, real-world examples, and our corrections.

\subsection{The Semantic Cleaner}

In this work, we build our semantic cleaner based on Chen et al.'s dataset \cite{DBLP:journals/tosem/ChenXHLL21}.
Chen \textit{et al}. have classified comments into six categories: ``what'', ``why'', ``how-to-use'', ``how-it-is-done'', ``property'', and ``others''.  Following their work, we consider the first five categories as summary, and the ``others'' as non-summary noises.

\textbf{GBDT Model.}
To classify non-summary noises, we employ the Gradient Boosting Decision Tree (GBDT) \cite{} as our classification model, which has been proven to be effective in comment classification tasks \cite{}.
The GBDT is an ensemble model consisting of $k$ decision trees:
\[\hat{y}_i=\sum\limits_{k=1}^K{f_k(x_i)},f_k\in F\]
where $f_k(x_i)$ is the $k-$th decision tree, $F$ is a functional space composed of all trees. It is trained in sequence, and each base model is added to correct the error produced by its previous tree models. For each step, the samples misclassified by previous models are more likely to be selected as the train data, producing more accurate prediction performance. We use the log-likelihood loss function to optimize the model:
\[L=\log(1+\exp(-y\hat{y}_i)\]
where $y$ is the ground-truth label, $\hat{y}_i$ is the predicted probability.

\input{table/RQ2_feature}

\textbf{Features.}
{Based on our observation of the external dataset, we find that non-summary data has statistical characteristics and content patterns that can be used to distinguish it from summary data. Thus, we define two categories features, which are statistical features and content features, as the input of the GBDT model.}
Table \ref{tab:rq2_feature} illustrates the detail information of these features.
For statistical features, after tokenization of the comment data, we extract TF-IDF \cite{} vector of each word, which could measure the mathematical significance of words in comments. This technique is a conventional feature extraction approach for the natural language text analysis in the software engineering literature \cite{}. We also extract the number of words and unique words, and calculate the percentage of punctuation and digits in comments.
For content features, we determine whether comments contain keywords such as ``todo", ``deprecate", ``copyright", ``FIXME:", and whether they contain code snippets or mathematical formulas. Such features are also used in prior software engineering literature \cite{}.

\textbf{SMOTE.} The classification of summary and non-summary from benchmark data is imbalanced in that the summary data are much more than the non-summary data. To alleviate the imbalance problem, we employ the synthetic minority over-sampling technique (SMOTE) \cite{}, which can avoid the over-fitting problem by generating new instances with the help of interpolation between the positive instances that lie together.

\subsection{Effectiveness Evaluation}
In this section, we evaluate the effectiveness our syntactic cleaner and semantic cleaner by manual inspection.
\input{table/RQ2_dataset_v4}
\subsubsection{Data Preparation.}
As introduced in section 3.1, we labeled 11 categories of syntactic noises from 1600 code-summary pairs sampled from the four datasets. Based on the data that are not labeled as syntactic noises, we further manually annotate them into summary and non-summary data. We use these manually annotated data to evaluate the performance of our cleaner.
Table XX shows the detail.
For syntactic cleaner, We build the heuristic rules based on observing the 80\% of our annotated data, and evaluate on the rest 20\%.
For semantic cleaner, we utilize the external dataset provided by Chen et al. \cite{DBLP:journals/tosem/ChenXHLL21} to train our model. We aggregate the ``what'', ``why'', ``how-it-is-done'',``how-to-use'', and ``property'' labeled data as our summary data, and use the ``others'' as non-summary data. We test the model on our manual annotated data.

\subsubsection{Experimental setup}
We implement the model-based semantic cleaner using scikit-learn \cite{} which is an open-source data analysis and machine learning library. We train the GBDT model with \textit{learning\_rate}=0.01. We set the \textit{n\_estimators}=100, \textit{subsample}=1, and \textit{min\_samples\_leaf}=2. For these hyper-parameters,
we use greedy search as the parameter selection method to
obtain the best performance.

\subsubsection{Evaluation Metrics.}

We use three commonly-used metrics to evaluate the performance of our cleaning tool, i.e., \textit{Precision}, \textit{Recall}, and \textit{F1}. (1) \textit{Precision} refers to the ratio of the number of correct predictions to the total number of predictions; (2) \textit{Recall} refers to the ratio of correct predictions to the total number of samples in the golden test set; and (3) \textit{F1} is the harmonic mean of precision and recall.

\subsubsection{Results}

\input{table/distilled-size}

Table \ref{table:rq2_dataset} demonstrates the performance of our syntactic cleaner in terms of noisy data detection. We can see that, the syntactic cleaner can precisely detect syntactic noises on the four benchmark datasets. The Micro F1 scores are ranging from 93.76\% to 95.5\%, and 94.65\% on average, which is highly satisfactory.
Table \ref{table:sem-perf} shows the performance of our semantic cleaner. We can see that, the F1 scores of non-summary classification are ranging from 86.67\% to 92.86\%, and 91.74\% on average. The results show that, the semantic cleaner can also achieve a highly satisfactory performance on filtering non-summary noises.

\noindent
\begin{boxedminipage}[t]{\columnwidth}
\jie{i think it isn't necessary to treat this as a separate finding. this is not a finding. }
\textbf{Finding 1}: Our code-summary cleaner can precisely filter noisy data syntactically and semantically, with all the F1 scores of over 90\%.
\end{boxedminipage}
\section{Discussion}
Results from this study suggest several interesting findings that can serve as the practical guidelines for improving code summarization regarding the collection and selection of benchmark data

\subsection{On the Importance of Data Quality.}
Data quality has been a growing concern, especially since
deep learning (DL) is being widely applied for massive SE tasks. As DL models typically require high-volume data, ensuring data quality at a large scale has become a compelling need.
Most existing studies focus on advances in modeling but typically overshadowed the quality of data. When investigating the existing code summarization models (as shown in Table \ref{table:dataset_intro}), we notice that a substantial amount of data preprocessing operations are cursory or lack of consistency.
In addition, there is the lack of principles or methods to guide and reinforce the data preprocessing associated effort while conducting code summarization research, regarding how to soundly preprocess benchmark datasets for different tasks.
For instance, our results show that noisy data extensively exist in the four widely-used benchmark datasets for code summarization tasks. Shi \textit{et al.}~\cite{DBLP:journals/corr/abs-2107-07112} reported that different code preprocessing operations can affect the overall performance of code-summarization models by a noticeable margin.

Therefore, it is recommended to pay more attention to data quality while training code summarization models, rather than directly reusing existing datasets without quality inspection. Based on our \textit{Finding 2}, we argue that the existing models used for code summarization tasks in the literature have a great potential of improvement given a large, good-quality dataset. \ye{see comment.}

\subsection{Implications for Data Preprocessing}

\textbf{Anti-patterns for data preprocessing.}
Based on the results of the quality assessment on benchmark datasets, we discuss the following anti-patterns to avoid when future code summarization researchers build their own datasets.
\begin{itemize}[leftmargin=4mm]
\item Assuming the first sentences of comments is representative. Rationale: the results show that using first sentences of comments is error-prone.
\item More comment is better. Rationale: the results show that we should be careful of ``what to comment'', check whether the following types of code-comment data are suitable for your scenarios: interrogative comments, auto code, duplicated code, block-comment code.
\item Over-splitting of variable identifiers in comments. Rationale: ...
\item Including under-development or out-of-date code, e.g., TODOs, commented-out methods. Rationale: ...
\item Remember to deal with abnormality, e.g., HTML tags, URL, code path, non-literal natural languages. \ye{This sentence needs to be converted into the description of an anti-pattern.}
\end{itemize}

\textbf{Need for researches on inconsistent code-comment detection.}
In this study, we define 12 categories of noisy data in code summarization datasets, and apply our cleaning tool to filter them out. For the filtered data, there might exist other hidden noises in that comments are not consistent with the code in benchmark datasets, e.g., obsolete comments during software evolution. Based on our manual inspection on the 1,174 code-comment pairs that are annotated as non-noisy comments in section 3, 14 of them are such inconsistent pairs. \ye{See comment.}
The following example shows a spotted inconsistency between the comment and its code. The comment explicitly states the return fact but the code does not.
Since only a few approaches are proposed for detecting inconsistent Java code-comment pairs  \cite{DBLP:journals/sqj/CorazzaMS18,DBLP:conf/iwpc/RabbiS20}, and there is a lack of inconsistent code-comment detection for other programming languages, such as Python, and C\#, it is quite challenging to assess and clean such noises in a parallel corpus. Therefore, there is a need for researches on comprehensive inconsistent code-comment detection, which can further benefit the quality improvement of code-comment datasets.

\begin{lstlisting}
/* Read information object and return pointer */
public void readInformationObject(...){
  try {objectDecoder.checkResolved(infoObj);
  } catch (  final Exception e) {
    LogWriter.writeLog("Exception: " + e.getMessage());
  ...
 \end{lstlisting}
    \vspace{-2mm}
\begin{small}
\texttt{Comment (TLC): read information object and \textcolor{red}{return pointer}}\\
\texttt{Code (TLC): public \textcolor{red}{void} read information object...}
\end{small}
\vspace{2.5mm}

\textbf{Considerable noise of duplicated code.}
Despite the debate on the harmfulness of code clone (i.e., duplicated code) in a software project, i.e., on the one hand-code cloning is considered a harmful practice for software maintenance as it requires consistent changes of the entities that share the same cloned fragment~\cite{lozano2007evaluating}, on the other hand, code cloning can often be used in a positive fashion such as maintaining system stability and tracking code ownership~\cite{kapser2008cloning}.
In this work, we consider duplicated code as noise and can be harmful to code summarization models. First of all, it is inline with the data preprocessing principle of learning-based models, i.e., the training set should not contain samples in the test set.
Second, keeping code duplication can result in model performance and generalizability not being comprehensively evaluated when applied the models to different benchmark datasets that do not share duplicates.\ye{not following the previous sentence.} Therefore, it is considerable to define duplicated code as noisy data.
To facilitate other researches that treat them as normal data, we make the duplicate-code rule switchable in our cleaning tool. \ye{See comment.}

\subsection{Application}\ye{Why "application"? should we rename it to "Tool implementation"?}

We release the implementation of our code-comment cleaning tool as a third-party Python library, named Code-comment cleAning Tool (CAT)\cite{cat_pypi}\ye{do we want to move the name of the tool to the early section?}, which can be easily integrated into the development pipeline of any code summarization model. The features of CAT are:
(1) \textbf{Configurable Rules}. The ruleset in CAT is configurable, which allows users to customize the rules based on the different data characteristics. Besides, users can easily extend the machine learning features that we defined in the model-based semantic cleaning tool by designing new features or removing existing features.
(2) \textbf{Support for Multiple Programming Language}. CAT is a language-independent tool that can support multiple programming languages such as java, python, and c#. Similarly, for applying CAT to other programming language datasets, users can replace the existing language-specific rules with the new rules.
We also release the distilled four benchmark datasets on our website~\cite{website} to facilitate future code summarization research.

In this study, we explore the data quality issues for code summarization exclusively. We believe the problems we found also exist in other tasks on the intersection of Natural Language Processing and Software Engineering, such as {commit message generation~\cite{jiang2017automatically} (generates a natural language summarization for each submitted code change), code search~\cite{gu2016deep} (generates API usage sequences for a given natural language
query), and code synthesis~\cite{wei2019code} (synthesizes code based on natural language intents), etc. \ye{the previous sentence is very long, and seems like a conclusion.}
Similar to code summarization, the above tasks also require datasets that contain a large number of code and natural language description pairs to train their models, the quality of their datasets can have critical impact on the performance of models. Thus, we recommend future research on these topics should also apply CAT to remove potential data preprocessing noises.}
In addition, we believe CAT can also facilitate the downstream tasks with building large pre-trained {code models~\cite{karmakar2021pre} to
learn code representations,
which require a large corpus of code. CAT can help remove code noises as listed in Table~\ref{tab:taxonomy}.}

\subsection{Threats to Validity}

One threat to validity relates to the random sampling process. Sampling may lead to incomplete results, \textit{e.g.}, noise taxonomy, we plan to enlarge the analyzed dataset and inspect whether new types of noises are emerging. Moreover, our heuristic rules for data cleaning are elaborated from the four popular benchmark datasets, covering Java and Python. Although we generally believe all similar code-comment datasets may benefit from our cleaning tool, future studies are needed to focus on datasets with other programming languages.

The second threat might come from the process of manual annotation and card sorting. We understand that such a process is subject to introducing mistakes. To reduce that threat, we establish a labeling team, and perform multiple rounds of labeling to make sure that all participants achieve conceptual coherence about noisy categories.

The third threat relates to the BLEU that is used to evaluate the performance of code summarization models. Recent researchers have raised concern over the use of BLEU \cite{DBLP:conf/sigsoft/RoyFA21}, warning the community that the way BLEU is used and interpreted can greatly affect its reliability. To mitigate that threat, we also adopt other metrics, i.e., ROUGE, METEOR, and CIDEr when evaluating performance.

Another threat to validity is the replication of each model. To ensure that the experimental results are consistent with their papers, we retrain the models using the source code provided by the authors, and reuse the parameters provided by the authors.
We compare the performance of our re-trained models with the performance reported in the papers. Their performances are comparable. For example,  the METEOR score of Rencos is 21.1 on {\texttt{PCSD}} in their paper, and our re-train Rencos model is 20.2.

\section{Introduction}
\vspace{0.05cm}

Code summarization concerns the production of a natural-language description of source code that facilitates
software development and maintenance by enabling developers to comprehend, ideate, and document code effectively.
Learning-based models have been widely leveraged for the advantages in semantic modeling and understanding of languages.
Similar to many other learning tasks, code summarization models require large-scale and high-quality training datasets. To that end, multiple benchmark datasets for code summarization tasks have been constructed from real-world project repositories, \textit{e.g.,} GitHub, and are popularly used {in many code summarization studies}. For example, {\texttt{Funcom}}~\cite{leclair2019neural} was released with over 2.1M code-comment pairs from over 29K Java projects in the Sourcerer repository. Many code summarization models, such as Re$^{2}$Com~\cite{wei2020retrieve}, DeepSumm~\cite{DBLP:journals/corr/abs-2004-00998}, and EditSum~\cite{DBLP:conf/kbse/LiL000J21}, are trained and evaluated to be relatively effective on it.
Similar popular datasets include {\texttt{TLC}}~\cite{hu2018summarizing}, {\texttt{CSN}}~\cite{husain2019codesearchnet}, and {\texttt{PCSD}} ~\cite{wan2018improving}.

Although the benchmark datasets are expected to be of good quality,
noise is inevitable due to the differences in coding conventions and assumptions employed in modern programming languages and IDEs, as well as ad hoc nature of development processes and practices~\cite{DBLP:conf/scam/RaniASBN21}.
For example, source code in GitHub is contributed by developers all around the world, thus their comments are likely to contain multiple natural languages that can lead to increases in complexity regarding the understanding and maintenance of source code.
Existing studies also have confirmed the existence of many different types of noise in various benchmark datasets, such as auto-generated code~\cite{DBLP:conf/acl/HasanMIMHHAIS21}, ``TODO'' comments~\cite{DBLP:journals/tosem/ChenXHLL21}, and incomplete comments~\cite{DBLP:journals/corr/abs-2107-07112}, despite their data cleaning efforts.
{Particularly, Steidl \textit{et al.}~\cite{DBLP:conf/iwpc/SteidlHJ13} analyzed five open source projects, and reported that nearly one third of the comments do not promote system understanding.}

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth, height=2.8cm]{fig/overview_0314.pdf}
\caption{Overview of our research methodology.}
\label{fig:overview}
\end{figure*}

To investigate the aforementioned concerns of data quality for code summarization, we conduct a systematic study to assess and improve the quality of four widely-used benchmark datasets, \textit{i.e.}, {\texttt{Funcom}}, {\texttt{TLC}}, {\texttt{CSN}}, and {\texttt{PCSD}}. The research methodology overview consists of four main steps, as illustrated in Figure~\ref{fig:overview}.
First, we propose a taxonomy of 12 different types of data noises due to inappropriate or insufficient data preprocessing in code summarization,
derived from observations on the selected four benchmark datasets.
Second, we build a rule-based cleaning tool, named {CAT} (Code-comment cleAning Tool), for {automatically scanning and} detecting {the occurrences and distribution of data} noises for a given dataset, based on the proposed taxonomy.
The manual {validation}
results show that the tool can accurately detect noisy data.
{Third,}
we conduct an evaluation study to assess the data quality of the four widely-used benchmark datasets. The results show that noisy data extensively exist in the four benchmark datasets (ranging from 31\% to 66\%).
{Finally,} we investigate the impacts of noises on three typical code summarization models (\textit{i.e.} NNGen~\cite{liu2018neural}, NCS~\cite{ahmad2020transformer}, and Rencos~\cite{zhang2020retrieval}) by comparing their performance trained on the same datasets before and after data cleaning.
The above four steps will be elaborated in later sections, \textit{i.e.}, sec. 3 to sec. 6, respectively.
The results show that, removing noisy data have a positive influence on model summarization ability.
Training three existing models with the filtered benchmark datasets improves the BLEU-4 by 27\%, 21\%, and 24\%, respectively.
The major contributions of this paper are as follows.
\begin{itemize}[leftmargin=4mm]
    \item To the best of our knowledge, it is the first to systematically study the {patterns and} impact of the noises in various code summarization datasets.
   \item We develop an automated data cleaning tool, named {CAT}, for code summarization datasets, which can help distill high-quality code-comment data.
    \item We perform a comprehensive assessment on data quality of benchmark datasests, which provides practical insights for future code summarization research.
    \item
    We conduct a comparative analysis on the performance of code summarization models trained on the origin and distilled benchmark datasets, our results demonstrate that removing noises yields significant model performance improvement.
     \item We release {CAT} and the distilled benchmark datasets~\cite{website} to the general public, in order to facilitate the replication of our study and its extensive application in other contexts.
\end{itemize}

In the remainder of the paper, Section~\ref{sec:pre} illustrates the preliminaries. Section~\ref{sec:tox} introduces the taxonomy of noisy data. Section~\ref{sec:cleaningtool} presents the code-comment cleaning tool. Section~\ref{sec:qassessment} demonstrates the quality assessment of benchmark datasets. Section~\ref{sec:modelperformance} shows the impact of noisy data on the performance of code summarization. Section~\ref{sec:threats} discusses results and threats to validity. Section~\ref{sec:relatedwork} surveys the related work to our study.  Section~\ref{sec:conclusion} concludes this paper.

\section{Preliminaries}
\label{sec:pre}

This section briefly introduces the literature of code summarization, as well as four widely-used benchmark datasets.

\subsection{Code Summarization}
\label{sec:codeSum}

{Code summarization~\cite{moreno2013automatic,sridhara2010towards} aims at generating a comment for a given block of source code that can help developers better understand and maintain source code.}
The essential task is to translate the code written in programming languages into {comments} written in natural languages. Meanwhile, {comments} may describe not only the functions, but also the design intents, program logic, and functionalities of programs behind the source code.
The existing code summarization models can be categorized into three different types based on the techniques used, \textit{i.e.}, Information Retrieval (IR) based approaches~\cite{eddy2013evaluating,haiduc2010supporting,wong2015clocom}, Neural Machine Translation (NMT) based approaches~\cite{ye2020leveraging,wan2018improving,wei2019code,leclair2020improved,haque2020improved,chen2018neural,cai2020tag,alon2018code2seq,ahmad2020transformer,iyer2016summarizing,wei2020retrieve}, and hybrid approaches~\cite{yuchao2021yet,hu2018deep,hu2020deep,leclair2019neural} that combine IR and NMT techniques.

Specifically, IR-based code summarization models use IR techniques to extract keywords from the
source code and compose them into term-based summarization for a given code snippet~\cite{eddy2013evaluating,haiduc2010supporting,wong2015clocom}. For example, Edmund \textit{et al.}~\cite{wong2015clocom} generated code summarization for a given code snippet by retrieving the replicated code samples from the corpus with clone detection techniques.
Recently, with the booming of deep learning techniques, many NMT based code summarization approaches have been proposed, which train the neural models from a large-scale code-comment corpus to automatically generate summaries~\cite{ye2020leveraging,wan2018improving,wei2019code,leclair2020improved,haque2020improved,chen2018neural,cai2020tag,alon2018code2seq,ahmad2020transformer,iyer2016summarizing,hu2018deep,wei2020retrieve}. For example, Iyer \textit{et al.}~\cite{iyer2016summarizing} treated the code summarization task as an end-to-end translation problem and first introduced NMT into code comment generation. The hybrid approaches~\cite{zhang2020retrieval,yuchao2021yet,hu2020deep,leclair2019neural} leverage the advantages of IR and NMT techniques for improving code summarization. For example, Zhang \textit{et al.}~\cite{zhang2020retrieval} first retrieved top similar code snippets in the training data for a given piece of code and then input them into an NMT model for summarization generation.

\subsection{Benchmark Datasets}
\input{table/dataset_intro_v2}
\label{sec:bench}
{As introduced earlier, this study} conducts various experiments on four widely-used code summarization datasets, including {\texttt{Funcom}}~\cite{leclair2019neural}, {\texttt{TLC}}~\cite{hu2018summarizing}, {\texttt{CSN}}~\cite{husain2019codesearchnet}, and {\texttt{PCSD}}~\cite{wan2018improving}.
{The data format of these datasets is primarily represented using \textbf{code-comment pairs}, where the code data is at the granularity of \textbf{method-level}. Each dataset applies its own operations when extracting and preprocessing the raw data.}
Table~\ref{table:dataset_intro} {summarizes the information of descriptive metadata and associated studies where each dataset has been employed in existing literature}.

{More specifically, }
\textbf{{\texttt{Funcom}}} is a collection of 2.1M code-comment pairs from 29K projects. For each method, it extracted its Javadoc comment and treated the first sentence in the Javadoc of each method as its summary.
\textbf{{\texttt{TLC}}} has 87K code-comment pairs collected from more than 9K open-source Java projects created from 2015 to 2016 with at least 20 stars. It extracted the Java methods and their corresponding Javadoc comments. These comments are considered as code summaries.
\textbf{{\texttt{CSN}}}
contains about 2M method and comment pairs mined from publicly available open-source non-fork GitHub repositories spanning six programming languages, \textit{i.e.}, Go, Java, JavaScript, PHP, Python, and Ruby.
{In this study, we conduct the experiments on the Java portion of the {\texttt{CSN}} dataset.}
\textbf{{\texttt{PCSD}}} contains 105K pairs of Python functions and their comments from open source repositories in GitHub. Specifically, it uses docstrings (\textit{i.e.}, the string literals that appear right after the definition of functions) as summaries for Python functions.

\section{The Taxonomy of Noisy Data}
\label{sec:tox}

{An essential and effective starting point is a systematic and robust categorization of data noises. This section presents details on how the noisy data taxonomy is built, and the descriptions and examples for every 12 categories.}

\subsection{Taxonomy Construction}
\label{sec:3.1}
We employ an \textit{open card sort} \cite{DBLP:journals/es/RuggM05} process by involving nine participants.
{Participants include two PhD students, four master students, and three senior researchers.
All of them have done either intensive research work with software development or have been actively contributing to open-source projects.}
The sorting process is conducted in multiple rounds.
{For each round, we randomly sample 160 code-comment pairs without replacement from the four benchmark datasets (40 pairs for each).}
In the first round,
all participants label the same sampled data,
with an intensive discussion session to achieve conceptual coherence about noisy categories. The average Cohen's Kappa is 0.86, which indicates substantial agreement.
Then, a shared pool of categories is utilized and carefully maintained, and
each participant could select existing categories from and/or add new category names into the shared pool. The sorting process ends when there is no new category added for two consecutive rounds. In total, we conducted 10 rounds and labeled 1,600 pairs of source code and the corresponding comments (400 pairs for each of the four benchmark datasets).
The detailed annotation results
can be found in Section~\ref{sec:4.2.3}.

\input{sec/taxonomy-2}

\section{The Code-Comment Cleaning Tool}
\label{sec:cleaningtool}

{To support automatic detection {of noises in} the proposed taxonomy, we develop a code-comment cleaning tool, named {CAT}, based on a set of heuristic rules.
This section introduces the design of the rule-based cleaning tool, and presents the analysis results of its effectiveness.}

\subsection{The Heuristic Rules}
\input{table/RQ1_v4}

\textbf{Construction criteria.}
The heuristic rules conform to the following criteria: (1) Each rule should define a unique and specific category without overlap; (2) Rules should limit the exclusion of valid data within an acceptable range, {\textit{i.e.}, all the F1 scores should be larger than 90\%};
and (3) Any rule is not a subrule of the others.

\begin{figure}[t]
\centering
\vspace{0.1cm}
\includegraphics[width=0.8\columnwidth]{fig/alg-v2.pdf}
\vspace{-0.3cm}
\caption{If-else rules of collecting the first sentence in the raw comments for partial and verbose sentence noises.}
\vspace{-0.3cm}
\label{fig:3-flow}
\end{figure}

\textbf{Construction process.}
For each category of noisy data, we develop a set of if-else rules to detect them by the following steps.
(1) Based on the manually annotated noises produced in Section~\ref{sec:tox}, we carefully identify syntax features for each category from 80\% of the manual data;
(2) We design a set of if-else rules to {detect}
the noisy data from the raw data (note that, the raw data refers to the source data that has not been processed for use, and the origin data refers to the processed raw data in the benchmark datasets);
(3) To avoid overfitting, we test the correctness of the rules on the rest 20\% of the manually annotated noises.
We iteratively adjust the if-else rules until the performance is acceptable, \textit{i.e.}, over 90\% F1 score.

\textbf{Example Rules.}
Figure \ref{fig:3-flow}
illustrates the if-else rules of collecting the first sentence in the raw comments for partial and verbose sentence noises. The key idea is to sequentially determine whether each line of comment in a raw comment contains a complete sentence. If so, return the first complete sentence; if not, save the content of the line and continue to determine the next line.
By comparing the first sentence we extracted from raw data with the processed sentence provided in the benchmark datasets, we can determine the verbose or partial sentence category.

Table \ref{tab:taxonomy} demonstrates the syntax feature of heuristic rules and our actions {to resolve noises detected}.
Details of the implementation of each category can be found on our website \cite{website}.

\subsection{Effectiveness Evaluation}

\subsubsection{Data Preparation.}
As introduced in Section~\ref{sec:3.1}, we labeled 12 categories of noisy data from 1,600 code-comment pairs sampled from the four benchmark datasets.
These manually annotated data are used to evaluate the performance of CAT.
We build the heuristic rules based on observing 80\% of the annotated noisy data, and evaluate on the rest 20\%.
The ``Dataset'' column in Table \ref{table:rq2_dataset} shows the detail.

\subsubsection{Evaluation Metrics.}

We use three commonly-used metrics to evaluate the performance of CAT, \textit{i.e.}, \textit{Precision}, \textit{Recall}, and \textit{F1}. (1) \textit{Precision} refers to the ratio of correct predictions to the total number of predictions; (2) \textit{Recall} refers to the ratio of correct predictions to the total number of samples in the golden test set; and (3) \textit{F1} is the harmonic mean of precision and recall. When comparing the performances, we care more about F1 since it is balanced for evaluation.

\input{table/RQ2_dataset_v5}
\subsubsection{Results}
\label{sec:4.2.3}

Table \ref{table:rq2_dataset} demonstrates the performance of {CAT}. We can see that, it can accurately detect noises on the four benchmark datasets. The F1 scores of detecting comment-related noises are ranging from 93.0\% to 100.0\%,  and 95.5\% on average.
The average F1 scores of detecting code-related noises are ranging from 95.6\% to 100.0\%,  and 98.3\% on average.
The results show that, {CAT} can achieve highly satisfactory performance on filtering noisy data from code-comment datasets.
In summary, our code-comment cleaning tool can
{accurately} filter noisy data, with all the F1 scores of over 90.0\%, {which can help build a high-quality dataset for the follow-up code summarization tasks.}

\section{Quality Assessment of Benchmarks}
\label{sec:qassessment}

\input{table/heat_v3}

In this step, the code-comment cleaning tool is applied to detect and correct noises through comment removal or update actions as listed in Table 2. Based on the noisy data output by the tool, we further analyze the quality of the four benchmark datasets.
Table~\ref{tab:heat} illustrates the distribution of each noise category on the four benchmark datasets.
The number on each cell presents the percentage of the noises in the corresponding benchmark dataset, directly generated from the cleaning tool. Note that, since one code-comment pair may involve multiple noises, the tool repeatedly counts those that involve multiple noise categories when calculating frequency for each category, and counts once for the total frequency. Thus, the sum of individual category percentages is slightly higher than the percentage of total noises.

\textbf{Overall,}
{\texttt{Funcom}} has the highest proportion of noisy data (65.8\%), followed by {\texttt{TLC}} (41.9\%), {\texttt{CSN}} (37.2\%), and {\texttt{PCSD}} (31.2\%).
We can also observe that, the benchmark datasets often contain multiple categories of noises.
{\texttt{Funcom}} contains the most noise categories. Except for the verbose sentence noises, every other category is included. The other three benchmark datasets contain seven or eight categories.

\textbf{Noise distribution in comments}. It is observed that
40.9\% comments in {\texttt{Funcom}} contain noises, followed by {\texttt{CSN}}, {\texttt{TLC}}, and {\texttt{PCSD}}.
Specifically, all the four benchmark datasets have content-tampering, interrogation, and under-development noisy comments.
24.4\% comments in {\texttt{CSN}} {are contaminated by the meaningless text such as \textit{HTML} tags, \textit{Javadoc} tags, or \textit{URLs}.}
24.1\% comments in {\texttt{Funcom}} are over-spliting by camelCase.
22.8\% comments in {\texttt{TLC}} are verbose sentences.

\textbf{Noise distribution in source code.} It is also observed that 40.7\% source code in {\texttt{Funcom}} contain noises, followed by {\texttt{TLC}}, {\texttt{PCSD}}, and {\texttt{CSN}}.
Specifically, all the four benchmark datasets have auto-code noises.
In {\texttt{Funcom}}, 29.8\% code is auto-generated such as \textit{setter}, \textit{getter}, \textit{tester}, and \textit{toString} methods.
Indeed, previous research~\cite{DBLP:conf/acl/HasanMIMHHAIS21} used to complain about similar issues that {{\texttt{Funcom}} contains much auto-generated code.}
In {\texttt{TLC}}, 18.4\% code is exactly duplicated while the other three benchmark datasets are nearly none. This phenomenon indicates that preprocessing operations applied on existing benchmark datasets are not coincident all the time in that, some benchmark datasets apply the dedup preprocessing operation while some do not.

\textbf{Distribution of updates and removals.}
The bottom part of Table~\ref{tab:heat} shows the frequency of different types of noise that were removed or updated from the four benchmark datasets based on {the corrective} actions introduced in Table \ref{tab:taxonomy}.
{\texttt{Funcom}} and {\texttt{TLC}} have high proportions of both removals and updates, \textit{i.e.}, 38.7\% and 27.1\% noisy data in {\texttt{Funcom}} are removed and
updated respectively.
The major correction for {\texttt{CSN}} is removals. While the major correction for {\texttt{PCSD}} is updating noises. This might be caused by the different noise distributions in these two benchmark datasets.

\begin{center}
\begin{tcolorbox}[colback=gray!10
                  colframe=black
                  width=8.3cm
                  arc=1mm, auto outer arc,
                  boxrule=0.5pt,
                  left=3pt,
                  right=3pt,
                  top=3pt,
                  bottom=3pt
                 ]
\textbf{Finding 1}: Noisy data extensively exist in the four widely-used benchmark datasets, ranging from 31.2\% to 65.8\%. 29.8\% of the code in {\texttt{Funcom}} is auto-generated; 22.8\% comments in {\texttt{TLC}} are verbose first sentences; 24.4\% comments in {\texttt{CSN}} are contaminated by the meaningless text; and 15.9\% comments in {\texttt{PCSD}} are the partial first sentences.
\end{tcolorbox}
\end{center}

\

\section{Impacts on the Performance of Code Summarization}
\label{sec:modelperformance}
In this section, we investigate the impact of noisy data on the performance of code summarization. Specifically, we choose three state-of-the-art code summarization models and train these models on three versions (\textit{i.e.}, original, controlled, and filtered) of each benchmark dataset.  Thus, we have 3 (the number of models) $\times$ 4 (the number of benchmark datasets) $\times$ 3 (the number of versions per benchmark dataset) = 36 experimental models in total. We evaluate the performance of all the models based on commonly-used metrics for code summarization tasks.

\subsection{Experimental Design}

\subsubsection{Data Preparation}
We use three versions of each benchmark dataset as training sets, as shown in Table \ref{tab:exp_and_total_v2}.
The ``Total'' rows illustrate the overall data before and after being distilled by our tool.
The ``Experimental'' rows show the data that are used for our experiments.
The ``Origin" refers to the original training dataset split by the benchmark dataset.
The ``Filtered" refers to the train/test dataset cleaned by our tool from the ``Origin" train/test datasets. To benchmark the performance variation brought by the size shrinking, we further build
the ``Controlled" set by randomly sampling from the ``Origin" set, which has an equal amount of data instances as the ``Filtered" dataset.

\input{table/exp_and_total_v2}
\input{table/RQ3_result_v4}

\subsubsection{Code Summarization Models} {As introduced in Section~\ref{sec:codeSum}, existing code summarization models can be divided into three categories}: Information Retrieval (IR) based approaches, Neural Machine Translation (NMT) based approaches, and hybrid approaches that combine IR and NMT techniques. We select one state-of-the-art method from each category to explore the impact of noisy data on model performance.

\textbf{NNGen~\cite{liu2018neural}} is an IR-based model for generating commit messages by utilizing the nearest neighbors algorithm. It first embeds code changes into vectors based on the bag of words and the term frequency. Then, NNGen retrieves the nearest neighbors of code changes by calculating the cosine similarity of vectors and the BLEU-4 score. Finally, it directly chooses the message of the code change with the highest BLEU score as the final result.

\textbf{NCS~\cite{ahmad2020transformer}} is an NMT-based model which replaces the previous RNN units with the more advanced Transformer~\cite{vaswani2017attention} model. NCS extends the vanilla Transformer in two aspects. Firstly, it incorporates the copying mechanism~\cite{see2017get} in the Transformer to allow both generating words from vocabulary and copying from the input source code. Secondly, NCS utilizes relative positional embedding rather than absolute positional embedding to capture the semantic representation of the code better.

\textbf{Rencos~\cite{zhang2020retrieval}} is a state-of-the-art model that combines the advantages of both IR-based and NMT-based techniques. Specifically, given an input code snippet, Rencos first retrieves its two most similar code snippets in the training set from the aspects of syntax and semantics, respectively. Then it encodes the input and two retrieved code-snippets, and generates the summary by fusing them during decoding.

\subsubsection{Experimental Settings.}
The experimental environment is a desktop computer equipped with an NVIDIA GeForce RTX 3060 GPU, Intel Core i5 CPU, 12GB RAM, running on Ubuntu OS. When training the three code summarization models with the benchmark datasets, we follow the implementation provided in their original papers, and adopt the recommended hyperparameter settings, except for the training epoch of NCS and Rencos. To save training time and computation resources, we set \textit{max\_epoch}=50 for NCS and \textit{max\_iteration}=100k for Rencos.

\subsubsection{Evaluation Metrics}
We evaluate the performance of the three models using four metrics including BLEU~\cite{DBLP:conf/acl/PapineniRWZ02}, METEOR~\cite{DBLP:conf/acl/BanerjeeL05}, ROUGE-L~\cite{lin2004rouge}, and CIDEr~\cite{DBLP:conf/cvpr/VedantamZP15}.
\textbf{BLEU} measures the $n$-gram precision by computing the overlap ratios of $n$-grams and applying brevity penalty on short translation hypotheses.
BLEU-1/2/3/4 correspond to the scores of unigram, 2-grams, 3-grams, and 4-grams, respectively.
\textbf{ROUGE-L} is defined as the length of the longest common subsequence between generated sentence and reference, and based on recall scores.
\textbf{METEOR} is based on the harmonic mean of unigram precision and recall, with recall weighted higher than precision.
\textbf{CIDEr} considers the frequency of $n$-grams in the reference sentences by computing the TF-IDF weighting for each $n$-gram. ${CIDEr}_n$ score for $n$-gram is computed using the average cosine similarity between the candidate sentence and the reference.

\subsection{Quantitative Results}
Table~\ref{table:rq3_result} shows the performance of the three models trained over different experimental datasets.
Overall, removing noisy data {from the training set} in the four benchmark datasets produces a positive effect on improving the performance of the three models. {Training three existing models with the filtered benchmark datasets improves the BLEU-4
by 26.9\%, 20.7\%, and 24.1\%, ROUGE by 19.1\%, 11.3\%, and 15.7\%, METEOR by
18.9\%, 7.1\%, and 15.7\%, CIDEr by 46.1\%, 19.4\%, and 33.2\%, respectively.}

Among the four benchmark datasets, the effect on the {\texttt{CSN}} dataset is the most significant,
which leads to the three models (NNGen, NCS, and Rencos) increasing by 31.0\%, 22.0\% and 42.1\% on BLEU-4, 28.4\%, 12.5\% and 40.4\% on ROUGE, 13.2\%, 10.9\%, and 38.3\% on METEOR, and 62.5\%, 18.8\%, and 67.3\% on CIDEr, respectively. This is followed by {\texttt{TLC}} and {\texttt{PCSD}}. Considering the fact that even the least effect obtained in {\texttt{Funcom}} still contributes to an increase of 10.3\%, 16.0\% and 13.6\% on BLEU-4, 0.7\%, 1.7\% and 1.7\% on ROUGE, 5.2\%, 1.1\%, and 1.3\% on METEOR, and 9.5\%, 5.8\%, and 4.8\% on CIDEr, respectively.
{The main reason that the three models exert the biggest performance difference on CSN is that, the primary noisy data on CSN are content tampering by HTML tags, and removing these noises will make the generated comments more accurate. We will illustrate this in the following qualitative analysis.}
We argue that the existing models used for code summarization tasks in the literature have a significant scope of improvement given a large, good-quality dataset.

By observing the performance of the three models trained on different filtered datasets, we find that the relative ranking among the three types of models is not consistent. For the filtered {\texttt{TLC}}, Rencos achieves the best performance on all metrics compared to the other two models. While NCS performs best when trained on filtered {\texttt{Funcom}}, {\texttt{CSN}}, and {\texttt{PCSD}}. This result implies that, to more comprehensively evaluate different code summarization models, it is better to use multiple datasets, as the ranking of the model can be inconsistent on different datasets.

\begin{center}
\begin{tcolorbox}[colback=gray!10
                  colframe=black
                  width=8.3cm
                  arc=1mm, auto outer arc,
                  boxrule=0.5pt,
                  left=3pt,
                  right=3pt,
                  top=3pt,
                  bottom=3pt
                 ]
\textbf{Finding 2}: Removing noisy data {from the training set} in the four benchmark datasets has a positive influence on the performance of the models.
{Training three existing models with the filtered benchmark datasets improves the BLEU-4 by 26.9\%, 20.7\%, and 24.1\%, respectively.}
\end{tcolorbox}
\end{center}

\subsection{Qualitative Analysis}
\label{sec:quaAna}
To qualitatively illustrate the impact of the noises on code summarization models, we present two cases generated by the three models trained on different datasets, as shown in Figure~\ref{fig:cases}.
\textbf{Overall}, the comments generated by the models trained on the distilled datasets tend to be more accurate and more readable than the comments generated by the models trained on the origin datasets.

\textbf{Case 1.}
Given the code, the comments generated by the three models trained on origin benchmark datasets are ``returns the value for the cell at code column index code and''. Compared with the human-written comment, we consider the following three errors are likely related to noisy data:
(1) The over-splitting ``columnindex'' as ``column index''. This error is likely to be caused by the over-splitting of variable identifiers in the comment;
(2) The redundant ``code'' around ``\textit{columnindex}''. It is mainly due to the fact that the origin datasets contain many unremoved HTML tags, thereby increasing the probability of HTML tags, \textit{e.g.}, ``<code>'', appearing in the context, making it easier for the model to generate these HTML words when encountering some specific patterns;
(3) The redundant ``and'' in the end. This is mainly because the widely-existing partial or verbose sentences in training sets would hinder the ability of the model to learn to determine when the generation process should end.
After being retrained on the distilled data, the three models can accurately generate the comments, where the BLEU-4 has an 87.5\% increase, from 53.32 to 100.00.

\begin{figure}[tbp]
\centering
\subfigure[Case 1: An example of  generated comments that contains HTML tags, over-splitting variable, and unfinished sentence]{
    \label{fig:example-2}
    \includegraphics[width=\columnwidth]{fig/example-1_v2.pdf}

}
\subfigure[Case 2: An example of  generated comments that contains HTML tags]{
    \label{fig:example-1}
    \includegraphics[width=\columnwidth]{fig/example-2_v2.pdf}
}
\vspace{-0.35cm}
\caption{Inaccurate comment generation affected by noises}
\label{fig:cases}
\vspace{-0.55cm}
\end{figure}

\textbf{Case 2.} Given the code, the comments generated by the three models trained on origin benchmark datasets have two redundant ``em'', which are caused by the unremoved HTML tag ``<em>'' that is
used to define emphasized text.
After being retrained on the distilled data, the three models can accurately generate the comments, where the BLEU-4 has a 90.3\% increase, from 52.54 to 100.00.

\section{Discussion}
\label{sec:threats}
In this section, we discuss several interesting implications that are derived from the results of this study, aiming to facilitate the code summarization research and the SE community.

\subsection{Impact of Noises on Code Summarization Datasets and Models}
\subsubsection{Impact of Noises on Datasets.}
\textbf{(1) Removing the noises in Funcom leads to a slight improvement in model performance} (i.e., the BLEU-1 score increases 2.46\% on average). It might be because that Funcom is the one with the most auto-generated code, and auto code offers "easy gain" in performance that is not available anymore. Therefore, the baseline performance could actually decrease if removing them from testset, thus making the improvements of other rules look smaller in comparison.\textbf{ (2) Removing the noises in TLC, CSN, and PCSD leads to a considerable improvement in performance} (i.e., the BLEU-1 score increases 24.1\%, 30.6\%, and 19.5\% on average respectively. It might be because the major noises of these three datasets are `Verbose Sentence', `Content Tampering', and `Partial Sentence', respectively, and removing them will benefit the models.

\subsubsection{Impact of Noises on Models.}
Based on the results shown in Table 4 and Table 6, we can observe that the noises affect code summarization models differently.
\textbf{(1) Removing `Verbose Sentence' noises might largely benefit the IR-based model NNGen.} The major noises in TLC are `Verbose Sentence', and removing them leads to the performance of NNGen increases 53.25\% on average of all the seven metrics, followed by Rencos (21.74\%) and NCS (12.17\%). It might be because the NNGen model directly outputs the retrieved results. Taking the retrieved verbose comments as the output leads to a substantial decline in the scores of the evaluation metrics, since these N-gram based metrics are less beneficial for longer comments.
\textbf{(2) Removing `Content Tampering' noises might largely benefit the hybrid model Rencos.} The major noises in CSN are `Content Tampering', and removing them leads to the performance of Rencos increasing 48.75\% on average, followed by NNGen (35.11\%) and NCS (15.73\%).
It might be because the hybrid model Rencos employs a more complex input including the test code and the retrieved data. Such models' effective training typically requires the ground-truth comments to be semantically correct. However, the `Content Tampering' noises cause the ground-truth comments to mingle with irrelevant text such as HTML tags or URLs, which alter the semantics of the ground-truth comments. Therefore, when removing the `Content Tampering' noises, the hybrid model Rencos increases the most.
\textbf{(3) Removing `Partial Sentence' noises might largely benefit the NMT-based model NCS.} The major noises in PCSD are `Partial Sentence', and removing them leads to the performance of NCS increasing 28.56\% on average, followed by NNGen (21.58\%) and Rencos (14.77\%). The main reason is that the comments belonging to the `Partial Sentence' noise are not complete sentences, and thus lack integrity for both syntactic and semantic, which hinders language models like NCS from learning the high-level syntactic and semantic information correctly.
We will further illustrate the impact of noises in qualitative cases.

\subsection{Lessons Learned of Data Preprocessing for Code Summarization}

Data quality has been a growing concern, especially since deep learning (DL) is widely applied for massive SE tasks. As DL models typically require high-volume data, ensuring data quality at a large scale has become a compelling need.
Most existing studies focus on advances in modeling but typically overshadow the data quality. When investigating the current code summarization models (as shown in Table \ref{table:dataset_intro}), we notice that a substantial amount of data preprocessing operations are cursory or lack consistency.
In addition, there is a lack of principles or methods to guide and reinforce the data preprocessing associated effort while conducting code summarization research, regarding how to soundly preprocess benchmark datasets for different tasks.
For instance, our results show that noisy data extensively exist in the four widely-used benchmark datasets for code summarization tasks. Shi \textit{et al.}~\cite{DBLP:journals/corr/abs-2107-07112} reported that different code preprocessing operations can affect the overall performance of code summarization models by a noticeable margin.
Therefore, paying more attention to data quality while training code summarization models is recommended, rather than directly reusing existing datasets without quality inspection.

Specifically, the study and its results lead to the identification of the following lessons learned from the quality assessment on benchmark datasets, for future code summarization researchers.
\begin{itemize}
\item When reusing existing datasets, check the quality of processed data by comparing with their raw data.
\item Extracting the first sentences of comments is error-prone.
\item Avoid including under-development or obsolete code, \textit{e.g.}, TODOs, commented-out methods.
\item Avoid over-splitting of variable identifiers in comments.
\item Be careful of "what to comment", check whether the following types of code-comment data are suitable for your scenarios: interrogative comments, auto code, duplicated code, and block-comment code.
\item Remember to deal with abnormality, \textit{e.g.}, HTML tags, URL, code path, and non-literal natural languages.

\end{itemize}

\subsection{Tool Support and Potential Applications}

We release the implementation of the {CAT} code-comment cleaning tool as a third-party Python library~\cite{cat_pypi}, which can be easily integrated into the development pipeline of most code summarization models. The features of {CAT} are:
(1) \textbf{Configurable and Extendable Rules}. The ruleset in {CAT} is configurable, which allows users to customize the existing rules based on the different data characteristics. Besides, {CAT} provides interfaces enabling users to design new rules or clean functions to extend the feature of {CAT}.
(2) \textbf{Support for Multiple Programming Language}. {CAT} is a tool that can support multiple programming languages such as Java, Python, and C\#. Similarly, for applying {CAT} to other programming language datasets, users can replace the existing language-specific rules with the new rules.
We also release the distilled four benchmark datasets on our website~\cite{website} to facilitate future code summarization research.

This study applies the {CAT} tool for exploring the data quality issues for code summarization exclusively.
Similar to code summarization, there exist other tasks on the intersection of Natural Language Processing and Software Engineering, such as {commit message generation~\cite{jiang2017automatically} (generates a natural language summarization for each submitted code change), code search~\cite{gu2016deep} (generates API usage sequences for a given natural language
query), and code synthesis~\cite{wei2019code} (synthesizes code based on natural language intents).
These tasks also require datasets that contain a large number of code and natural language description pairs to train their models, the quality of their datasets can have a critical impact on the performance of models. Thus, we recommend future research on these topics should also apply {CAT} to remove potential data preprocessing noises.}
In addition, we believe {CAT} can also facilitate the downstream tasks with building large pre-trained {code models~\cite{karmakar2021pre} to
learn code representations,
which require a large corpus of code. {CAT} can help remove code noises as listed in Table~\ref{tab:taxonomy}.}

\subsection{Implications for Research and Practice}

\textbf{Need for collaborative community effort on principles of data preprocessing.}
Improving data quality takes collaborative, community effort to establish and maintain principles, methods, and tools to govern the data extraction and preprocessing pipelines. This study takes an initial step towards addressing the challenges of building high-quality datasets for code summarization exclusively. Although we have proposed quality criteria to measure data quality, methods to help collect data, and filters to remove preprocessing noises, our solutions might be not sufficient for other research tasks in software engineering or data science areas that require massive data as inputs because of the diversity of data sources, the complexity of different data structures, and the scale of data volume. Thus, we urge for collaboration and effort from the whole research community to help build a comprehensive and reliable principle set for data collection, preprocessing, and quality assessment, which we believe can benefit our research community.

\textbf{Need for research on comprehensive noisy code-comment detection.}
In this study, we define 12 categories of noisy data in code summarization datasets, and apply our cleaning tool to filter them out. For the filtered data, we observed several additional quality issues that require a deeper understanding of the content of the comments and the corresponding code.
\textbf{(1) Inconsistent code-comment pairs.}
The following example shows a spotted inconsistency between the comment and its code. The comment explicitly states the return fact, but the code does not.
Since only a few approaches are proposed
\begin{lstlisting}
/* Read information object and return pointer */
public void readInformationObject(...){
  try {objectDecoder.checkResolved(infoObj);
  } catch (  final Exception e) {
    LogWriter.writeLog("Exception: " + e.getMessage());
  ...
 \end{lstlisting}
    \vspace{-2mm}
\begin{small}
\texttt{Comment (TLC): read information object and \textcolor{red}{return pointer}}\\
\texttt{Code (TLC): public \textcolor{red}{void} read information object...}
\end{small}
\vspace{2.5mm}

\noindent
for detecting inconsistent Java code-comment pairs  \cite{DBLP:journals/sqj/CorazzaMS18, DBLP:conf/iwpc/RabbiS20}, and there is a lack of inconsistent code-comment detection for other programming languages, such as Python and C\#, it is quite challenging to assess and clean such noises in a parallel corpus.
\textbf{(2) Low-readability comments}. We noted that some comments are not fluency or have syntactic errors. E.g., a comment in the PCSD dataset ``transforms a doc in content in one document in presentation''.
If trained on datasets with such comments, the end-to-end code summarization models are also at risk of producing low-readability comments.
\textbf{(3) Less-informative comments}. We found that many methods in benchmark datasets do not need comments as methods are self-explainable with their names. For example, two methods in Funcom dataset are named "renderImgTag" and "createTextPane", and their corresponding comments are "render img tag" or "create the text pane". Since such comments are highly similar to the method names, they can hardly convey additional information for better understanding of the source code. If trained on datasets with such comments, the code summarization models are also
likely to produce less-informative comments.
Therefore, there is a need for research on comprehensive noisy code-comment detection, which can further benefit the quality improvement of code-comment datasets.

\textbf{Integrating tool support to aid publication peer review.}
Following the open science policies~\cite{open-science-policies}, most existing research makes their raw and transformed data publicly accessible during the peer review process. For those datasets that are in the format of code-comment pairs, the {CAT} cleaning tool can be used to automatically detect their noise data in a reasonable time. The output statistic could objectively reflect the inside quality of the open datasets to some extent, thus can help professional peer reviewers to infer the quality and reliability of the under-review research.

\subsection{Threats to Validity}

One threat to validity relates to the random sampling process. Sampling may lead to incomplete results, \textit{e.g.}, noise taxonomy, we plan to enlarge the analyzed dataset and inspect whether new types of noises are emerging. Moreover, our heuristic rules for data cleaning are elaborated from the four popular benchmark datasets, covering Java and Python. Although we generally believe all similar code-comment datasets may benefit from our cleaning tool, future studies are needed to focus on datasets with other programming languages.

The second threat might come from the process of manual annotation and card sorting. We understand that such a process is subject to introducing mistakes. To reduce that threat, we establish a labeling team, and perform multiple rounds of labeling to make sure that all participants achieve conceptual coherence about noisy categories.

The third threat relates to the BLEU that is used to evaluate the performance of code summarization models. Recent researchers have raised concern over the use of BLEU \cite{DBLP:conf/sigsoft/RoyFA21}, warning the community that the way BLEU is used and interpreted can greatly affect its reliability. To mitigate that threat, we also adopt other metrics, \textit{i.e.}, ROUGE, METEOR, and CIDEr, when evaluating performance.

Another threat to validity is the replication of each model. To ensure that the experimental results are consistent with their papers, we retrain the models using the source code provided by the authors, and reuse the parameters provided by the authors.
Our experiments show that the performance of our retrained models is comparable to the performance of models reported in the papers.
For example, the METEOR score of Rencos is 21.1 on {\texttt{PCSD}} in their paper, and our retrained Rencos model is 20.2.

\section{Related Work}
\label{sec:relatedwork}
Recently, more and more researchers have realized that there are some underlying threats to the validity of existing code summarization research. These empirical studies mainly focused on data, evaluation metrics, and model effectiveness.

\textbf{Biases in data.}
{Existing research of data biases in the code summarization related tasks mainly focused on data quality, data representativeness, code preprocessing, and data selection.}
Sun \textit{et al.} \cite{DBLP:journals/corr/abs-2202-06649} applied syntactic and semantic query cleaning to improve the data quality for code search tasks. Their experiment results show that, training the popular code-search model with the filtered dataset improves its performance significantly.
Gro \textit{et al.} \cite{DBLP:conf/kbse/GrosSDY20} examined the underlying assumption {about data representativeness that:}  the task of generating comments sufficiently resembles the task of translating between natural languages, and so similar models and evaluation metrics could be used. By comparing four code-comment datasets, \textit{i.e.}, CodeNN, DeepCom, FunCom, and DocString, with a standard natural language translator dataset WMT19, they reported that comments are far more saturated with repeating trigrams than English translation datasets, and the repetitiveness has a very strong effect on measured performance.
Shi \textit{et al.} \cite{DBLP:journals/corr/abs-2107-07112} analyzed the influence of code preprocessing operations and dataset size on code summarization model performance. They found that different code preprocessing operations can affect the overall performance by a noticeable margin, and the code summarization approaches perform inconsistently on different datasets.
Huang \textit{et al.} \cite{DBLP:journals/spe/HuangJSHCZ20}
{reported the biases in data selection that, not all code is necessarily commented.}
They analyzed 136 well-known projects in GitHub, and reported that only a small part (4.4\%) of methods have header comments in real software projects. They proposed a machine learning technique to automatically identify commenting necessity, based on the structural features, syntactic features, and textual features of code.
There is a lack of in-depth analysis of the benchmark datasets. Our study bridges that gap with a large-scale analysis of {data preprocessing errors} and {low-quality comments} in the benchmark datasets, and investigates performance variation of existing models on the distilled dataset.

\textbf{Biases in evaluation metrics. }
Roy \textit{et al.} \cite{DBLP:conf/sigsoft/RoyFA21} conducted an empirical study with 226 human annotators to assess the degree to which automatic metrics reflect human evaluation for code summarization tasks. Their results indicated that metric improvements of less than 2 points do not guarantee systematic improvements in summarization quality, and are unreliable as proxies of human evaluation.
Gro \textit{et al.} \cite{DBLP:conf/kbse/GrosSDY20} measured 5,000 code-comment pairs, and found that the variants of BLEU chosen can cause substantial variation in the measured performance.
Shi \textit{et al.} \cite{DBLP:journals/corr/abs-2107-07112} also examined the BLEU variants. They concluded that BLEU variants used in prior work on code summarization are different from each other and the differences can carry some risks such as the validity of their claimed results.
Mahmud \textit{et al.} \cite{DBLP:journals/corr/abs-2106-08415} observed that some auto-generated comments provide a semantic meaning similar to the ground truth, despite exhibiting fewer n-gram matches. Therefore, they argued the feasibility of n-gram metrics such as BLEU.
Most of these work focus on validating the evaluation procedure for code summarization, while our work targets to validate the benchmark datasets, which would be important and valuable for building sound code summarization models.

\textbf{Analysis on Model Effectiveness.}
Mahmud \textit{et al.} \cite{DBLP:journals/corr/abs-2106-08415} compared three recently proposed code summarization models, and performed a manual open-coding of the most common errors committed by the models. They reported that missing information and incorrect construction are the most prevalent error types.
Chen \textit{et al.} \cite{DBLP:journals/tosem/ChenXHLL21} classified code comments into six categories (``what'', ``why'', ``how-to-use'', ``how-it-is-done'', ``property'', and ``others'') according to the intention, and conducted an experiment to perform six code summarization approaches on them to explore the impact of comment categories on code summarization.  They reported that no models perform the best for ``why'' and ``property'' comments among the six categories.
Most of the previous work focused on assessing the model effectiveness in terms of error types, comment intentions, preprocessing operations, and dataset size, while our work aims to investigate the model effectiveness on difficult levels of the code summarization task, complementing the existing studies. In addition, we report {data preprocessing errors} and {low-quality comments} in the code-comment dataset, which could provide a sounder foundation for existing work.

\section{Conclusion}
\label{sec:conclusion}

We propose a taxonomy of data preprocessing noises in four popularly-used benchmark datasets for code summarization, which contains 12 different types
of noise. We further build a rule-based cleaning tool for detecting noisy data of each category.
Experiments show that, the tool can accurately detect noises in our manually annotated data.
We then apply the cleaning tool to the four benchmark datasets, and assess their data quality. The results show that noisy data extensively exist in the four widely-used benchmark datasets (ranging from 31\% to 66\%).
Finally, we investigate the impacts of noisy data on three types of code summarization models (\textit{i.e.}, NNGen, NCS, and Rencos) by comparing their performance trained with datasets before and after the cleaning. The results show that the performance of three existing models trained with the filtered benchmark datasets improves BLEU-4 by 27\%, 21\%, and 24\%, ROUGE by 19\%, 11\%, and 16\%, METEOR by 19\%, 7\%, and 16\%, CIDEr by 46\%, 19\%, and 33\%, respectively.
We release our tool as a python library, named CAT, to facilitate relevant research in both academia and industry.

In our future work, we plan to extend our research methodology to other text generation tasks in software engineering such as commit message generation and code synthesis.
\section*{Acknowledgments}
We sincerely appreciate anonymous reviewers for their constructive and insightful suggestions for improving this manuscript.
This work is supported by the National Key Research and Development Program of China under Grant No. 2018YFB1403400, the National Science Foundation of China under Grant No. 61802374, 62002348, 62072442, 614220920020 and Youth Innovation Promotion Association Chinese Academy of Sciences.

\vspace{1cm}

\subsection{Methodology Overview}
\label{sec: methodology}

Figure \ref{fig:overview} shows an overview of the research methodology. In this work, we focus on evaluating the data quality of four widely used benchmark datasets, \textit{i.e.}, {\texttt{PCSD}} ~\cite{barone2017parallel}, {\texttt{TLC}}~\cite{hu2018summarizing}, {\texttt{Funcom}}~\cite{leclair2019neural}, and {\texttt{CSN}}~\cite{iyer2016summarizing}.

For RQ1, we evaluate the syntactic noises in the provided data from summary level and code level. After discarding the syntactically noisy data, we obtain the SYN-clean datasets.

For RQ2, we evaluate the semantic noises and the quality of the provided summaries in the SYN-clean datasets by three means:
(1) We conduct a model-based evaluation to analyze the concentration of actual summaries. The model can predict whether a given sentence is a code summary or not. It is trained on the external  dataset manually labeled by Chen et al. \cite{DBLP:journals/tosem/ChenXHLL21}, and tested on a randomly sampled summaries from the SYN-clean datasets.
(2) We conduct a human evaluation to analyze the accuracy, conciseness, and adequacy on the randomly sampled summaries.
(3) We conduct a metric-based evaluation to analyze the readability and coherence of the provided summaries.

For RQ3, we investigate the performance variation of the state-of-the-art code summarization approaches after removing both syntactically and semantically noisy summaries in the training sets.

\subsection{Analysis Design for RQ1}
According to common practice guidance, e.g., Javadoc guidance \cite{},
the first sentences of the code comments typically describe the code functionalities.
Therefore, nearly all the existing researches extract the first sentences of the code comments as the target summaries. However, we have a concern that developers may not fully follow the guidance when writing comments.
Meanwhile, since code comments contain rich information, e.g., URLs for external references and HTML tags for documentation auto-generation, existing researches use various preprocessing operations when constructing the code-summary pairs for the benchmark datasets. We have another concern about how well are these preprocessing operations performed.
In this work, we define the incorrectly extracted first sentences and inappropriate preprocessing consequences as \textbf{syntactic noises}.
To address the above concerns, we develop a taxonomy of syntactic noises with a set of heuristic rules that can precisely identify and correct them.

\textbf{Taxonomy of syntactic noises.} To develop a taxonomy of syntactic noises for both code data and summary data, we employ an \textit{open card sort} \cite{DBLP:journals/es/RuggM05} process by involving XXX participants.
The sorting process is conducted in multiple rounds. In the first round,
all participants label the same sample of benchmark data, with an intensive discussion session to achieve conceptual coherence about noisy categories. The average Cohen's Kappa is 0.86.
Then, a shared pool of categories is utilized and carefully maintained, and
each participant could select existing categories from and/or add new category names into the shared pool. The sorting process ends when there is no new category added for two consecutive rounds. In total, we conducted XX rounds and labeled XXX pairs of code summary (XXX pairs for each four benchmark datasets). We introduce the taxonomy of syntactic noises and their examples in Table XXX.

\textbf{Heuristic rules.} To automatically identify each category of syntactic noises, we build a set of heuristic rules. The heuristic rules comply with three criteria: (1) Any rule should define a unique and
specific construction category; (2) The rules should limit the preclusion of valid data within an acceptable range; and (3) Any rule is not a subrule of other rules in the set.
We build the heuristic rules based on observing the XX\% of our labeled data, and evaluate on the rest 20\%. The average accuracy is XX. Limited by the space, the details of the heuristic rules can be found on our website \cite{}.

Note that we focus on syntactic noises on summaries and code separately in RQ1, and leave the fine-grained semantic analysis, such as coherence between code and summary, to RQ2.

\subsection{Analysis Design for RQ2}
\mfw{Code summarization refers to the task of generating concise summaries that describe functionality of the code \cite{}. Directly taking the entire or first sentence of a code comment as the ground-truth summary would introduce noise, since developers write comments with a large variety of different meanings. For example, comments may also be used for other intentions, such as copyright and to-do, instead of describing the core functionalities, thus should not be regard as the code summary semantically. In this section, we define the non-summary comments that do not describe the functionalities of the corresponding code as semantic noises. We first introduce a model-based semantic filter to further remove the semantic noises from the SYN-clean datasets. Then, A human evaluation and metric-based evaluation are conducted to analyze the quality of the cleaned benchmark datasets.}

\subsubsection{Semantic noise analysis }XXX.

How do we define Semantic noise, summary and non-summary, refer to existing two papers.

Summary Classifier Construction.

Dataset.

Model.(Feature, GBDT)

Summary Classifier Evaluation.

Filtering non-summaries.

\textbf{Model-based Semantic Filter}
In this work, we employ the Gradient Boosting Decision Tree (GBDT) \cite{} as the semantic filter. GBDT is a widely-used machine learning model, and has been proven to be effective in comment classification in software engineering \cite{}. It is an ensemble model of decision trees, which are trained in sequence, and each base model is added to correct the error produced by its previous tree models. For each step, the samples misclassified by previous models are more likely to be selected as the train data, producing more accurate prediction performance.

To enhance features and improve model performance, we consider three categories of features: textual features, statistical features, and syntactic features. For textual features, after tokenization of the comments, we extract tf-idf \cite{} vector of each word, which could determine the mathematical significance of words in comments. Such a technique is a conventional feature extraction approach for the natural language text analysis in the software engineering literature \cite{}. For statistical features, we extract the number of words, unique words, punctuation and digits. For syntactic features, we determine whether a comment contains keywords such as ``todo", ``deprecate", ``copyright", ``lincense". These features are also used in prior software engineering literature \cite{}.

\textbf{Evaluation Metrics}
We use three commonly-used metrics to evaluate the performance of the semantic filter, \textit{i.e.}, Precision, Recall, and F1. (1) Precision refers to the ratio of the number of correct predictions to the total number of predictions; (2) Recall refers to the ratio of correct predictions to the total number of samples in the golden test set; and (3) F1 is the harmonic mean of precision and recall. When comparing the performances, we care more about F1 since it is balanced for evaluation.

\subsection{Analysis Design for RQ3}
\subsubsection{Human-Based Quality Assessment of Code Summary}
Most existing benchmark datasets for code summarization task are constructed based on real-world code repositories, such as GitHub.
Since developers have the freedom to write comments in various ways \cite{DBLP:conf/scam/RaniASBN21}, we have a concern about the quality of code summaries provided by the widely used benchmark datasets.
Therefore, we further assess the summary contents in the four benchmark datasets that are cleaned by our summary classifier.

Deriving from previous evaluations of code summarization \cite{DBLP:conf/iwpc/McBurneyM14,DBLP:conf/kbse/SridharaHMPV10,DBLP:conf/iwpc/SridharaPV11,DBLP:journals/access/SongSWY19}, the ``quality'' of the code summaries was assessed in terms of accuracy, conciseness, and adequacy by human evaluators.

\begin{itemize}
\item {\textbf{Accuracy}} measures to what extent the summaries reflect the semantics of the relevant code.
\item \textbf{Conciseness} refers to what extent the summaries contain the unnecessary information.
\item \textbf{Adequacy} evaluates how much information the summaries miss with regard to the information contained in code.
\end{itemize}

We use a five-level Likert scale \cite{chisnall1993questionnaire} to score the three aspects (Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree).
We randomly sample XXX code-summary pairs from the four benchmark data that are classified as `summary' (XXX for each) as our evaluation target.
{We recruit XX participants, including XX PhD students, XX master students, XX professional developers and XXX senior researchers, all familiar with the java, python and c\# programming languages. They all have at least three years of software development experience, and four of them have more than ten years of development experience.}\lin{@fangwen update}

\subsubsection{Metric-Based Quality Assessment of Code Summary}
Besides the three human-based quality assessment, we also evaluate the \textbf{Readability} \cite{DBLP:journals/information/EleyanOE20} and \textbf{Coherence} \cite{DBLP:conf/iwpc/SteidlHJ13}
by using predefined metrics following previous studies.

\textbf{Readability} measures the difficulty of understanding the contents of the summaries.
The Flesch Reading Ease has been used to measure the readability of code summaries \cite{DBLP:journals/information/EleyanOE20}.
The Flesch Reading Ease is calculated with the following formula:
\begin{equation}
    FRE=206.835-0.296S-84.6W
\end{equation}
where $S$ is the average number of words used per sentence, and $W$ is the average number of syllables per word. Higher FRE scores indicate summaries that are easier to read; lower scores indicate summaries that are more difficult to read.

\textbf{Coherence} reflects aspects of how the summary and code relate to each other.
The coherence coefficient has been used to measure coherence between the summary and code \cite{DBLP:conf/iwpc/SteidlHJ13}.
The coherence coefficient is calculated with the following formula:
\begin{equation}
    c\_coeff=
\end{equation}
Summaries with $c\_coeff = 0$ indicate that the coherence between method name and the summary is not sufficient. They should have additional information to emphasize the relation to the method name or indicate a poor method name.
Summaries with $c\_coeff > 0.5$ are trivial as
they do not contain additional information.

\subsection{Analysis Design for RQ3}

\section{Results}
\input{table/RQ1_v2}
\subsection{RQ1: Syntactical Noises Analysis }

\subsubsection{Taxonomy}
Table \ref{tab:taxonomy} exhibits the taxonomy of syntactical noises, and demonstrate some real-world examples.
For the summary data in the benchmark datasets, we identify the following six categories of syntactical noises:

\textbf{Incomplete first sentence.}
{Most researchers use the first sentences of the code comments as the target summaries. We observe that some inappropriate processing will lead to incomplete first sentences collected by researchers. As shown in Table \ref{tab:taxonomy}, the benchmark data only extracts the first line as the summary, and the next line that should be part of the first sentence is missing. This may be because the researchers directly use `new line' as the delimiter. Taking the incomplete first sentence as the target summary will lose some information, making the ground-truth unable to summarize the overall functionality of the source code. Besides, the incomplete first sentence will make the ground-truth summary shorter and thus easier to generate. }

\textbf{Excessive first sentence.}
{When collecting the first sentence as the target summary, some inappropriate processing used by researchers will lead to excessive first sentences collected. As the example shown in Table \ref{tab:taxonomy}, the {\texttt{PCSD}} dataset uses the entire code comments as the summary.
This is mainly because the usage of `period' as the unique delimiter. Excessive first sentences usually contain too much noise, \textit{e.g.}, argument definition or return type of the code, making the code summarization models hard to train.}

\textbf{Content tampering.} Developers may use HTML tags for documentation auto-generation or URLs for external references in comments. We observe that some inappropriate processing will keep the tags or URL contents together with the comments, thus contaminate the benchmark data with meaningless text, \textit{e.g.}, keeping the ``https://developers.google.com/protocol-buffers/docs/encoding'' as ``https developers google com protocol buffers doc encoding'' in the code summary. This kind of noises will lead to the summaries more longer and more difficult to generate, and thus typically will degrade the performance of automatic code summarization approaches.
Therefore, we remove the code-summary pairs with content tampering noises.

\textbf{Unnecessary Splitting.} Code summaries likely contain API terms when describing code functionality. Splitting code by camelCase or snake\_case is a common operation for code understanding \cite{leclair2019neural,DBLP:conf/iwpc/ShahbaziSF21,DBLP:conf/acl/HasanMIMHHAIS21}. However, we observe that some researchers perform this operation on summaries by splitting the API terms in the code summaries, \textit{e.g.}, split `This method initializes jTextField' into `This method initializes j text field'. We consider such operation will change the original meaning of code summaries. Therefore, we update the code-summary pair by recovering the splitted terms.

\textbf{Non-English.} Developers from different countries may write comments in their first languages, mixing with English language in the comments sometimes. We observe that existing benchmark dataset researches occasionally discard the Non-English text but remain the English text as code summaries, as shown in Table X. Since the remaining summary data are typically incomplete and meaningless, we remove such noises.

\textbf{Interrogation.}
Based on our observation, some of the summaries in the benchmark dataset are interrogations, \textit{e.g.}, 'Is it a good idea using USERID from ASPNET membership table as a foreign key?'. Such interrogations are mainly used for communication, rather than summarizing functionalities. Therefore, we remove such noises.

For the code data in the benchmark datasets, we identify the following five categories of syntactical noises:

\textbf{Empty Function.} Developers often take on technical debt to speed software development~\cite{wehaibi2016examining}. It has been widely observed that empty function is a common type of technical debt. However, the code-summary pairs extracted from these empty functions can introduce non-trivial noises, this is because an unimplemented empty function and its summary do not match either syntactically or semantically. We remove the code-summary pairs extracted from empty function for data purification.

\textbf{Commented-out \rev{Code}\add{Method}\song{is this right?}.} Developers often comment out a whole method for deprecating a specific functionality~\cite{falessi2015five}. We observer that the in the studied code summarization datasets, many commented-out methods are collected as the comments for the sequential methods. We consider these code-summary pairs extracted from commented-out methods as noises and remove them for cleaning data.

\textbf{Commented-out block code.} We have observed that commenting out code can also at block level except at method level introduced above. Thus, we also remove the commented-out block code for purifying code-summary pairs.

\textbf{Auto/Easy Code.} Developers often use modern IDEs like Eclipse or Visual Studio to generate auxiliary functions such as \textit{getter}, \textit{setter}, \textit{toString} for some predefined variables. The comments for these auto generated methods are often the same as the names of these methods, which makes the code-summary pairs less informative. We consider the code-summary pairs extracted from auto code as noise and remove them for data cleaning purpose.

\textbf{Duplicated code.}
{Developers often reuse code by copying, pasting and modifying to speed up software development \cite{DBLP:conf/oopsla/Allamanis19, DBLP:conf/icse/SajnaniSSRL16}. Identical code appear both in the training and test sets would make the model learn these cases easily and reduce performance on the ``real'' examples while falsely inflating performance metrics such as BLEU or Rouge. We remove exact duplicates from benchmark datasets and only keeping one copy of them.}

\subsubsection{Frequency and Distribution}

Figure \ref{fig:rq1-pie} shows the frequency of total syntactical noises that we removed or updated from the four benchmark datasets.
Note that, since one code-summary pair may involve multiple noises, we repeatedly count code-summary pairs that involve multiple noise categories when calculating frequency for each category , and count once for the total frequency. Thus, the sum of individual category percentages are slightly higher than the percentage of total noises.\lin{how to simplify this NOTE description.}
We can see that the frequency of total syntactic noises is range from 33.76\% to 60.53\% among the four benchmark datasets, and the average frequency is 43.77\%. Among them, Funcom is the most noisy dataset with 60\% syntactically noisy code-summary pairs, while PCSD is the least noisy dataset with 33\% syntactically noisy code-summary pairs. In the {\texttt{TLC}} and {\texttt{PCSD}} datasets, the updated noises are more frequent than removal noises. The other two datasets ({\texttt{Funcom}} and {\texttt{CSN}}) have a lower frequency of updated noises, especially the {\texttt{CSN}} dataset has no updated noise. The reason is that the {\texttt{CSN}} has only Interrogation and Duplicated Code noise, both of which are removal noise.

 \begin{figure}[h]
\centering
 \includegraphics[width=\columnwidth]{fig/rq1-pie.pdf}

 \caption{Percentage of syntactic noise in terms of removal and update.}
 \label{fig:rq1-pie}
 \end{figure}

Figure \ref{fig:rq1-heat} illustrates the distribution of each syntactical noise category on the four benchmark datasets.
The number on each cell presents the percentage of the syntactical noises in the corresponding benchmark dataset.
We can observe that, the benchmark datasets often contain multiple categories of {syntactic noise}s.
{\texttt{Funcom}} contains the most {syntactic noise} categories (10), followed by {\texttt{PCSD}} (7) and {\texttt{TLC}} (6). {\texttt{CSN}} contains the least {syntactic noise} categories (2).
Specifically, there are two primary syntactical issues with Funcom. First, 34.66\% of the code are auto-generated or easy functions such as \textit{setter}, \textit{getter}, \textit{tester}, and \textit{toString} methods\lin{can we format the code with sth like html tag?}.
Second, 20.01\% of the summaries are split unnecessarily by camelCase. Indeed, previous researches used to complain similar issues that {\texttt{Funcom}} XXX.\lin{@fangwen} \lin{what to say after this...}
The major syntactical issues for {\texttt{TLC}} is that,  22.85\% summaries are excessive first sentences, and 18.42\% code are duplicated.
The {syntactic noise}s in {\texttt{CSN}} are concentrated in Interrogation. 38.07\% summaries are questions that are used to communicate with collaborators, rather than summarizing code functionalities.
We consider that this phenomenon is related to the collection resources. The code-summary pairs in {\texttt{CSN}} are collected from Stack Overflow, which is a question-answer platform for developers, while other benchmark datasets are collected from real-world code repositories.
The primary syntactical issues with {\texttt{PCSD}} is that, 15.91\% summaries are the incomplete first sentences.

 \begin{figure}[h]
\centering
 \includegraphics[width=\columnwidth]{fig/rq1-heat.pdf}

 \caption{Heatmap for syntactical noises distribution}
 \label{fig:rq1-heat}
 \end{figure}

\textbf{Finding 1}: Syntactical noises are extensively existed in the four widely-used benchmark datasets, ranging from 33.76\% to 60.53\%. The primary syntactical issues with the benchmark datasets are: 34.66\% of the code in {\texttt{Funcom}} are auto-generated or easy functions; 22.85\% summaries in {\texttt{TLC}} are excessive first sentences; 38.07\% summaries in {\texttt{CSN}} are interrogative; and 15.91\% summaries in {\texttt{PCSD}} are the incomplete first sentences.

\subsection{RQ2: Semantic Noises Analysis}
\subsubsection{Summary V.S. Non-summary}

\subsubsection{Quality of Code Summary}

\subsection{RQ3: Impact on SOTA Performance}
\subsection{Comment-related Noisy Data}
\textbf{Partial Sentence.}
{Since it is a common practice to place a method's summary at the first sentence of its comment~\cite{technetwork_articles_java}, most researchers use the first sentences of the code comments as the target summaries. While, we have observed that some inappropriate processing can lead to partial first sentences collected. For example, {\texttt{Funcom}} only collects the first line from the following java doc as the comment, \textit{i.e.}, ``Returns the high value'', where the next line that should be part of the first sentence is missing. This is primarily due to automatic splitting using new line characters such as ``$\backslash$n''.
}

\begin{lstlisting}
/* Returns the high-value
 * for an item within a series. */
\end{lstlisting}
\vspace{-2mm}
\begin{small}
\texttt{Comment ({\texttt{Funcom}}): returns the high value}
\end{small}
\vspace{3pt}

\noindent \textbf{Verbose Sentence.}
When collecting the first sentence as the target comment, some inappropriate processing  will lead to verbose first sentences collected. For example, {\texttt{PCSD}} excessively includes the argument description ``arguments course data'' into the functionality summary.
\begin{lstlisting}
"""
Generate a CSV file containing a summary of the xBlock usage
Arguments:course_data
"""
\end{lstlisting}
\vspace{-2mm}
\begin{small}
\texttt{Comment ({\texttt{PCSD}}): generate a csv file containing a summary of the xblock usage \textcolor{red}{arguments course data}}
\end{small}

\vspace{3pt}
\noindent \textbf{Content Tampering.} Developers may use HTML tags for documentation auto-generation or URLs for external references in comments. We observe that some inappropriate processing will keep the tags or URL contents together with the comments, thus contaminating the benchmark data with meaningless text.
For example, {\texttt{CSN}} reserves the HTML tag ``p'' at the beginning and end of the comment.

\begin{lstlisting}
/* <p> Builds the JASPIC application context.</p> */
 \end{lstlisting}
 \vspace{-2mm}
 \begin{small}
\texttt{Comment ({\texttt{CSN}}): \textcolor{red}{p} builds the jaspic application context \textcolor{red}{p}}
 \end{small}
\vspace{3pt}

\noindent \textbf{Over-Splitting of Variable Identifiers.} Code comments are likely to contain variable identifiers or API terms when describing code functionalities. Splitting code by camelCase or snake\_case is a common operation for code understanding \cite{leclair2019neural,DBLP:conf/iwpc/ShahbaziSF21,DBLP:conf/acl/HasanMIMHHAIS21}. However, we observe that some studies perform this operation on {every matched token in the} comments including the predefined variable identifiers or API terms.
For example, {\texttt{Funcom}} splits {a variable named} ``jTextField'' into ``j text field'' when collecting comments.
We consider such an operation can change the original meaning of code comments.

\begin{lstlisting}
/* This method initializes jTextField. */
\end{lstlisting}
\vspace{-2mm}
\begin{small}
\texttt{Comment ({\texttt{Funcom}}): this method initializes \textcolor{red}{j text field}}
\end{small}
\vspace{3pt}

\noindent\textbf{Non-Literal.} Developers from different countries may write comments in their first languages, mixing with the English language in the comments sometimes. We observe that existing benchmark datasets occasionally discard the Non-English text but remain the English text as code comments.
For example, {\texttt{CSN}} only extracts the English words, \textit{i.e.,} ``jsonarray bean list arraylist'' {from the following mixed comment that contains both Chinese and English words} as the summarization for the corresponding source code.
Since the remaining comment data are typically incomplete and meaningless, we consider them as noises.

\vspace{3mm}

\begin{small}
\begin{CJK*}{UTF8}{gbsn}
\color[HTML]{009A17}{\makecell[l]{\  \colorbox{backcolour}{\texttt{/*\ 将JSONArray转换为Bean的List,默认为ArrayList\  */    }}\\}}
\end{CJK*}
\vspace{-2mm}
\texttt{Comment ({\texttt{CSN}}): jsonarray bean list arraylist}
\end{small}

\vspace{3mm}
\vspace{3pt}

\noindent \textbf{Interrogation.}
Based on our observation, some of the comments in the benchmark dataset are interrogations.
For example, in {\texttt{CSN}}, the comment for the \textit{isDue()} method is ``do we need to show the upgrade wizard prompt''.
Such interrogations are mainly used for communication, rather than summarizing functionalities.

\begin{lstlisting}
/* Do we need to show the upgrade wizard prompt? */
public boolean isDue() {
  if (isUpToDate)
     return false; ...
 \end{lstlisting}
    \vspace{-2mm}
\begin{small}
\texttt{Comment ({\texttt{CSN}}): do we need to show the upgrade wizard prompt ?}
\end{small}
\vspace{2pt}

\noindent \textbf{Under-Development Comments.} Based on our observation, some of the comments are related to ongoing and future development, including temporary tips, notes, \textit{etc}. For example, {\texttt{TLC}} has a comment ``description of the method'' for the \textit{openFile} method, which is of little worth for understanding code.
Since the under-development comments are typically inappropriate for the scenario of automated code summarization, we consider them as noises.

\begin{lstlisting}
/* Description of the Method */
protected void openFile(File f) {
  if (f == null) { ...
\end{lstlisting}
\vspace{-2mm}
\begin{small}
\texttt{Comment ({\texttt{TLC}}): description of the method}
\end{small}
\vspace{2.5mm}

\subsection{Code-related Noisy Data}

\textbf{Empty Function.} Developers often take on technical debt to speed up software development~\cite{wehaibi2016examining}. It has been widely observed that empty function is a common type of technical debt. However, the code-comment pairs extracted from these empty functions can introduce non-trivial noises, this is because an unimplemented empty function and its comment do not match either syntactically or semantically.
For example, {\texttt{Funcom}} includes an empty method \textit{end()} with a 10-word comment.

\begin{lstlisting}
/*Specifies the behaviour of the automaton in its end state*/
protected void end(){}
 \end{lstlisting}
    \vspace{-2mm}
\begin{small}
\texttt{Code ({\texttt{Funcom}}): protected void end}
\end{small}
\vspace{2.5mm}

\textbf{Commented-Out Method.} Developers often comment out a whole method for deprecating a specific functionality~\cite{falessi2015five}. We observe that, in the studied benchmark datasets, some commented-out methods are collected as the comments for the sequential methods.
For example, the commented-out method \textit{transformTypeID} and its comments are still included in {\texttt{Funcom}}.

\begin{lstlisting}
/* for now try mappig full type URI  */
// public String transformTypeID(URI typeuri){
// return typeuri.toString();}
 \end{lstlisting}
   \vspace{-2mm}
\begin{small}
\texttt{Code ({\texttt{Funcom}}): public string transform type id ...}
\end{small}
\vspace{2.5mm}

\textbf{Block-Comment Code.} We have observed that some code in the benchmark datasets contains block comments inside their bodies. The blocked comments could be natural-language comments or commented-out code.
For example, the block comment ``TODO: Why is he using Math.round'' is considered as a piece of code for the \textit{getFixQuality} method in {\texttt{Funcom}}.
If keeping these blocked comments in the source code, the original logics of the code are likely to be distorted when tokenizing it for code summarization models.

\begin{lstlisting}
/* Get GPS Quality Data  */
public int getFixQuality(){
  checkRefresh();
  // TODO: Why is he using Math.round?
  Return Math.round(quality);}
 \end{lstlisting}
  \vspace{-2mm}
\begin{small}
\texttt{Code ({\texttt{Funcom}}): public int get fix quality check refresh \textcolor{red}{todo why is he using math round} return math round quality}
\end{small}
\vspace{2.5mm}

\textbf{Auto Code.} Developers often use modern IDEs like Eclipse or Visual Studio to generate auxiliary functions such as \textit{getter}, \textit{setter}, \textit{toString}, or \textit{tester} for some predefined variables.
The comments for these {auto generated methods} are often  similar to or the same as the method names, which makes the code-comment pairs less informative.
{For example, in {\texttt{Funcom}}, the comment for the auto-generated test method (\textit{i.e.}, \textit{testConstructor}) is ``Test the constructor'', which is almost the same as the method name after splitting.}

\begin{lstlisting}
/* Test the constructor */
public void testConstructor() {
  System TestResult str;
  System TestID testID1; ...
\end{lstlisting}
\vspace{-2mm}
\begin{small}
\texttt{Comment ({\texttt{Funcom}}): test the constructor}\\
\texttt{Code ({\texttt{Funcom}}): public void test constructor ...}
\end{small}
\vspace{2.5mm}

\textbf{Duplicated Code.}
Developers often reuse code by copying, pasting and modifying to speed up software development~\cite{DBLP:conf/oopsla/Allamanis19, DBLP:conf/icse/SajnaniSSRL16}.
These code snippets often have similar or the same comments. Sharing identical code and summarization pairs in the training and test sets is inappropriate and would make the model learn these cases easily.

redpajama_set_name: RedPajamaArXiv