## Automatic Pull Request Description Generation Using LLMs: A T5 Model Approach

Md Nazmus Sakib Department of CSE Pabna University of Science and Technology Pabna, Bangladesh nazmus.200103@s.pust.ac.bd

Md Athikul Islam Department of Computer Science Boise State University Boise, ID, USA

Md Mashrur Arifin Department of Computer Science Boise State University Boise, ID, USA

mdathikulislam@u.boisestate.edu

Abstract -Developers create pull request (PR) descriptions to provide an overview of their changes and explain the motivations behind them. These descriptions help reviewers and fellow developers quickly understand the updates. Despite their importance, some developers omit these descriptions. To tackle this problem, we propose an automated method for generating PR descriptions based on commit messages and source code comments. This method frames the task as a text summarization problem, for which we utilized the T5 text-to-text transfer model. We fine-tuned a pre-trained T5 model using a dataset containing 33,466 PRs. The model's effectiveness was assessed using ROUGE metrics, which are recognized for their strong alignment with human evaluations. Our findings reveal that the T5 model significantly outperforms LexRank, which served as our baseline for comparison.

mdmashrurarifin@u.boisestate.edu

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00921v1-figure-6cae22ae-87a9-4d80-a873-041781ff9b14.jpg)

Index Terms -T5 Model, Text Summarization, NLP, LLMs, Large Language Models, Commit Messages, Pull Requests, Software Development, Git, GitHub

## I. INTRODUCTION

Developers annotate their code changes with commit messages, which serve as summaries of the changes and explain the reasons behind them [2], [20]. A well-maintained commit history provides a tree view of the software's evolution over time. In contrast, a pull request (PR) or merge request combines one or more commits. During development, a milestone typically involves several commits, and upon completion, developers create a PR to request the repository owner to merge their changes. The PR undergoes a review process and discussions, and once the reviewer is satisfied, the changes are merged into the main branch [16]. Figure 1 illustrates a typical GitHub workflow, including regular commits, and the process of creating and merging PRs. When creating a PR, developers have access to all new commits and their messages that differ from the main branch. The merge process of a PR takes place after discussions and potential modifications. Since PRs provide snapshots of developers' previous work (often called 'patches'), they have become an integral part of modern software development processes, such as continuous integration [21].

To create a PR, developers need to provide a title and description. These are essential for explaining the purpose and details of the PR [13]. Figure 2 illustrates the process of

Fig. 1: A typical GitHub workflow, showing the creation of feature branches from the master branch, the addition of new commits to the feature branches, and the final merge back into the master branch using PRs.

creating a PR for a GitHub repository [1]. One major benefit of a PR description is that it allows reviewers to gain an overall understanding of the PR without having to examine each commit message or the entire codebase, thereby saving a significant amount of time [7], [22]. However, developers often neglect to include PR descriptions, either intentionally or unintentionally, leading to a substantial number of PRs with empty descriptions, which is detrimental to the project [13]. To address this issue, it is necessary to automatically generate efficient PR descriptions.

## Open a pull request

Fig. 2: PR creation in GitHub [1].

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00921v1-figure-5f37b874-a13d-4ec5-b287-2971bb0bb038.jpg)

Some researchers have proposed various approaches to generate automatic commit messages from source code. Since a PR consists of one or multiple commits, generating descrip- tions for these PRs requires additional care. While practitioners have attempted to generate summaries using different methods, a state-of-the-art approach to generate PR descriptions based on the T5 transformer model is lacking. Therefore, we propose an automatic PR description generation method using a large language model (LLM) such as the unified text-to-text transfer transformer model T5. By leveraging this technique, teams can easily populate all empty PR descriptions using existing commit messages.

## II. BACKGROUND

Researchers have proposed numerous approaches to automate software changes, including the generation of commit messages. One notable example is CommitBERT, proposed by Tae-Hwan Jung, which summarizes code changes and addresses the challenge developers face in writing commit messages [10]. Another tool, ChangeScribe, automatically generates commit messages by summarizing source code changes [12]. Additionally, practitioners have utilized translation-based machine learning techniques to generate commit messages from git diffs [9]. These approaches typically employ natural language processing (NLP) techniques, using datasets that comprise code changes and commit messages to train their models and automate commit message generation.

In addition to commit messages, practitioners have also explored automating the process of generating descriptions for PRs. Fang et al. aimed to enhance the efficiency of PR description generation models [8]. They first identified the limitations of RNN-based models and proposed a novel approach called a hybrid attention network, which outperformed RNN-based models in terms of speed and effectiveness. Kuang et al. focused on addressing large-granularity PRs [11]. They introduced a model that captures extensive information about PR sentences to establish connections between them. Liu et al. treated the PR description generation problem as a sequence-to-sequence learning task and developed an innovative encoder-decoder model [13]. This model addresses outof-vocabulary words and enhances ROUGE optimization by incorporating a specialized loss function.

To tackle the challenge of description generation, researchers have explored various NLP and machine learning techniques. Among the most commonly utilized methods are the Seq2Seq model, attention network, and global attention network [8]. Transformer-based models, which utilize the selfattention mechanism, have emerged as preferred choices for many natural language generation tasks, including machine translation and text summarization [18], [19]. BERT (Bidirectional Encoder Representations from Transformers) is a prominent example of such models, capable of pretraining deep bidirectional representations [5]. These pre-trained representations can be fine-tuned by adding an extra output layer [5]. One significant advantage of fine-tuning these pretrained models is their versatility across various languagerelated tasks, such as question answering, summarization, and language interface, without substantial architectural changes [5].

Following the success of BERT, Google introduced the Text-To-Text Transfer Transformer, or T5, a unified model architecture capable of performing various text-to-text tasks [15]. While BERT excels at tasks like classification or span prediction, where it outputs labels or spans corresponding to input sentences, T5 transformers are particularly effective for generative tasks such as translation or abstractive summarization [15]. Unlike BERT, which is not well-suited for these types of tasks, T5 demonstrates superior performance in such scenarios.

Practitioners have already employed various strategies, including the use of a hybrid attention network and BERT, to automate the process of generating PR descriptions. However, the approach of fine-tuning a widely-used pre-trained unified text-to-text transfer transformer model, such as T5, for this purpose, along with its evaluation, is lacking in the current state of the art. Our plan is to fine-tune a pre-trained T5-small model and evaluate its performance using ROUGE metrics.

We pose the following research question to drive the research -

- · RQ: Is the T5 model effective for automatic PR description generation?

## III. DATASET

## A. Data Collection

We utilized a dataset sourced from GitHub, containing 33,466 filtered PRs selected from a total of 333,000 PRs [13]. Table II displays a sample PR from the training dataset, including its ID, description, and associated commit messages. The data was gathered from 95,804 Java repositories, of which 22,700 had at least one merged PR [13]. These repositories were sorted in descending order by the number of merged PRs, and the dataset was compiled using GitHub's APIs [13]. The input data consists of a combination of commit messages and source code comments, while the output is the PR description. Table I outlines the dataset split: 80% for training, 10% for validation, and 10% for testing.

TABLE I: Statistics of Dataset

| Split      | # of examples   | Language   |
|------------|-----------------|------------|
| Train      | 26,772          | JAVA       |
| Validation | 3,347           | JAVA       |
| Test       | 3,347           | JAVA       |

## B. Data Preprocessing

The first preprocessing step involved removing HTML or rich-text elements from both the commit messages and PR descriptions. Additionally, all URLs, references (e.g., '#456'), signatures (e.g., 'reviewed-by'), email addresses (e.g., '@placeholder'), and markdown headlines were eliminated using the NLTK library [13]. In the datasets, comments associated with a specific PR were separated by the ' &lt; cmsep &gt; ' separator. We replaced all instances of this separator to ensure that the model does not include them in the learning process. Subsequently, we created two bar charts to visualize the length distribution of abstracts (PR descriptions) and articles (commit messages). We also calculated the average length of articles and abstracts, which were found to be 76.02 and 36.41, respectively.

## IV. METHODS

To evaluate the effectiveness of the T5 transformer for PR description generation, we approach this text processing problem as a 'text-to-text' task [15]. This means that we take text as input and generate another text as output. Figure 3 illustrates how the text-to-text framework can be applied to tasks such as translation, sentiment analysis, and, most importantly, summarization in our case.

Fig. 3: Text-to-text framework [15]

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00921v1-figure-bb9fc01c-cf92-4656-98d3-1c7ae039fc50.jpg)

We leveraged transfer learning by utilizing the pre-trained Hugging Face T5 (t5-base) transformer, which incorporates a multi-task mixture of unsupervised and supervised tasks. The T5 model was pre-trained on the publicly available Colossal Clean Crawled Corpus (C4) dataset [15]. Subsequently, we fine-tuned the model using our smaller labeled datasets, aiming to achieve significantly improved performance compared to training the model solely on our data.

Our model construction and approach involve the following steps:

## A. Tokenization

Tokenization is the initial step in processing inputs, where text is split into tokens before being passed into the model. In the final stage, tokens are converted back into words. The T5 tokenizer uses the SentencePiece approach, which tokenizes and detokenizes text in an unsupervised manner. Figure 4 illustrates how we instantiated the tokenizer. Additionally, we utilized the batch encode plus method of the tokenizer to construct the source and target. These dictionaries contain encoded sequences or sequence pairs along with additional information. The max length parameter is employed to limit the total sequence length.

## # tokenzier for encoding the text

tokenizer TsTokenizer.from\_pretrained( model\_params [ "MODEL" ]

Fig. 4: T5 Tokenizer

## B. Constructing Target Sequence

The target sequence of the PRs comprises their descriptions. These target sequences are constrained to a maximum length of 50 tokens, which includes both words and punctuation. After calculating the average length of the abstracts as 36.41 tokens, we opted to set the maximum length to 50 tokens. Additionally, we enabled the truncation parameter to be true to ensure that the model can handle longer sequences.

## C. Constructing Source Sequence

The source sequence of the PRs consists of both the commit messages and source code comments. We set a maximum length of 512 tokens for the source sequences, which includes both words and punctuation. Similar to the target sequence, the source sequence underwent direct preprocessing using a standard text preprocessing procedure.

## D. Trainer model

We developed a T5-trainer to fine-tune the T5 model using our labeled source and target data. This T5-trainer first initializes the tokenizer instance from the tokenizer class and then adds a language model layer on top to generate the target sequence. It processes the source and target texts, constructing a data frame from them and generating the corresponding sequences using the tokenizer. Subsequently, it utilizes PyTorch DataLoader to create separate train and test data loaders. For optimization, the Adam optimizer from PyTorch is employed to adjust the network weights during training. The T5-trainer also specifies the number of epochs and other hyperparameters, training the model accordingly.

## V. EVALUATION

For the evaluation, we employed ROUGE metrics. In this section, we will describe the ROUGE metrics utilized in our evaluation and discuss how they compare to the baseline.

1) ROUGE metrics:

$$\text{its, where} \\ \text{model. In} \\ s. \text{ The T5} \\ \text{tokenizes}$$

$$R _ { L C S } = \frac { L C S ( X, Y ) } { | Y | }$$

$$P _ { L C S } = \frac { L C S ( X, Y ) } { | X | }$$

$$F _ { L C S } = \frac { ( 1 + \beta ^ { 2 } ) \cdot R _ { L C S } \cdot P _ { L C S } } { R _ { L C S } + \beta ^ { 2 } \cdot P _ { L C S } } \quad \quad ( 4 )$$

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) assesses the quality of summaries by comparing them to manually created references [22]. In our evaluation, we utilized ROUGE-N (ROUGE-1 and ROUGE-2) and ROUGEL metrics. Equation 1 demonstrates the calculation process

PR Description: a new plugin has been declared maven-java-formatterplugin . to use it , use ' mvn java-formatter : format ' . this plugin stands upon a new esigate module : esigate-tool-conf which should be used in the future to hold pmd ( already present in the new module ) and checkstyle ( present at project root level ) configuration . ( code formatting has not been made )

ID: esigate/esigate 11

Commit 1:

Commit Message: fix git ignore multiplicated settings.

Commit 2:

Commit Message: change path to formater config file.

Commit 3:

Commit Message:

format plugin attached to compile and defined in each module that needs format .

Commit 4:

Commit Message:

checkstyle configured the right way for multimodule project and complete makeover for site generation.

TABLE II: The left column contains the pull request descriptions and IDs. The right column lists the four commit messages from which the PR was created.

for ROUGE-N. Here, the subscript 'n' represents the length of the n-gram, and ROUGE-N is a recall-based measure as the ratio of the number of matched n-grams to the total number of n-grams in the reference summaries.

On the other hand, ROUGE-L is based on LCS (Longest Common Subsequence). Equations 2, 3, and 4 illustrate the calculations for recall, precision, and F1 score for ROUGE-L, respectively. The recall-based LCS ( R LCS ) is the ratio of the length of the LCS to the length of the reference summary. The precision-based LCS ( P LCS ) is the ratio of the length of the LCS to the length of the candidate summary. The F1 score based on LCS ( F LCS ) is the harmonic mean of R LCS and P LCS , weighted by β . They identify the longest subsequence between the output and target from the test dataset. We evaluated both ROUGE-N and ROUGE-L metrics using Python's ROUGE package.

tasks, which typically range from 0.20 to 0.40 [14], [17]. Furthermore, our T5 approach outperforms the baseline LexRank across all metrics, with average absolute gains of 4.35, 13.92, and 6.65 for recall, precision, and F1 score, respectively. These results indicate that our model can accurately capture the main points of PR descriptions compared to LexRank.

- 2) Baseline: To establish a baseline for PR description generation, we utilized LexRank, which is superior to centroidbased methods [6]. LexRank constructs clusters using sentences from documents and then selects the central sentences that reflect the summarized idea of each cluster. It employs eigenvector centrality within a graph representation of sentences to determine the importance of word tokens.
- 3) RQ: Is the T5 model effective for automatic PR description generation?: To evaluate our research question, we initially trained the model using the maximum batch size that the GPU memory supports (batch size=16) for 5 epochs. Subsequently, during evaluation, we used the fine-tuned T5trainer model to generate predictions for the test split. We saved these predictions, along with the actual PR descriptions, to a file named 'predictions.csv' within the 'model files' directory. Our model validation process begins with validating the 'predictions.csv' file. This involves calculating the ROUGE-1, ROUGE-2, and ROUGE-L metrics based on the predicted and original PR descriptions. We then compare these metric values to those obtained using the baseline LexRank.

We present our evaluation results in Table III, comparing our approach (T5) with the baseline approach (LexRank). The F1 scores produced by T5 are 32.16, 20.26, and 28.92 for ROUGE-1, ROUGE-2, and ROUGE-L, respectively. These scores align with the standard F1 scores for summarization

One reason T5 excels at producing high-quality PR descriptions is its training on the diverse 'Colossal Clean Crawled Corpus' (C4), a cleaned and structured version of the Common Crawl web dump. The C4 dataset, which encompasses 250 billion pages collected over the past 17 years enhances the model's ability to understand and generate natural language text [4]. In addition to this extensive pre-training, T5 leverages the inherent strengths of the Transformer architecture, particularly its self-attention mechanism. This self-attention mechanism allows T5 to excel in text-to-text tasks by effectively capturing long-range dependencies within the text [3]. This ability to maintain context and coherence across longer passages is crucial for producing accurate and relevant PR descriptions. Furthermore, the benefits of transfer learning are clearly demonstrated in our use case. While our specific dataset of PR descriptions was relatively small, fine-tuning the pre-trained T5 model on this dataset allowed us to harness the extensive knowledge encoded in the large C4 dataset. Overall, the combination of T5's training on a diverse and extensive corpus, its advanced Transformer-based architecture, and the effective application of transfer learning contributes to its superior performance in generating high-quality PR descriptions.

## VI. IMPLICATIONS

Our model is designed to assist developers who may forget to write PR descriptions or encounter difficulties in crafting meaningful ones. For instance, during the development of a PR that spans several days, developers may overlook key points from individual commits. In such cases, our PR summaries serve as valuable reminders. Additionally, developers can compare our generated summaries with their handwritten descriptions to identify any missed points.

Furthermore, our model aids reviewers in swiftly grasping the main ideas of PRs without the need to sift through each

TABLE III: Comparison of T5 and baseline model LexRank. The best performance is marked in bold .

| Approach      | ROUGE-1   | ROUGE-1   | ROUGE-1   | ROUGE-2   | ROUGE-2   | ROUGE-2   | ROUGE-L   | ROUGE-L   | ROUGE-L   |
|---------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| Approach      | Recall    | Precision | F1 score  | Recall    | Precision | F1 score  | Recall    | Precision | F1 score  |
| LexRank       | 25.19     | 30.11     | 25.30     | 12.69     | 13.81     | 12.26     | 25.28     | 28.33     | 23.82     |
| T5            | 29.68     | 46.89     | 32.16     | 19.56     | 26.08     | 20.26     | 26.96     | 41.03     | 28.92     |
| LexRank vs T5 | +17.82%   | +55.73%   | +27.11%   | +54.14%   | +88.85%   | +65.26%   | +6.65%    | +44.83%   | +21.41%   |

In summary, the T5 model outperforms the baseline LexRank in terms of ROUGE-1, ROUGE-2, and ROUGE-L, and produces more precise descriptions.

commit message and code change. This efficiency enables quicker decision-making, saving time in the review process. Moreover, automated PR descriptions can facilitate ongoing software maintenance efforts.

## VII. CONCLUSION AND FUTURE WORK

In this paper, our objective was to automate the generation of PR descriptions from commit messages and assess the effectiveness of our approach. We framed the PR description generation task as a sequence-to-sequence problem and employed a text-to-text T5 transformer. Leveraging transfer learning, we utilized a pre-trained T5 transformer and finetuned it on our smaller dataset comprising 10k PRs. Evaluation using ROUGE metrics demonstrated that our comprehensive model surpassed the baseline LexRank model in performance.

Moving forward, our future endeavors include training different LLMs, also the models on larger datasets and taking into account environmental constraints encountered during training on extensive datasets. Additionally, we aim to investigate whether our model maintains the same level of accuracy for other language generation tasks. Furthermore, we intend to extend the application of our model to other software engineering tasks, such as source code summarization and bug report summarization.

## REFERENCES

- [1] Creating a pull request -GitHub Docs - docs.github.com. https: //docs.github.com/en/pull-requests/collaborating-with-pull-requests/ proposing-changes-to-your-work-with-pull-requests/ creating-a-pull-request. [Accessed 21-05-2024].
- [2] R. P. Buse and W. R. Weimer. Automatically documenting program changes. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering , ASE '10, page 33-42, New York, NY, USA, 2010. Association for Computing Machinery.
- [3] J. Cheng, L. Dong, and M. Lapata. Long short-term memory-networks for machine reading. In J. Su, K. Duh, and X. Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 551-561, Austin, Texas, Nov. 2016. Association for Computational Linguistics.
- [4] Common Crawl. Common crawl. http://commoncrawl.org, 2024. http: //commoncrawl.org.
- [5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.
- [6] G. Erkan and D. R. Radev. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research , 22:457-479, 2004.
- [7] Y. Fan, X. Xia, D. Lo, and S. Li. Early prediction of merged code changes to prioritize reviewing tasks. Empirical Software Engineering , 23(6):3346-3393, 2018.
- [8] S. Fang, T. Zhang, Y.-S. Tan, Z. Xu, Z.-X. Yuan, and L.-Z. Meng. Prhan: Automated pull request description generation based on hybrid attention network. Journal of Systems and Software , 185:111160, 2022.
- [9] S. Jiang, A. Armaly, and C. McMillan. Automatically generating commit messages from diffs using neural machine translation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages 135-146, 2017.
- [10] T. Jung. Commitbert: Commit message generation using pre-trained programming language model. CoRR , abs/2105.14242, 2021.
- [11] K. Li, S. Ruyi, Z. Leihao, Z. Huan, and G. Honghao. Automatic generation of large-granularity pull request description. International Journal of Software &amp; Informatics , 11(4):383 - 403, 2021.
- [12] M. Linares-V´ asquez, L. F. Cort´ es-Coy, J. Aponte, and D. Poshyvanyk. Changescribe: A tool for automatically generating commit messages. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , volume 2, pages 709-712, 2015.
- [13] Z. Liu, X. Xia, C. Treude, D. Lo, and S. Li. Automatic generation of pull request descriptions. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages 176-188, 2019.
- [14] R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization, 2017.
- [15] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
- [16] M. M. Rahman and C. K. Roy. An insight into the pull requests of github. In Proceedings of the 11th Working Conference on Mining Software Repositories , MSR 2014, page 364-367, New York, NY, USA, 2014. Association for Computing Machinery.
- [17] A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointer-generator networks, 2017.
- [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.
- [19] Y. You, W. Jia, T. Liu, and W. Yang. Improving abstractive document summarization with salient information modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2132-2141, Florence, Italy, July 2019. Association for Computational Linguistics.
- [20] Y. Yu, H. Wang, V. Filkov, P. Devanbu, and B. Vasilescu. Wait for it: Determinants of pull request evaluation latency on github. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories , pages 367-371, 2015.
- [21] Y. Yu, H. Wang, V. Filkov, P. Devanbu, and B. Vasilescu. Wait for it: Determinants of pull request evaluation latency on github. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories , pages 367-371, 2015.
- [22] Y. Yu, H. Wang, G. Yin, and C. X. Ling. Reviewer recommender of pullrequests in github. In 2014 IEEE International Conference on Software Maintenance and Evolution , pages 609-612, 2014.
Md Nazmus Sakib Md Athikul Islam Md Mashrur Arifin
Automatic Pull Request Description Generation Using LLMs: A T5 Model Approach
Developers create pull request (PR) descriptions to provide an overview of
their changes and explain the motivations behind them. These descriptions help
reviewers and fellow developers quickly understand the updates. Despite their
importance, some developers omit these descriptions. To tackle this problem, we
propose an automated method for generating PR descriptions based on commit
messages and source code comments. This method frames the task as a text
summarization problem, for which we utilized the T5 text-to-text transfer
model. We fine-tuned a pre-trained T5 model using a dataset containing 33,466
PRs. The model's effectiveness was assessed using ROUGE metrics, which are
recognized for their strong alignment with human evaluations. Our findings
reveal that the T5 model significantly outperforms LexRank, which served as our
baseline for comparison.