## LLM as Runtime Error Handler: A Promising Pathway to Adaptive Self-Healing of Software Systems

Zhensu Sun Singapore Management University Singapore zssun@smu.edu.sg

## Xiaoning Du

Monash University

Australia xiaoning.du@monash.edu

## Haotian Zhu

Singapore Management University Singapore htzhu@smu.edu.sg

## Li Li

Bowen Xu North Carolina State University USA

bxu22@ncsu.edu

## David Lo

Beihang University China lilicoding@ieee.org

## ABSTRACT

Unanticipated runtime errors, lacking predefined handlers, can abruptly terminate execution and lead to severe consequences, such as data loss or system crashes. Despite extensive efforts to identify potential errors during the development phase, such unanticipated errors remain a challenge to to be entirely eliminated, making the runtime mitigation measurements still indispensable to minimize their impact. Automated self-healing techniques, such as reusing existing handlers, have been investigated to reduce the loss coming through with the execution termination. However, the usability of existing methods is retained by their predefined heuristic rules and they fail to handle diverse runtime errors adaptively. Recently, the advent of Large Language Models (LLMs) has opened new avenues for addressing this problem. Inspired by their remarkable capabilities in understanding and generating code, we propose to deal with the runtime errors in a real-time manner. The runtime context, such as error messages and program states, can be exploited to produce corresponding exception handling strategies with extended diversity and quality.

Singapore Management University Singapore davidlo@smu.edu.sg handlers and naturally uncaught, can lead to program crashes, exposure of sensitive information, or other severe consequences [1]. For example, the infamous Heartbleed bug in OpenSSL [14] was caused by an unanticipated out-of-bound error, which allows attackers to request a server to return a chunk of its own memory. To prevent such errors, substantial efforts have been dedicated to identifying possible errors in the source code through techniques such as software testing and verification [28, 61]. However, identifying all possible runtime errors and offering them corresponding exception handlers remains a challenging and often impractical goal.

Motivated by our idea, we propose Healer, the first LLM-assisted self-healing framework for handling runtime errors. When an unexpected, thus unhandled, runtime error occurs, Healer will be activated to generate a piece of error-handling code with the help of its internal LLM and the code will be executed inside the runtime environment owned by the framework to obtain a rectified program state from which the program should continue its execution. Our exploratory study evaluates the performance of Healer using four different code benchmarks and three state-of-the-art LLMs, GPT-3.5, GPT-4, and CodeQwen-7B. Results show that, without the need for any fine-tuning, GPT-4 can successfully help programs recover from 72.8% of runtime errors, highlighting the potential of LLMs in handling runtime errors. Moreover, in our experiments, Healer introduces negligible latency in normal code execution (less than 1 ms per program) and an acceptable overload for error handling (less than 4 seconds for the LLM to generate the handling code), making it a suitable real-time reactor.

## 1 INTRODUCTION

Runtime error handling [55] is an essential component of software reliability engineering, requiring developers to be able to anticipate possible errors and prepare corresponding error handlers beforehand. Unanticipated runtime errors, being lack of corresponding

Given the near inevitability of unanticipated runtime errors, dynamic countermeasures are demanded to minimize their impact when they really happen. This requirement has led to the development of self-healing systems [19, 41], where the one at the program level aims to automatically recover from abnormal program states and restore program functionality as much as possible. Notably, to heal a program execution is not to patch the code against an error but to revise the runtime state, such as variable values and environment configurations, to recover the program execution. A successful healing should not only proceed the execution without termination, but also correct the runtime state so that the rest of the execution can continue meaningfully. The core challenge for a self-healing system is that the specific errors and their locations are unknown in advance, requiring the system to react in real time. Consequently, it relies on predefined heuristic strategies, such as rolling back to a checkpoint [10] and matching existing error handlers [20]. Although self-healing systems have been widely studied for decades, limited by their rule-based nature, these strategies are not adaptive enough to deal with diverse errors that impede their adoption. A large-scale industry practitioner study highlights that one of the main challenges encountered with engineering such systems is the complexity of defining the adaptation rules [56].

Ideally, a self-healing system might employ human developers to be always on call and manually correct the state according to the error that occurred, which is impractical for real-world applications. Recently, the emergence of Large Language Models (LLMs) has introduced new opportunities to revolutionize this field. LLMs have shown remarkable performance in code-related tasks [21]. For instance, AlphaCode2 [3] has outperformed 85% of human participants in a programming competition, showcasing impressive capabilities in understanding both source code and natural language.

SMU Classification: Restricted

An execution successfully recovers from the error using Healer

Figure 1: A motivating example of how Healer handles the runtime errors to recover the execution of the program.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01055v1-figure-39a5cfa1-f1a2-4577-ba2c-141a03c0bbe1.jpg)

This skill set enables LLMs to understand the context of runtime errors, such as error messages and program states, and provide case-by-case solutions for each unanticipated runtime error in real time. Since human developers cannot monitor programs around the clock, we propose leveraging LLMs as virtual 'developers' who are always available to handle the error. To be specific, when an unanticipated runtime error occurs, an LLM can be invoked with the error context in the prompt and generate an appropriate code script that is not to be merged into the original program but is executed dynamically to recover the program from the faulty state. Moreover, LLMs can generate solutions within seconds, enabling real-time error resolution.

To explore the feasibility of our idea, we design an LLM-assisted runtime error handling framework, Healer. Healer acts as the default error handler for runtime exceptions that lack corresponding handlers and is also a last resort to prevent program crashes. Upon an error's occurrence, Healer gathers its context, including program state, error messages, and the erroneous source code. Using the collected information as a prompt, Healer instructs its LLM to generate a handling code snippet. The goal of the handling code is to survive the error and correct the program state, i.e., avoid the termination caused by the error and enable the program to continue its execution with the corrected state. The code is executed in an isolated environment to produce an error-free new program state. Based on this new state, the program can continue its execution from the point of error without termination.

run and produce the correct result for the input number. Handling in this way appears simple but is difficult to achieve with rulebased error-handling techniques. For example, Martin [42] proposes transforming the input value to fit the 'comfort zone,' an input space within which the program behaves acceptably. In this case, since the input num is 0, which the program cannot handle correctly, it would be transformed to a safe value (any value higher than 0), which avoids the error but leads to an incorrect final result. Similarly, Carzaniga et al. [11] proposed an automated method that replaces the failed operations with equivalent ones. This method generates equivalent alternatives by rewriting the code according to predefined human-written rules, which is labor-intensive and requires developers to anticipate potential errors.

To illustrate how Healer facilitates runtime error handling, we present a motivating example in Figure 1. The program aims to find the complement of a given number. However, the original code does not consider the case when the input number is less than 1. Thus, when the input number is 0, the code raises a ValueError because the variable com is an empty string, an invalid value for the int() function. Healer understands the root cause of the error and corrects the invalid value of com by assigning a '0' to it. With the corrected variable value, the program can continue to

To assess the feasibility of using LLMs for runtime error handling, we implement Healer as an executor for Python and conduct an exploratory study. We evaluate Healer on four different code benchmarks, containing a total of 1,185 (code, test case) pairs, where the code raises runtime errors with the test case, and test Healer empowered by three state-of-the-art LLMs, including two closed-source LLMs, GPT-3.5 and GPT-4, and one open-source LLM, CodeQwen. GPT-4 and CodeQwen are the best-performing closed-source and open-source LLMs, respectively, according to the EvalPlus leaderboard [31]. Initially, we investigate whether Healer, under a zero-shot setting, could effectively avoid the interruption and correct the faulty program state for the runtime errors during execution. The results show that without any fine-tuning, GPT-4 could continue execution in 72.8% of the instances and produce correct results in 39.6% of the instances, demonstrating the significant potential of LLMs for runtime error handling. We further examine the performance of Healer across various types of runtime errors. Among the 12 types of runtime errors in the benchmarks, LLMs display varying performance levels. For example, GPT-4 successfully handles 88.1% of AttributeErrors but only 50.0% of FileNotFoundErrors. Additionally, we create an instruction-tuning dataset using the (prompt, handling code) pairs generated in our experiments,

and fine-tune the GPT-3.5 and CodeQwen to assess its effectiveness. Wedo not fine-tune GPT-4 because we cannot access its fine-tuning API. After fine-tuning, the performance of GPT-3.5 and CodeQwen improves significantly. Specifically, there are 31.7% and 23.1% increases in instances that are free of interruption and 26.8% and 65.8% improvements in instances yielding correct results. Notably, the fine-tuned GPT-3.5 achieves performance even comparable to GPT4. Finally, we conduct a qualitative analysis to assess the overhead introduced by Healer. The results show that Healer introduces negligible latency in normal code execution (less than 1 ms per program) and an acceptable overload for error handling (less than 4 seconds for the LLM to generate the handling code), enabling real-time responses to runtime errors.

The main contribution of this paper is summarized as follows:

- · We, for the first time, propose to use LLMs to handle runtime errors, rejuvenating the research on self-healing software systems with a new direction.
- · We design an LLM-assisted runtime error handling framework, Healer, which mitigates runtime errors using the handling code generated by LLMs.
- · We conduct an exploratory study to assess the feasibility of LLMassisted runtime error handling , evaluating Healer on four different code benchmarks and three state-of-the-art LLMs.

All the data and source code used in this study are available at https://anonymous.4open.science/r/ErrorHandler-3861.

## 2 BACKGROUND AND RELATED WORK

In this section, we first recap the error-handling mechanisms in modern programming languages and then introduce the research on self-healing systems and the use of LLMs for code, respectively.

## 2.1 Error Handling in Modern Programming Languages

Modernprogramminglanguages typically incorporate built-in errorhandling mechanisms, with the try-catch block being a core component. This construct consists of a try block, which encloses code that might throw exceptions, and a catch block, which defines the error handler for exceptions thrown within the try block 1 . When developers anticipate an exception in a specific code segment, they can encapsulate that segment within a try block and implement an error handler within the catch block. If an exception occurs inside the try block, execution jumps to the catch block where the error handler is executed. Once the error handler completes without terminating the program, the program continues running the code following the try-catch block.

For errors occurring in code segments not enclosed by a try-catch block, the runtime environment halts execution and searches for an error handler by traversing up the call stack. This search continues until a suitable exception handler is found or the top of the call stack is reached. If no matching handler is found, the program typically terminates, often generating an error message and stack trace to assist with debugging, which can be seen as the default error handler. When this default error handler is triggered, the program's normal

1 Notably, we use the terms 'exceptions' and 'errors' interchangeably throughout the rest of this paper.

flow is disrupted, leading to potential unintended consequences or crashes. For example, in a financial application, the disruption might result in inconsistent transaction states, causing significant data integrity issues.

## 2.2 Self-healing Systems

Self-healing systems, as defined by Ghosh et al. [19], are designed to ' recover from the abnormal (or 'unhealthy') state and return to the normative ('healthy') state, and function as it was prior to disruption '. It is a long-standing research area aimed at enhancing system reliability and availability, often classified as a subclass of fault-tolerant [40] or self-adaptive [34] systems. The self-healing process involves three primary steps: (1) Maintenance of health: ensuring the normal functionality of the system, such as preparing redundant components [23]; (2) Detection of system failure: identifying abnormal states within the system, such as monitoring [17]; and (3) Recovery from failure: transforming a system state containing one or more errors (and possibly faults) into a state without detected errors [6].

In this paper, the proposed method is a recovery step at program level. For this step, various methods have been proposed, including restarting [9], rolling back to a checkpoint [10], trying alternative methods [11], modifying input [33], reusing existing error handlers [20], preparing default error handlers [43], or combining multiple strategies [12]. The core idea behind these studies is to heuristically construct a set of fixed recovery strategies and apply them when some specific errors occur, which is typically rule-based. For example, Failure-Oblivious Computing [43] ignores invalid memory writes (such as out-of-bounds writes and null dereferences) and generates default values for invalid memory reads. Gu et al. [20] handle errors by synthesizing handlers in two ways: (1) transforming the type of the error to fit existing error handlers and (2) returning early with a default value. Carzaniga et al. [10] propose restoring the application state to a previously set checkpoint and then selecting and executing an equivalent sequence of operations that might avoid the error. While effective in certain scenarios, their rule-based nature limits their generalization and adaptability. To the best of our knowledge, this is the first work to demonstrate that LLMs can effectively achieve the recovery, creating a new technical pathway for this long-standing research field (widely studied for decades). Compared with these rule-based methods, LLMs can dynamically adapt to a wide range of scenarios by understanding the source code context and natural language error messages, making them more flexible and generalizable.

## 2.3 Large Language Models for Code

Typically, LLMs are Transformer [51] models pre-trained on large corpora of textual data, including books, research papers, web contents, etc. These models have gained great popularity because of their versatility and effectiveness, and software engineering is no exception. Recent studies [21, 60] have highlighted the capability of LLMs in code recognition, understanding, and generation, etc. Numerous LLMs designed for code have been proposed, such as CodeGen [35], StarCoder [29], and CodeLlama [44]. These models are pre-trained on code corpora to predict the next token, making

them proficient at continuing a given source code but less effective at understanding human instructions.

More recently, the instruction tuning technique has been proposed and enables LLMs to understand and follow complex instructions [8, 38], such as ' You are working on a 3D block-based game, where each block can have a specific state that determines its behavior and appearance. Your task is to implement the following features: ... '. The instruction-tuned LLMs, such as GPT-4 [2], CodeQwen [7], and MagiCoder [54], apply to more complex code-related tasks and better aligns with human developers' intention. In this paper, we focus on such instruction-tuned LLMs, as handling runtime errors requires understanding multiple aspects of the error context, such as error messages and program state, which involves multiple modalities.

Instruction-tuned LLMs can be applied to specific tasks through either prompting or further fine-tuning. Fine-tuning involves updating the model's weights through additional training on a dataset tailored to a particular task. In contrast, prompting does not require additional training. It involves providing the LLM with a task description and, optionally, a few examples that break down how the task can be accomplished. Studies have shown that even without fine-tuning, LLMs can still perform impressively in many coding tasks, including bug fixing [49], code execution [48], and fault localization [27, 57]. In this paper, we apply LLMs to recover the program from a faulty state, a unique and unexplored task that is extremely rare in the practice of LLMs but important for selfhealing systems. We address several technical challenges, including how to construct prompts that help the LLM understand a specific task, how to interact with the runtime environment for the LLM to handle errors, and how to improve the LLM's performance in the task.

## 2.4 Why not LLM-based APR?

Recently, LLM has been widely applied in Automate Program Repair (APR) and shows promising performance. For example, Xia and Zhang [58] reveal that the LLM-based APR method is able to achieve the new state-of-the-art in program repair performance. This naturally raises the question: if we can statically fix the root causes of runtime errors using LLMs, why do we still need LLMs to recover from failures?

Firstly, it's important to clarify that LLM-assisted runtime error handling is not a replacement for LLM-based APR but rather a supplement. APR requires locating the bug first, which has long been a challenge in both the research community and industrial practices. Undetected bugs, even those that are easy to fix if identified, can escape APR tools and lead to dangerous errors during runtime. In contrast, when a runtime error occurs, its location is immediately available, allowing the handler to create a specific context-aware solution. We thus argue that LLM-assisted runtime error handling creates an additional layer of defense in software robustness, serving as a last-resort mechanism to prevent the consequences of unhandled errors.

To support this argument, we experimentally investigate this supplementary relationship between LLM-based APR and LLM-based runtime error handling on an APR benchmark, DebugBench [49],

Figure 2: The workflow of Healer. When a runtime error occurs, Healer collects the error context, prompts the LLM with the context, and generates the handling code. The handling code is then executed in an isolated environment to recover the program from the faulty state.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01055v1-figure-ef9c6f98-5da2-4f19-acca-747785c4d886.jpg)

containing over 1,000 buggy Python code. The authors of DebugBench employed LLMs to statically fix these buggy code snippets and released the revised versions. We execute these fixed snippets, finding that those fixed by GPT-3.5 and GPT-4 raised 251 and 109 runtime errors, respectively. These runtime errors are subsequently handled using our proposed method, leveraging the same LLM that performed the initial fixing. After the handling, 153 (60.0%) sessions for GPT-3.5 and 93 (85.33%) sessions for GPT-4 no longer raised errors, with 51 (20.3%) and 43 (39.4%) of these sessions ultimately producing the correct results as expected by the test cases. Therefore, our findings highlight the effectiveness of integrating LLM-assisted runtime error handling with LLM-based APR to further enhance software reliability and robustness.

## 3 PROPOSED FRAMEWORK

In this section, we introduce Healer, the first error-handling framework that leverages LLMs to handle unanticipated runtime errors. We begin with an overview of the framework, followed by detailed descriptions of its main components, prompt strategy, and a proofof-conception implementation for Python.

## 3.1 Overview

Healer is an error-handling framework designed to handle unanticipated runtime errors during program execution. We first define the problem and then introduce the workflow of Healer.

Requirements : As described in Section 1, unanticipated runtime errors are almost inevitable in practice, necessitating a mechanism to recover the program from such errors when they really occur. A key assumption according to the nature of unanticipated runtime errors is the lack of prior knowledge about their occurrence, such as their location, preventing the pre-implementation of specific error handlers. Under this assumption, an effective error-handling mechanism must recover the program from the faulty state in real time, allowing it to continue executing its intended functionality. Specifically, a successful handling should satisfy the following requirements:

- · Proceed the execution : The execution session should not terminate due to the error and should continue executing its remaining code. This requirement alone is easy to achieve by simply ignoring the error. However, it leaves the faulty state unchanged, potentially causing unexpected behaviors in the remaining code.
- · Correct the program state : The handling should correct the faulty state that caused the error, enabling the remaining code to execute with a correct initial state. For example, if the type of a variable is invalid and causes a type error, the handling should correct the type of the variable to the expected type. Notably, an incorrect program state could still proceed the execution, but it may lead to unexpected behaviors, such as wrong computing results, or even more severe errors.
- · Real-time response : The handling should be completed within a reasonable time frame to avoid excessively blocking the execution, thus minimizing side effects.

Workflow : Healer acts as a default error handler for the program, which can be seen as a last resort for handling unanticipated runtime errors. It will be activated when a runtime error occurs without an available error handler. The framework handles the errors relying on an internal LLM, which generates a piece of code, named handling code by understanding the error context. The workflow of Healer is illustrated in Figure 2. To ease the understanding, we assume the framework is applied for an interpreted programming language, such as Python and JavaScript, where the code is executed line by line, and the program state is inherently maintained. We discuss the application of Healer for compiled languages in Section 6. Formally, consider a program 𝑃 with 𝑛 lines as 𝑃 = { 𝐿 , 𝐿 1 2 , ..., 𝐿 𝑛 } . If an error 𝐸 occurs at line 𝐿 𝑖 without an available error handler, Healer is activated. It first collects the error context, including the error message, the program state 𝑆 , the entire code snippet, and the erroneous line, to construct a prompt. The internal LLM is then instructed with this prompt to generate a piece of handling code 𝐿 ′ 𝑖 . This code 𝐿 ′ 𝑖 is executed under the state 𝑆 of the original program using an executor isolated from the original program environment. If the execution of 𝐿 ′ 𝑖 is successful, the resulting program state 𝑆 ′ is integrated back to the original program state, allowing the program to continue from line 𝐿 𝑖 + 1. Notably, although the state is corrected for the current session, the original program is not fixed and thus still needs to be reproduced and fixed in future development, which is beyond the scope of this paper. If unsuccessful, the program is terminated, and the error is reported, following traditional error handling mechanisms.

## 3.2 Main Components

Healer consists of two main components: an LLM and an isolated executor. The former is used for generating the handling code given the error context, while the latter executes the handling code in an isolated environment and updates the program state of the original program.

Large Language Model : The LLM inside Healer needs to be capable of the error-handling task. As introduced in Section 2.3, LLMs trained with instruction tuning can understand and follow complex instructions. Using instruction-based LLMs, such as GPT-4 [2], we can prompt the LLM to generate handling code by providing proper prompts. We present our prompt strategy in Section 3.3. Moreover, the LLM can be further fine-tuned on a task-specific dataset to better align with the intention of the task. However, to the best of our knowledge, a decent dataset specifically for the error-handling task is currently unavailable. Although there are many human-crafted error handlers in open-source codebases, they do not consider the dynamic context of errors, especially the program state, and thus lack the knowledge of properly modifying the program state to recover from the error. To fill this gap, we create an instruction-tuning dataset to fine-tune the LLMs for the runtime error-handling task. Weintroduce and evaluate our dataset in Section 4.4 and Section 5.3, respectively.

Executor : The handling code generated by the LLM needs to be executed to produce a new program state. However, directly executing the code in the original session can be risky, as the code generated by LLMs may contain malicious code or vulnerabilities [39, 59]. To mitigate potential security risks, Healer adopts an executor that is isolated from the original program. It could be a new session, a sandbox, or a virtual machine, depending on the security level required by the system. The execution is not to re-run the original program from the beginning but to execute the handling code with the program state before the error occurs. Given an error at line 𝐿 𝑖 and handling code 𝐿 ′ 𝑖 , the executor initializes its state with 𝑠 𝑖 -1 (the program state before the error) and then executes 𝐿 ′ 𝑖 . If the handling code executes successfully, a new state 𝑠 ′ 𝑖 is produced. Healer then updates the original program state to 𝑠 ′ 𝑖 and continues the execution of the original program from line 𝐿 𝑖 + 1.

## 3.3 Prompt Construction

Theperformance of LLMs heavily relies on the provided prompt [32]. In Healer, the interaction between the LLM and Healer is conversational, where Healer first prompts its LLM with a system prompt and then a user prompt. The system prompt is a predefined instruction guiding the LLM's behavior, tone, or response style, while the user prompt is the input provided by the user during the conversation. The LLM does not respond to the system prompt but follows its instructions when it responds to the user prompt. In Healer, the system prompt introduces the requirements and constraints of the error handling task, and the user prompt provides the context of the occurred error.

- 3.3.1 System Prompt. The system prompt is structured into four parts: task description, goals, constraints, and examples. These four parts are concatenated to form the system prompt, which will be sent to the LLM as the first prompt in the conversation. In the

following, we introduce each part and present the prompt of that part. Notably, the presented prompt is a template, and the actual prompt will fill in its placeholders, i.e., the terms surrounded with curly braces, with specific information. There are two placeholders, {Language} and {Keywords}, which will be filled with the specific programming language name and unsupported keywords, respectively.

At the beginning, we describe the error-handling task as a code completion task, where the LLM is asked to replace a specified code block (the block that raises the error) with the correct code based on the provided information. Compared with directly instructing the LLM to recover runtime errors, this proxy task achieves the same effects but is more unambiguous and understandable for the LLM.

## Task Descriptions

You are an intelligent {Language} expert. The user will provide you with a piece of buggy {Language} code, along with an error message and program state. Your task is to generate correct lines to replace the lines where the errors occur. However, you can NOT change the rest of the code except the erroneous lines. The erroneous lines will be wrapped with &lt;error&gt; and &lt;/error&gt;. You can use any built-in functions or libraries in the generated lines. You can also use the variables in the program state.

In addition to the task description, we specify the goals of the generated code. The goals are designed according to the requirements defined in Section 3.1.

## Goals

Your code should achieve two goals: 1) Survive the error to ensure no more execptions will be raised, such as correcting wrong values in the program state, initializing undefined variables, or importing missing libraries; 2) Maintain the same functionality as the original code.

We also establish constraints to ensure that the generated code adheres to a specific format suitable for execution.

## Constraints

The code MUST be written in {Language} and should not omit any details. The code MUST be complete and correct in syntax. The code will be executed so its indentation should start from the first column. You are NOT allowed to use these keywords: {Keywords}. You MUST put &lt;code&gt;&lt;/code&gt; on the boundary of the generated code. Do not write anything else in your response.

At the end of the system prompt, we include a user prompt and its corresponding handling code, as the example to help the LLM understand the task better.

## Example

Here is one example: ... (omitted due to space limitation)

3.3.2 User Prompt. In addition to the source code, many runtime details can be collected, such as the error message, error location, and program state. As exemplified in Figure 3, Healer constructs prompt by combining this information from the runtime environment and then feeding it to the LLM.

- · Source Code : The source code of the program provides the basic context for the LLM to understand and survive the error.
- · Error Message : The error message is a string describing the error, either the default exception message or a customized message. It provides hints about the error type and potential cause.
- · Error Location : The error location is the position in the source code where the exception is raised. It saves the LLM from locating the error by itself and thus avoids potential misunderstanding. In the prompt, the position is not directly indicated using line numbers. Instead, we enclose the code block where the error occurs with the tags &lt;error&gt; and &lt;/error&gt;, as a more explicit representation.
- · Program State : The program state is a collection of variables and their values at the moment when the error occurs. By checking the program state, the LLM can understand the context of the error and generate the handling code accordingly.

## 3.4 Implementation

We implement Healer for Python as a proof of concept, which is available in our artifacts. The implementation is to demonstrate the feasibility and effectiveness of Healer, and it is not optimized for performance, security, or scalability, which would require further enhancement for practical use.

Healer is implemented as a code executor, accepting source code strings as input and executing them using the Healer process. Upon receiving the source code, the implementation first instruments the code using Python's ast module, wrapping each statement in a tryexcept block. Notably, the statements inside loops are not wrapped to avoid the overhead of exception handling in each iteration. In the except block, the implementation uses a customized exception handler to handle the runtime errors. The customized handler follows the steps illustrated in Figure 2. Given a caught exception, the handler collects the error type and message from the exception object and the program state using Python's built-in functions, locals() and globals() , which respectively return a dictionary of all the local and global variables. The collected information is formatted into the user and system prompts based on the templates introduced in Section 3.3. It then interacts with the LLM through RESTful APIs, sending the prompts to the LLM server as HTTP requests and receiving the handling code as the response. Upon receiving the handling code, the handler starts a new Python session and executes the handling code using the exec() function, a built-in Python function for dynamically executing source code. The program state for exec() is initialized with the collected program state. If the handling code executes successfully, the implementation compares the new program state with the original to identify differences and integrates these differences into the original program's state. To prevent infinite loops or other malicious behaviors, the implementation enforces a 60-second timeout limit for the execution of the handling code.

SMU Classification: Restricted

## Error Message

## System Prompt

Figure 3: An example of the prompt construction workflow in Healer. Both the system prompt and user prompt are constructed based on pre-defined templates.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01055v1-figure-7d617657-c832-4bd1-8e95-aa259a887c2c.jpg)

## 4 EXPERIMENTAL SETUP

This section details the setup of our experiments, including benchmarks, large language models, the instruction tuning dataset, evaluation metrics, and implementation details. The rationale behind our set up is driven by answering the following four research questions:

- · RQ1: How effective is Healer in handling runtime errors?
- · RQ2: How does the performance of Healer vary in handling different types of runtime errors?

six different LLMs, including GPT-3.5 (ChatGPT) [37], GPT-4 (gpt4-1106-preview) [2], CodeLlama (34B) [44], CodeT5+ (16B) [52], Mistral (7B) [25], and StarCoder (15B) [29]. These snippets are executed with the provided test cases in our local environment, and those (code, test case) pairs that raise runtime errors are kept. Totally, we obtain 295 instances from HumanEval+ and 226 instances from MBPP+. The detailed statistics of the runtime errors from each LLM are shown on the left of Table 1.

- · RQ3: How much performance of Healer gained by instruction tuning?
- · RQ4: What is the overhead introduced by Healer?

## 4.1 Benchmarks

In these experiments, we focus on Python due to its popularity and robust support for dynamic code execution. We evaluate the performance of Healer against runtime errors in three types of programs: (1) human-written code, (2) LLM-generated code, and (3) code with implanted bugs. To this end, we use four popular benchmarks: CodeNet for human-written code, HumanEval+ and MBPP+ for LLM-generated code, and DebugBench for code with implanted bugs.

CodeNet [24] includes 4,053 programming challenges for various programming languages from the AIZU Online Judge [26] and AtCoder [4] platforms, each with multiple implementations submitted by distinct programmers. For each challenge, at least one test case is provided. We run the code snippets on our local machine with the provided test cases and keep the (code, test case) pairs that raise runtime errors. This process yields 228 instances.

HumanEval+ and MBPP+ [31] are enhanced versions of HumanEval [13] and MBPP [5], benchmarks used to evaluate the performance of LLMs in code generation. They contain 164 and 400 Python coding tasks, respectively, each associated with several test cases In the paper of these two benchmarks, multiple LLMs are evaluated, and the code snippets generated during the evaluation are publicly available [15]. We reuse the code snippets generated by

DebugBench [49] is a benchmark designed to evaluate the debugging ability of LLMs, which is also applicable for runtime error handling. We use its Python subset, containing 1,414 code snippets from the LeetCode platform, each implanted with one or more bugs. This benchmark does not provide test cases, as they test the correctness of the code snippets by submitting them to the LeetCode platform. We thus extract example test cases from the problem descriptions, associating each code snippet with 2 to 4 test cases. Similarly, we run all implementations with the extracted test cases and keep the (code, test case) pairs that raise runtime errors, resulting in 436 instances.

## 4.2 Large Language Models

In our experiments, we respectively evaluate the best-performing closed-source and open-source LLMs, GPT-4 [2] and CodeQwen [7], in the code generation task according to the EvalPlus leaderboard [31] (accessed on May 2024). Additionally, we include GPT-3.5 [37], which is the previous version of GPT-4, to compare the performance of the two versions. All models are pre-trained by their creators, and by default, we do not perform further training unless otherwise stated. Notably, as the ground truth, i.e., the handling code for each sample, does not exist in the benchmarks, potential data leakage issues are avoided.

## 4.3 Evaluation Metrics

In each benchmark, all code snippets raise runtime errors when executed with their associated test cases. The effectiveness of Healer

is measured by its ability to recover from these runtime errors. As defined in Section 3.1, except for the time constraint, two requirements are considered for a successful recovery: (1) survive the error and (2) heal the program state. Accordingly, we define two statuses for each instance: PROCEED and CORRECT .

PROCEED means the program, used to be faulty, now completes its execution without being interrupted by any runtime errors. CORRECT indicates that the program not only executes without interruption but also produces correct results, i.e., the output matches the ground truth of the test case. Achieving a correct result requires the remaining code after the error point to execute correctly, relying on Healer to heal the program state. However, the program in the benchmarks may not be correctly implemented, where a healed program state could still lead to an incorrect result. We count the number of CORRECT and PROCEED instances and calculate their percentages.

## 4.4 Instruction-tuning Dataset

We created an instruction-tuning dataset to fine-tune the LLMs for runtime error handling. It consists of 586 (prompt, handling code) pairs, where the prompt is constructed using the strategy described in Section 3.3 and the handling code leads to the CORRECT status. These instances are produced using samples from all benchmarks except the human-written benchmark, CodeNet, which is reserved as the test set for the fine-tuned LLMs. Specifically, we run GPT3.5 and GPT-4 to handle the runtime errors in each instance from the selected benchmarks and record the handling code. For each CORRECT instance, we pair the prompt and its corresponding handling code as a sample in the dataset. This dataset will be used to answer RQ3 (Section 5.3) and is available in our artifacts to facilitate future research or applications.

## 4.5 Implementation Details

In our experiments, we use the OpenAI APIs to invoke GPT-3.5 (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-turbo-2024-04-09) and the Huggingface Transformers library with PyTorch to implement CodeQwen (CodeQwen1.5-7B-Chat). The experiments for CodeQwen are conducted on a machine with 20 vCPU, 100GB RAM, and an NVIDIA GeForce RTX L20 GPU (48GB RAM). During inference, we use the default hyper-parameters for all models but set the maximum input length to 4,096 tokens for CodeQwen due to GPU memory limitations. This may result in input text truncation for CodeQwen, and we will discuss the potential impact of this truncation in Section 6.1.

## 5 RESULTS

In this section, we report our experimental results and answer the four research questions.

## 5.1 RQ1: Effectiveness of Healer

This experiment assesses the effectiveness of Healer by analyzing the execution outcomes of the instances in the benchmarks. For each benchmark, we execute each code snippet with its paired test case and respectively handle the runtime errors using the three LLMs, GPT-3.5, GPT-4, and CodeQwen, following the framework described in Section 3. We record the final execution status of each instance and compute the two metrics: CORRECT and PROCEED .

The results, as shown in Table 1, indicate that Healer can effectively heal the runtime errors, thereby improving program availability. On average of all benchmarks, GPT-4, GPT-3.5, and CodeQwen can respectively proceed 72.8%, 48.2%, and 35.4% of the execution, and help 39.6%, 24.0%, and 13.3% of the program produce correct results. It provides compelling evidence that handling runtime errors using LLMs is a feasible and promising direction. Among the LLMs, GPT-4 consistently achieves the best performance across all benchmarks, with the highest CORRECT and PROCEED metrics. For example, on the DebugBench benchmark, GPT-4 successfully completes the execution of 338 instances and corrects the faulty states for at least 177 instances, accounting for 77.5% and 40.6% of the total errors. Even for code snippets written by LLMs, such as those in HumanEval and MBPP, Healer facilitated more correct results during execution. For example, GPT-4 enhances the execution outcomes of the code it generates by successfully correcting 11 additional instances in HumanEval and 1 additional instance in MBPP. This illustrates that LLMs can be used to mitigate the runtime errors they cause during code generation.

Answer to RQ1: Healer can effectively handle runtime errors and improve the availability and reliability of programs, particularly GPT-4-based Healer is able to continue execution in 72.8% of the instances and produce correct results in 39.6% of the instances.

## 5.2 RQ2: Effects of Error Types

In RQ2, we investigate the performance of Healer on different types of runtime errors. To be specific, we merge the instances of all the benchmarks and categorize them based on their error types. There are 12 types of runtime errors in total, but we analyze only those with more than 10 occurrences, combining the rest into the 'Others' category. This resulted in 7 categories: TypeError, AttributeError, IndexError, ValueError, NameError, FileNotFoundError, and Others. Each category is accompanied by an example error message to facilitate understanding. All of them are common errors in Python programs and are frequently encountered in practice. We calculate the number and proportion of instances with CORRECT and PROCEED for each category across the three LLMs.

As revealed in Table 2, LLMs exhibit varying performance across different error types. For example, GPT-4 continues the execution for 88.1% of AttributeError instances and 80.9% of IndexError instances but only 50.0% of FileNotFoundError instances and 62.8% of ValueError instances. This suggests that the effectiveness of LLMs varies by error type, indicating that error type could be an important factor when applying Healer. This observation implies that the reliability of error handling could be enhanced by applying Healer selectively based on the error type.

Interestingly, we find that different LLMs excel at handling different error types. For GPT-3.5, IndexError instances are the easiest to handle, with 55.2% of them free of interruption, while CodeQwen handles AttributeError best, with a 45.2% PROCEED. It suggests that the LLMs may have different capabilities in handling different

Table 1: The performance of GPT-4, GPT-3.5, and CodeQwen on all the benchmarks in a zero-shot setting.

|            |            | #Errors   | GPT-4       | GPT-4       | GPT-3.5    | GPT-3.5     | CodeQwen   |
|------------|------------|-----------|-------------|-------------|------------|-------------|------------|
| Benchmark  | Benchmark  |           | CORRECT     | PROCEED     | CORRECT    | PROCEED     | CORRECT    |
| CodeNet    | CodeNet    | 228       | 96 (42.1%)  | 167 (73.2%) | 71 (31.1%) | 120 (52.6%) | 38 (16.7%) |
|            | CodeLlama  | 70        | 33 (47.1%)  | 44 (62.9%)  | 25 (35.7%) | 40 (57.1%)  | 9 (12.9%)  |
|            | CodeT5+    | 57        | 21 (36.8%)  | 37 (64.9%)  | 12 (21.1%) | 29 (50.9%)  | 8 (14.0%)  |
|            | GPT-3.5    | 43        | 10 (23.3%)  | 31 (72.1%)  | 3 (7.0%)   | 16 (37.2%)  | 6 (14.0%)  |
|            | GPT-4      | 19        | 11 (57.9%)  | 19 (100.0%) | 4 (21.1%)  | 12 (63.2%)  | 2 (10.5%)  |
|            | Mistral    | 37        | 13 (35.1%)  | 24 (64.9%)  | 7 (18.9%)  | 17 (45.9%)  | 4 (10.8%)  |
|            | StarCoder  | 69        | 26 (37.7%)  | 40 (58.0%)  | 21 (30.4%) | 27 (39.1%)  | 13 (18.8%) |
|            | CodeLlama  | 30        | 12 (40.0%)  | 24 (80.0%)  | 8 (26.7%)  | 15 (50.0%)  | 4 (13.3%)  |
|            | CodeT5+    | 49        | 22 (44.9%)  | 37 (75.5%)  | 8 (16.3%)  | 19 (38.8%)  | 5 (10.2%)  |
|            | GPT-3.5    | 14        | 1 (7.1%)    | 13 (92.9%)  | 0 (0.0%)   | 7 (50.0%)   | 0 (0.0%)   |
|            | GPT-4      | 3         | 1 (33.3%)   | 3 (100.0%)  | 1 (33.3%)  | 1 (33.3%)   | 0 (0.0%)   |
|            | Mistral    | 73        | 25 (34.2%)  | 50 (68.5%)  | 10 (13.7%) | 25 (34.2%)  | 8 (11.0%)  |
|            | StarCoder  | 57        | 21 (36.8%)  | 36 (63.2%)  | 17 (29.8%) | 28 (49.1%)  | 7 (12.3%)  |
| DebugBench | DebugBench | 436       | 177 (40.6%) | 338 (77.5%) | 97 (22.2%) | 215 (49.3%) | 53 (12.2%) |

Table 2: The performance of GPT-4, GPT-3.5, and CodeQwen in handling different types of runtime errors.

| Error Type        | Example Error Message                            | Total   | GPT-4       | GPT-4       | GPT-3.5     | GPT-3.5     | CodeQwen   | CodeQwen    |
|-------------------|--------------------------------------------------|---------|-------------|-------------|-------------|-------------|------------|-------------|
| Error Type        | Example Error Message                            | Total   | CORRECT     | PROCEED     | CORRECT     | PROCEED     | CORRECT    | PROCEED     |
| TypeError         | can't multiply sequence by non-int of type 'str' | 312     | 145 (46.5%) | 214 (68.6%) | 85 (27.2%)  | 152 (48.7%) | 56 (17.9%) | 114 (36.5%) |
| AttributeError    | 'map' object has no attribute 'sort'             | 42      | 14 (33.3%)  | 37 (88.1%)  | 9 (21.4%)   | 19 (45.2%)  | 6 (14.3%)  | 19 (45.2%)  |
| IndexError        | list index out of range                          | 183     | 57 (31.1%)  | 148 (80.9%) | 41 (22.4%)  | 101 (55.2%) | 21 (11.5%) | 61 (33.3%)  |
| ValueError        | not enough values to unpack (expected 2, got 0)  | 78      | 22 (28.2%)  | 49 (62.8%)  | 15 (19.2%)  | 38 (48.7%)  | 4 (5.1%)   | 28 (35.9%)  |
| NameError         | name 'lst' is not defined                        | 520     | 211 (40.6%) | 379 (72.9%) | 122 (23.5%) | 241 (46.3%) | 68 (13.1%) | 187 (36.0%) |
| FileNotFoundError | No such file or directory: 'pypy3'               | 16      | 5 (31.3%)   | 8 (50.0%)   | 4 (25.0%)   | 6 (37.5%)   | 0 (0.0%)   | 1 (6.3%)    |
| Others            | integer division or modulo by zero               | 34      | 15 (44.1%)  | 28 (82.4%)  | 8 (23.5%)   | 14 (41.2%)  | 2 (5.9%)   | 9 (26.5%)   |

types of errors, highlighting the importance of understanding the strengths and weaknesses of each LLM for effective error handling.

Table 3: The comparison between fine-tuned and original versions for GPT-3.5 and CodeQwen. GPT-4 is used as a reference as we don't have access to its fine-tuning API.

Answer to RQ2: LLMs exhibit different performance levels across various runtime error types. GPT-4 handles AttributeError and IndexError most effectively, with PROCEED of 88.1% and 80.9%, respectively.

## 5.3 RQ3: Impact of Fine-tuning

To answer RQ3, we fine-tune the LLMs to find out how much their performance could be improved. Two LLMs, GPT-3.5 and CodeQwen, are fine-tuned in this experiment, while GPT-4 is used as a reference as we do not have access to fine-tune it. The fine-tuning dataset is the instruction-tuning dataset introduced in Section 4.4, which contains 586 training samples. With this dataset, GPT-3.5 is fine-tuned via the commercial API provided by OpenAI, where the hyperparameters are automatically determined [36]. CodeQwen is fine-tuned on our machine using LoRA [22], a parameter-efficient

| LLM      | Version   | CodeNet (228)   | CodeNet (228)   | CodeNet (228)   | CodeNet (228)   |
|----------|-----------|-----------------|-----------------|-----------------|-----------------|
| LLM      | Version   | CORRECT         | CORRECT         | PROCEED         | PROCEED         |
| GPT-4    | Original  | 96              |                 | 167             |                 |
| GPT-3.5  | Original  | 71              |                 | 120             |                 |
| GPT-3.5  | Finetuned | 90              | 26.8% ↑         | 158             | 31.7% ↑         |
| CodeQwen | Original  | 38              |                 | 91              |                 |
| CodeQwen | Finetuned | 63              | 65.8% ↑         | 112             | 23.1% ↑         |

fine-tuning method. After the fine-tuning, we evaluate the performance of the fine-tuned LLMs on CodeNet.

The results, presented in Table 3, show that fine-tuning significantly enhances LLM's performance in handling runtime errors. Both GPT-3.5 and CodeQwen exhibit substantial improvements, with GPT-3.5 achieving 26.8% more CORRECT and 31.7% more PROCEED instances, and CodeQwen achieving 65.8% more CORRECT and 23.1% more PROCEED instances. Remarkably, the fine-tuned GPT-3.5 successfully produces 90 CORRECT instances, closely approaching GPT-4's performance, which yields 96 CORRECT instances. Similarly, the fine-tuned CodeQwen also achieved competitive results comparable to GPT-3.5. This indicates that finetuning can be a powerful approach to developing task-specific LLMs for runtime error handling, potentially surpassing general-purpose LLMs in effectiveness.

Answer to RQ3: Fine-tuning significantly improves LLM performance in handling runtime errors. GPT-3.5 demonstrates a 26.8% increase in CORRECT instances and a 31.7% increase in PROCEED instances on CodeNet, achieving a performance level comparable to GPT-4.

## 5.4 RQ4: Overhead of Healer

In RQ4, we evaluate the overhead introduced by Healer when handling runtime errors. The overhead is divided into two parts: the overhead to normal program execution and the overhead during the error-handling process.

To determine the impact of Healer on normal program execution, we test the overhead introduced by the try-catch blocks, the error-handling mechanism to trigger Healer in our implementation. Specifically, we prepare two pieces of Python code: one containing an assignment statement and the other containing the same statement but wrapped in a try-catch block. The assignment statement will not raise any exceptions, and thus the time difference between the two code snippets can be attributed to the cost for the try-catch to detect exceptions. We execute both code snippets one million times and compute the difference in the consumed time. The results show that the code with a try-catch block costs only 4.5 ms more for one million runs, indicating that the overhead to normal program execution is negligible. It aligns with the claim in the official Python document, i.e., 'A try/except block is extremely efficient if no exceptions are raised' [16].

The primary overhead during the error-handling process arises from the LLMs' inference times. We measured the average time required by CodeQwen (deployed on our local machine using Huggingface Transformers) and GPT-3.5 and GPT-4 (deployed on OpenAI's remote server) to generate the handling code for the benchmarks. On average, CodeQwen, GPT-3.5, and GPT-4 take 1.7 seconds, 1.6 seconds, and 3.1 seconds, respectively, to generate the handling code for the occurred error. These durations are deemed acceptable, compared with the consequences of the execution interruption. Given the rapid advancements in LLMs, we anticipate continued improvements in their efficiency.

Answer to RQ4: The try-catch mechanism adds only 4.5 ms latency for one million runs, and the LLMs' inference times (1.7s for CodeQwen, 1.6s for GPT-3.5, and 3.1s for GPT-4) allow for real-time error handling, demonstrating the feasibility of using Healer in practice.

## 6 DISCUSSION

In this section, we discuss the threats to validity and the issues with Healer, including its trustworthiness, operational cost, and extension to other programming languages.

## 6.1 Threats to Validity

Internal validity: The main threat to internal validity is the difference in hyper-parameters among the LLMs used in this study. We evaluate the performance of Healer using the commercial LLMs, GPT-3.5 and GPT-4, and the open-source LLM, CodeQwen. In our experiments, we use the default hyper-parameters for these models during inference, except for the CodeQwen's context window. The context window for CodeQwen is set to 4096 tokens, which is significantly smaller than the context windows of GPT-3.5 and GPT-4. This limitation is due to the deployment of CodeQwen on our local machine, where 4096 tokens is the maximum context window supported by the server. This constraint may affect CodeQwen's performance in handling runtime errors, as some prompts exceed this window size and are truncated. This could result in an unfair comparison between CodeQwen and GPT-4. However, we qualitatively analyzed the prompt length distribution, where less than 1% of the prompts are truncated. Therefore, we believe the impact of this truncation is trivial.

External validity: The threats to external validity mainly lie in the selection of benchmarks. The benchmarks used in this paper are limited to four Python benchmarks, which may not encompass all practical scenarios. Although these benchmarks include the most common error types in Python programs, certain error types may still be unrepresented, potentially limiting the generalizability of the conclusions to real-world scenarios.

Construct validity: Bias in performance evaluation is a common threat to construct validity. In our experiment, we introduce two metrics, CORRECT and SURVIVED, to measure the effectiveness of Healer in handling runtime errors. While these metrics are intuitive, they may not be optimal. For instance, even if an error is successfully handled, the resulting program output may still be incorrect due to flaws in the original source code.

## 6.2 Trustworthiness of the Handling Code

As revealed by various studies [39, 46, 60], the code generated by LLMs is not guaranteed to be secure. It may contain malicious code injected by attackers or vulnerable code, leading to security issues. Moreover, due to the randomness in the generation process, the generated code may exhibit unpredictable behaviors. Healer, operating without human intervention, directly executes the generated code to handle runtime errors, which could be risky. Currently, popular AI systems, such as ChatGPT, mitigate such security risks by running model-generated code in a sandboxed environment with limited permissions. Similarly, Healer adopts this strategy.

However, since Healer needs to update the program state of the target program rather than merely displaying execution results, more sophisticated security mechanisms are required to safeguard the target program. Continued research is essential to ensure the security and reliability of handling code in Healer.

## 6.3 Operational Cost for Using Healer

Healer involves an LLM for generating handling code, which introduces additional costs associated with the LLM inference. Typically, Healer can interact with LLMs either deployed on remote servers or on user devices. Remote servers may incur additional costs due to operational expenses or commercial fees. In our experiments, we used the commercial APIs of GPT-3.5 and GPT-4 to generate handling code, with costs around $0.002 and $0.02 per handling code, respectively. On-device inference can significantly reduce costs and enhance the reliability of Healer by eliminating the dependence on remote servers. Various techniques, such as model compression [47] and quantization [30], have been developed to enable model inference on user devices. For example, LLAMA.cpp [18] can run Llama2-13B [50] on a single M2 Ultra MacBook with fast inference speeds (61.44 ms per token). As the field of LLMs progresses, we anticipate further advancements in reducing the resource requirements for LLMs.

## 6.4 Extension to Other Programming Languages

As introduced in Section 3.4, we implement Healer in Python. However, as a general framework for handling runtime errors, Healer is not limited to Python and can be extended to other programming languages. The applicability of Healer to a programming language depends on two factors: (1) the availability of LLMs for the language, and (2) the feasibility of executing dynamically generated code during runtime. Currently, most of SOTA LLMs [2, 7, 29] are pre-trained on multiple programming languages. For example, CodeQwen [7] supports 92 programming languages, covering all the mainstream languages. Therefore, the first factor is a minor concern. The second factor depends on the programming language itself. Typically, interpreted languages, such as Python, JavaScript, and Ruby, are more suitable for Healer, as they natively support the execution of dynamically generated code. Compiled languages, such as Java, C++, and Rust, precompile the source code before execution, introducing extra complexity to dynamic code execution. However, there are usually some workarounds in such languages. For example, Java provides the Java Compiler API to compile the code at runtime. Therefore, technically, Healer can be applied to a wide range of programming languages.

## 7 CONCLUSION AND FUTURE WORK

In this paper, we introduced Healer, an innovative framework that uses Large Language Models (LLMs) to handle unanticipated runtime errors dynamically. Our study demonstrated that LLMs, particularly GPT-4, can effectively generate handling code snippets, successfully surviving a significant portion of runtime errors, 72.8% of errors in our experiments. Moreover, fine-tuning further enhanced performance, showing promise for LLM-assisted error recovery. Healer opens up a new technical pathway towards selfhealing systems, where LLMs are employed as error handlers to recover from runtime errors automatically.

Throughout the study, we identified several avenues for future research. Firstly, the performance of Healer can be further enhanced through methods like prompt engineering [53] and instruction tuning [45]. Secondly, addressing the trustworthiness of code generated by LLMs is crucial, necessitating mechanisms specifically designed for Healer. Lastly, integrating Healer into the runtime environment is an interesting topic, offering more accessible and efficient error-handling mechanisms that benefit the development of self-healing systems.

## REFERENCES

- [1] 2024. CWE - CWE-248: Uncaught Exception (4.14). https://cwe.mitre.org/data/ definitions/248.html Accessed: 2024-06-03.
- [2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
- [3] AlphaCode Team, Google DeepMind. 2023. AlphaCode 2 Technical Report. https://storage.googleapis.com/deepmind-media/AlphaCode2/ AlphaCode2\_Tech\_Report.pdf Accessed: 2024-05-23.
- [4] AtCoder. 2024. AtCoder. https://atcoder.jp/ Accessed: 2024-05-26.
- [5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021).
- [6] Algirdas Avizienis, J-C Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing 1, 1 (2004), 11-33.
- [7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen Technical Report. arXiv preprint arXiv:2309.16609 (2023).
- [8] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023).
- [9] George Candea, Emre Kiciman, Steve Zhang, Pedram Keyani, and Armando Fox. 2003. JAGR: An autonomous self-recovering application server. In 2003 Autonomic Computing Workshop . IEEE, 168-177.
- [10] Antonio Carzaniga, Alessandra Gorla, Andrea Mattavelli, Nicolo Perino, and Mauro Pezze. 2013. Automatic recovery from runtime failures. In 2013 35th International Conference on Software Engineering (ICSE) . IEEE, 782-791.
- [11] Antonio Carzaniga, Alessandra Gorla, Nicolò Perino, and Mauro Pezzè. 2010. Automatic workarounds for web applications. In Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering . 237-246.
- [12] Hervé Chang, Leonardo Mariani, and Mauro Pezze. 2013. Exception handlers for healing component-based systems. ACM Transactions on Software Engineering and Methodology (TOSEM) 22, 4 (2013), 1-40.
- [13] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
- [14] Zakir Durumeric, Frank Li, James Kasten, Johanna Amann, Jethro Beekman, Mathias Payer, Nicolas Weaver, David Adrian, Vern Paxson, Michael Bailey, et al. 2014. The matter of heartbleed. In Proceedings of the 2014 conference on internet measurement conference . 475-488.
- [15] EvalPlus. 2024. EvalPlus Releases. https://github.com/evalplus/evalplus/releases Accessed: 2024-05-26.
- [16] Python Software Foundation. 2024. Python FAQ: How fast are exceptions? https://docs.python.org/3/faq/design.html#how-fast-are-exceptions Accessed: 2024-08-01.
- [17] David Garlan and Bradley Schmerl. 2002. Model-based adaptation for self-healing systems. In Proceedings of the first workshop on Self-healing systems . 27-32.
- [18] Georgi Gerganov. 2023. llama.cpp: Port of Facebook's LLaMA model in C/C++. https://github.com/ggerganov/llama.cpp Accessed: 2024-05-30.

- [19] Debanjan Ghosh, Raj Sharman, H Raghav Rao, and Shambhu Upadhyaya. 2007. Self-healing systems-survey and synthesis. Decision support systems 42, 4 (2007), 2164-2185.
- [20] Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Jian Lü, and Zhendong Su. 2016. Automatic runtime recovery via error handler synthesis. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering . 684-695.
- [21] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620 [cs.SE]
- [22] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
- [23] Michael N Huhns, Vance T Holderfield, and Rosa Laura Zavala Gutierrez. 2003. Robust software via agent-based redundancy. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems . 1018-1019.
- [24] IBM. 2024. Project CodeNet. https://github.com/IBM/Project\_CodeNet Accessed: 2024-05-26.
- [25] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023).
- [26] Aizu Online Judge. 2024. Aizu Online Judge Home. https://onlinejudge.uaizu.ac.jp/home Accessed: 2024-05-26.
- [27] Sungmin Kang, Gabin An, and Shin Yoo. 2023. A preliminary evaluation of llm-based fault localization. arXiv preprint arXiv:2308.05487 (2023).
- [28] Pavneet Singh Kochhar, Ferdian Thung, Nachiappan Nagappan, Thomas Zimmermann, and David Lo. 2015. Understanding the test automation culture of app developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) . IEEE, 1-10.
- [29] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023).
- [30] Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing 461 (2021), 370-403.
- [31] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024).
- [32] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1-35.
- [33] Fan Long, Vijay Ganesh, Michael Carbin, Stelios Sidiroglou, and Martin Rinard. 2012. Automatic input rectification. In 2012 34th International Conference on Software Engineering (ICSE) . IEEE, 80-90.
- [34] Frank D Macías-Escrivá, Rodolfo Haber, Raul Del Toro, and Vicente Hernandez. 2013. Self-adaptive systems: A survey of current approaches, research challenges and applications. Expert Systems with Applications 40, 18 (2013), 7267-7279.
- [35] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022).
- [36] OpenAI. 2024. Fine-Tuning Integrations. https://platform.openai.com/docs/ guides/fine-tuning/fine-tuning-integrations Accessed: 2024-05-26.
- [37] OpenAI. 2024. GPT-3.5 Turbo Model Documentation. https://platform.openai. com/docs/models/gpt-3-5-turbo Accessed: 2024-05-21.
- [38] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730-27744.
- [39] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot's code contributions. In 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 754-768.
- [40] William H Pierce. 2014. Failure-tolerant computer design . Academic Press.
- [41] Harald Psaier and Schahram Dustdar. 2011. A survey on self-healing systems: approaches and systems. Computing 91 (2011), 43-73.
- [42] Martin C Rinard. 2007. Living in the comfort zone. ACM SIGPLAN Notices 42, 10 (2007), 611-622.
- [43] Martin C Rinard, Cristian Cadar, Daniel Dumitran, Daniel M Roy, Tudor Leu, and William S Beebee. 2004. Enhancing Server Availability and Security Through Failure-Oblivious Computing.. In Osdi , Vol. 4. 21-21.
- [44] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023).
- [45] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 (2021).
- [46] Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. In chatgpt we trust? measuring and characterizing the reliability of chatgpt. arXiv preprint arXiv:2304.08979 (2023).
- [47] Jieke Shi, Zhou Yang, Bowen Xu, Hong Jin Kang, and David Lo. 2022. Compressing pre-trained models of code into 3 mb. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering . 1-12.
- [48] Beatriz Souza and Michael Pradel. 2023. Lexecutor: Learning-guided execution. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1522-1534.
- [49] Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621 (2024).
- [50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
- [51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
- [52] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 (2023).
- [53] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824-24837.
- [54] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120 (2023).
- [55] Westley Weimer and George C Necula. 2004. Finding and preventing run-time error handling mistakes. In Proceedings of the 19th annual ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications . 419-431.
- [56] Danny Weyns, Ilias Gerostathopoulos, Nadeem Abbas, Jesper Andersson, Stefan Biffl, Premek Brada, Tomas Bures, Amleto Di Salle, Patricia Lago, Angelika Musil, et al. 2022. Preliminary results of a survey on the use of self-adaptation in industry. In Proceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems . 70-76.
- [57] Ratnadira Widyasari, Jia Wei Ang, Truong Giang Nguyen, Neil Sharma, and David Lo. 2024. Demystifying Faulty Code with LLM: Step-by-Step Reasoning for Explainable Fault Localization. arXiv preprint arXiv:2403.10507 (2024).
- [58] Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023).
- [59] Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022. Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering . 1482-1493.
- [60] Zhou Yang, Zhensu Sun, Terry Zhuo Yue, Premkumar Devanbu, and David Lo. 2024. Robustness, security, privacy, explainability, efficiency, and usability of large language models for code. arXiv preprint arXiv:2403.07506 (2024).
- [61] Jian Zhou, Hongyu Zhang, and David Lo. 2012. Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports. In 2012 34th International conference on software engineering (ICSE) . IEEE, 14-24.
Zhensu Sun Haotian Zhu Bowen Xu Xiaoning Du Li Li David Lo
LLM as Runtime Error Handler: A Promising Pathway to Adaptive Self-Healing of Software Systems
Unanticipated runtime errors, lacking predefined handlers, can abruptly
terminate execution and lead to severe consequences, such as data loss or
system crashes. Despite extensive efforts to identify potential errors during
the development phase, such unanticipated errors remain a challenge to to be
entirely eliminated, making the runtime mitigation measurements still
indispensable to minimize their impact. Automated self-healing techniques, such
as reusing existing handlers, have been investigated to reduce the loss coming
through with the execution termination. However, the usability of existing
methods is retained by their predefined heuristic rules and they fail to handle
diverse runtime errors adaptively. Recently, the advent of Large Language
Models (LLMs) has opened new avenues for addressing this problem. Inspired by
their remarkable capabilities in understanding and generating code, we propose
to deal with the runtime errors in a real-time manner using LLMs.
  Specifically, we propose Healer, the first LLM-assisted self-healing
framework for handling runtime errors. When an unhandled runtime error occurs,
Healer will be activated to generate a piece of error-handling code with the
help of its internal LLM and the code will be executed inside the runtime
environment owned by the framework to obtain a rectified program state from
which the program should continue its execution. Our exploratory study
evaluates the performance of Healer using four different code benchmarks and
three state-of-the-art LLMs, GPT-3.5, GPT-4, and CodeQwen-7B. Results show
that, without the need for any fine-tuning, GPT-4 can successfully help
programs recover from 72.8% of runtime errors, highlighting the potential of
LLMs in handling runtime errors.