## Granting GPT-4 License and Opportunity: Enhancing Accuracy and Confidence Estimation for Few-Shot Event Detection

## Steven Fincke

University of Southern California Information Sciences Institute sfincke@isi.edu

## Adrien Bibal

## Elizabeth Boschee

InferLink abibal@inferlink.com

## Abstract

Large Language Models (LLMs) such as GPT4 have shown enough promise in the few-shot learning context to suggest use in the generation of "silver" data and refinement of new ontologies through iterative application and review. Such workflows become more effective with reliable confidence estimation. Unfortunately, confidence estimation is a documented weakness of models such as GPT-4, and established methods to compensate require significant additional complexity and computation. The present effort explores methods for effective confidence estimation with GPT-4 with few-shot learning for event detection in the BETTER ontology as a vehicle. The key innovation is expanding the prompt and task presented to GPT-4 to provide L icense to speculate when unsure a nd O pportunity to quantify and explain its uncertainty (L&amp;O). This approach improves accuracy and provides usable confidence measures (0.759 AUC) with no additional machinery.

## 1 Introduction

Large language models such as GPT-4 have shown particular utility in the few-shot learning context, offering the promise of facilitating the creation of large annotation sets for new tasks. This study explores prompting strategies to enhance confidence estimation, which we refer to as L icense &amp; O pportunity (L&amp;O). Our task is the detection of events in English-language news stories in the BETTER ontology (Mckinnon and Rubino, 2022). L&amp;O simply uses one query to the LLM to obtain the output along with the confidence estimation, and it does not require access to the LLMs internal statistics, nor any LLM fine-tuning. Reliable confidence estimation is particularly useful when such a tool is used for tasks with categories that are

University of Southern California Information Sciences Institute boschee@isi.edu easily confused, leading to low inter-rater agreement. It also guides the evaluation of LLM output, both for purposes of refining an ontology under development and prioritizing the review of "silver" annotations.

## 2 Related Work

Few-shot learning has been successfully applied with neural LMs, including LLMs, to the task of event detection (Barth, 2022; Gao et al., 2024); i.e., the detection of event phrases ( triggers or anchors ) and labeling the according to a specified ontology of event types. Such methods have even been applied to highly inclusive event ontologies such as Open Information Extraction (Ling et al., 2023; McCusker, 2023). Deng et al. (2023) provide a survey of recent work in information extraction and directly compare various approaches built upon BERT-like LMs and those using LLMs; they report 0.539 micro F1 for the full ACE05 event detection task using a 5-shot training strategy with GPT-4, where the SOTA fine-tuning a BERT-like LM with a full training set is at 0.837 micro F1.

Confidence estimation, however, has not been a focus of prior scholarship on few-shot event detection with LLMs. If we consider a broader array of NLP tasks, we see that various methods have been developed to extract reliable confidence estimations from LLMs. The fact that GPT-4 usually produces high confidence values when asked naively significantly complicates the task (Singh et al., 2023). Many studies frame the problem as confidence calibration ; that is calibrating the output confidence probability to the actual observed probability of correctness in a labeled dataset (Guo et al., 2017; Tian et al., 2023). One family of approaches exploits the internal statistics of the LLM, such as the log-probs of tokens conveying a particular answer.

Table 1: Selected topics within the BETTER Basic ontology (Mckinnon and Rubino, 2022). ( NPI : "nonpharmaceutical intervention")

| Topic    | Description                                       | Event types                                                                                                                                                                                                           |
|----------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Law      | Law enforcement and judicial actions              | Law-Enforcement-Arrest, Law-Enforcement-Other, Judicial- Indict, Judicial-Prosecute, Judicial-Convict, Judicial-Sentence, Judicial-Acquit, Judicial-Seize, Judicial-Plead, Judicial-Other                             |
| Violence | Associated with mili- tary action and terror- ism | Violence, Violence-Attack, Violence-Bombing, Violence-Kill, Violence-Wound, Violence-Damage, Violence-Other Kidnap- ping                                                                                              |
| Disease  | Disease outbreaks and responses                   | Impose-Quarantine, Apply-NPI, Hospitalize, Vaccinate, Test- Patient, Treat-Patient, Conduct-Medical-Research, Disease- Outbreak, Disease-Infects, Disease-Exposes, Disease-Kills, Disease-Recovery, Restrict-Business |

For example, Wu et al. (2024) compare variants of "Einstein was born in the year X ", where X is 1878 , 1879 , or 1880 . They observe the log probabilities for the various year strings, hopefully providing the highest probability for 1879 . Such white box approaches contrast with black box techniques which do not use such internal statistics. Some find an advantage in using natural language expressions of confidence instead of generating numbers (Lin et al., 2022; Tian et al., 2023). These techniques, e.g. Singh et al. (2023), often involve complex, multi-stage strategies such as "Chain of Thought" (Wei et al., 2023) and "Tree of Thoughts" (Yao et al., 2023).

formation extraction from news stories in the crosslingual context. The Basic portion has an event ontology which eventually expanded, in the third and final phase of the program, to 114 categories; these are grouped into 12 topics ranging from crime to finance. All the annotations for development are for English texts. The full three phases provided annotations for 732 English-language news stories. However, the program evaluations only considered performance on texts in Arabic, Farsi, Russian, Chinese, and Korean.

One important feature of L&amp;O (our approach) is requesting explanations in addition to answers and confidence ratings. Some pre-existing "black-box" methods also include the generation of explanations as part of their confidence pipelines. Li et al. (2024) generate justifications for each of a few possible answers and then estimates confidence from these explanations. Xiong et al. (2024) report that their self probing prompting method is particularly effective for GPT-4; this approach considers each possible answer to a question separately and requests an explanation and confidence; these are reviewed together to generate normalized confidence levels for all the options. Unlike L&amp;O, these do not provide explanations along with the final confidence output.

## 3 Data and Task

We utilize here a portion of IARPA's BETTER task (Mckinnon and Rubino, 2022), which focuses on in-

An effective solution to the BETTER Basic event extraction task uses BIO token labeling to mark event phrases for the full ontology in one pass (Jenkins et al., 2023). XLM-RoBERTa-large (Conneau et al., 2020) was fine-tuned to the full training set with the base model providing effective zeroshot cross-lingual transfer from the English training data to the BETTER program languages.

By contrast, this effort utilizes few-shot learning with a single submission to an LLM. We eliminate the cross-lingual aspect, evaluating on English data, instead. We ask the LLM to mark only the beginning of each event phrase with a vertical pipe (|); we do not ask for the entire span because of the difficulty of crafting effective guidelines for selecting the exact scope of phrases in the few-shot context. For resource reasons, we constrain our efforts to only three of the twelve basic topics, as detailed in Table 1. We also exclude all event types with fewer than 10 instances in all our available annotations.

## 4 System

We provide example prompts and raw output in Figure 1. We will first highlight some important features of our approach and then explain details. L&amp;O adopts two related strategies: 1) urging the LLM to provide guesses when in doubt, and 2) providing the LLM ample opportunity to characterize uncertainty in generating the event type label (or lack thereof). The latter was provided, in part, to allow consumers of the output to differentiate between outright guesses and confident answers.

The LLM is prompted to provide a confidence rating ranging 1-5, where 5 indicates the highest confidence in the presence of an event of the specified type in the sentence, and 1 marks the greatest confidence in the absence of the event type. 1 is appropriate when no event instances are predicted, as well as for very unlikely guesses. Unlike more conventional approaches, this does not convey the LLM's confidence in its answer, regardless of its content; the consequences of such a strategy are discussed in Section 5.

A complete response also includes an explanation. Additionally, the LLM is encouraged to ask a (fictional) expert yes/no questions about the specific event type, with an eye on refining annotation conventions. These requests are inspired by techniques such as CoT (Wei et al., 2023), but our prompt provides no example of expected explanations or yes/no questions and does not encourage decomposing the task into simpler logical steps. Though not explicitly encouraged by our prompt, the LLM often uses the explanation to account for its confidence rating, not just the presence or absence of the label.

## 4.1 Task and prompting details

Each topic is treated as a separate task, and each query to the LLM asks for labels for a single event type at a time, but the LLM is told which other event types fall within the same topic. This design simplifies the task by allowing the LLM to focus on one event type at a time; it increases the possibility of accuracy suffering due to incompatible outputs for different event types for the same sentence, but the scope of this effort does not include a mechanism for reconciling output for related event types.

Our prompt provides up to 5 sentences with at least one instance of the event type in question marked with a vertical pipe before the first word of each event phrase. To select these, we group each instance of the given event type by the lemma of the first event word, ignoring those with only one instance, and then randomly sample according to lexical type. If fewer than five lexical types are attested, more examples are taken from more frequent types. All few-shot examples are excluded from the testing pool, as well as all sentences with fewer than 25 characters. The test repeats a cycle of one sentence with at least one event in the chosen topic in the reference annotations and then three sentences with no in-topic events. The prompt lists the names of other event types in the same topic and notes that no word can bear more than one event type label, i.e., they are mutually exclusive.

The LLM provides its answer by repeating the sentence with vertical pipes marking the beginning of event phrases; our scripts include text alignment code to be robust to the imperfections in the LLM's copy. If there are no instances of the event in the sentence, the LLM is simply instructed to repeat the whole sentence unaltered. As discussed earlier, the LLM is also asked to provide a confidence rating, explanation and a list of yes/no questions. When more than one event phrase is indicated, the LLM generates a separate triplet for each event phrase.

## 4.2 Scoring

We evaluate the LLM's predictions according to the output confidence level. The explanation and yes/no questions are not used by any subsequent step (unlike Li et al., 2024; Xiong et al., 2024) and not in adjudication, but Section 5 will demonstrate their contribution to system performance. A response is judged correct if the marked word coincides with the first word of a reference event phrase of the specified type. No credit is given for marking another word within a phrase or indicating a related but different event type. If the reference has multiple phrases for the same event, the LLM is expected to mark the beginning of each. As we sweep our confidence levels, we include all the outputs for higher levels of confidence (if any). We characterize the performance for each topic with precision, recall, and macro F1 score of the confidence level with the highest F1. We also compute a ROC AUC 1 : for each positive output from our system, we provide the generated confidence score and a label for correctness. This statistic indicates the probability of a randomly selected true positive having a

1 roc\_auc\_score from sklearn

## Prompt; Part 1

You are an expert at finding and classifying events in news stories. The following are examples Of the event type: Disease-Kills In each , the event phrase is preceded immediately by vertical pipe

Lexically diverse 5-shot training

## Other event for types same topic

Mutual exclusivity

Plea to guess and explain

Conf. level description

Explain response

## Prompt, Part 2

The southern African country, which has far reported more than 35 , 000 COVID-19 cases and 1,410 deaths aims to vaccinate 608 Of its nearly 16 million population in three phases On Friday also, two Covid patients had died

Iran to

82

More than 11,000 people had been killed by 2016 Sources from the two foundations said the aid was made to help the Spanish government combat the COVID-19 which has s0 far affected over 11,000 people in Spain and lclaimed almost 500 lives

COVID-19

death

Itoll drops

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00914v1-figure-20dc67dc-4810-49e9-af0d-586f6772966d.jpg)

Use the following json format:

[RESPONSE , [ [ CONFIDENCE SCORE EXPLANATION , YES NO\_QUESTIONS ] , [CONFIDENCE\_SCORE , EXPLANATION YES NO\_QUESTIONS ]

All output , including your response, should appear inside this json list.

Consider the following sentence:

First Egyptian dies from MERS in Aswan: Al-Ahram

Plea to guess and explain

Provide your response and explanation the sentence in the specified json If you are not confident in marking an event phrase, mark it anyway and indicate your concerns with your confidence rating explanation and yes no questions Generate no additional text for

Sample output

Figure 1: Sample prompt and output for Disease-Kills within the Disease topic.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00914v1-figure-c993c19c-b9d3-4939-8b52-458325a9612f.jpg)

higher confidence score than a randomly selected false positive. Table 2 presents performance for three selected topics: Law Violence , , and Disease . We have no external baseline for our version of the Basic task, but we can provide some related reference points. The SOTA for event detection from English text for the full BETTER Basic task ranges 0.642-0.646 F1, varying according to phase variants of the task (Jenkins et al., 2023). The ACE05 event detection has been well explored; on average, it is somewhat easier than BETTER Basic;

Jenkins et al. (2023) report 0.712 F1 for English ACE05. Deng et al. (2023) cite 0.837 F1 as the SOTA for the full English ACE05, but 0.539 F1 with GPT-4 employing a few-shot strategy. This prior work leads us to regard the performance of L&amp;O as reasonably accurate.

To facilitate the interpretation of the AUC scores, we include an ROC-style plot of the performance for the three topics: Figure 2; The portions of true positives and false positives are calculated in the same manner as Table 2: i.e., lower levels of con-

Table 2: Precision, recall, F1 and AUC for the topperforming confidence level for selected topics.

| Topic    |     P |     R |    F1 |   AUC |
|----------|-------|-------|-------|-------|
| Law      | 0.466 | 0.547 | 0.503 | 0.774 |
| Violence | 0.432 | 0.451 | 0.441 | 0.729 |
| Disease  | 0.491 | 0.396 | 0.439 | 0.645 |

fidence include all predictions with higher confidence. (Recall that the best performance is towards the top-left corner of such graphs.) Please note, however, that the diameter of each plot point is proportional to the number of instances at an indicated confidence level. This plot shows that most

Figure 2: AUC plot for three topics with diameter proportional to the number of outputs at the specified confidence level.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00914v1-figure-3109d3ec-3cb2-4ef1-9afd-a3aa2fa6ec7f.jpg)

predictions have a confidence score of 5 or 4, with significant additions of both true and false positives when including 4. The values for 5 and 4 are remarkably similar, indicating that GPT-4 was consistent in determining the precision/recall trade off between 5 and 4 in the separate runs. All three have AUC values much greater than chance at 0.5; indeed, Law and Violence are above 0.7.

## 5 Ablation studies

We will explore here the impact of various design features of L&amp;O. To contain this effort, we will focus on the initial 210 sentences of the run for the Law topic. In one set of variants, we exclude components of L&amp;O that are means for the LLM to characterize its stance to its output. First, we request a confidence level for each output but do not ask for an explanation or any yes/no questions. We add a variant of this where the confidence score is more conventional ; i.e., the prompt asks the LLM to generate a 5 if it is highly confident in its answer and 1 if quite uncertain regardless of the content of the answer; a 5 score can be provided when highly confident in the absence, as well as presence, of an event. We also consider the inverse, where the prompt requests both an explanation and list of yes/no questions but no confidence score. We exclude our L&amp;O pleas for guessing with explanation in the last variant. Table 3 indicates performance on our 210-sentence Law subset for our full system and these ablations.

Table 3: Baseline for the Law topic with ablations excluding explanation and yes/no questions.

| Accounting       |     P |     R |    F1 |   AUC |
|------------------|-------|-------|-------|-------|
| Full             | 0.543 | 0.667 | 0.599 | 0.759 |
| Conf. only       | 0.775 | 0.378 | 0.508 | 0.818 |
| Conv. conf. only | 0.35  | 0.448 | 0.393 | 0.652 |
| No conf.         | 0.417 | 0.577 | 0.484 | 0.5   |
| No guess         | 0.54  | 0.627 | 0.58  | 0.58  |

Requesting confidence only, i.e., dropping the explanation and yes/no questions, degrades F1 somewhat, especially for recall, but the AUC is actually somewhat better. Since explanations often provide rationales for the confidence rating, not just the labeling decision, it appears that GPT-4 generates better confidence rating when given the opportunity to account for its ouptut. However, the conventional variant, where 5 is requested when the LLM is highly confident in both positive labels and the lack thereof, is markedly worse in terms of F1 and AUC. Excluding the confidence ratings necessarily reduces the AUC to chance, but the F1 is comparable to Confidence only . Quite strikingly, dropping the appeals to guess leaves F1 largely unchanged in relation to the full system, except for a modest drop in recall, but the confidence scores lose most of their value, and the AUC is only 0.080 above chance.

Figure 3 is an ROC plot for all configurations producing confidence scores. The true positive and false positives reflect the inclusion of predictions with higher confidence, and the diameter of points is proportional to the count for each confidence label. When we compare the confidence-only system to its "conventional" variant, we notice that

Figure 3: AUC plot for individual confidence levels for our full system and ablation variants. The diameter proportional to the number of outputs at the specified confidence level.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00914v1-figure-4a17cf07-d246-4b99-83e2-60bc00169fec.jpg)

the performance for level 5 shifts dramatically to the 0.5 random-choice line, encompassing a greater share of false positives and a smaller portion of true positives. We also see that no guess provides the best AUC performance of all systems at level 5, but it presents no meaningful alternation with 4 (or any other confidence level.)

## 6 Discussion

This study shows that GPT-4 provides increasingly better output as it is given more opportunity to characterize and explain its response. This is more effective when the structure avoids complicating the logical structure of the task. For example, we interpret the absolute 0.111 F1 gain from using our default confidence scoring scheme instead of the conventional one as indicating that GPT-4 performs better when fewer logical operations are required. The task already probes the presence or absence of events of the given type; we maintain consistency by allowing the LLM to provide confidence in the presence of the event. The conventional approach requires an additional level of indirection, by asking for a characterization of the LLMs response, not the input data.

If we also urge GPT-4 to guess and explain itself when uncertain, we get useful confidence estimations and a modest improvement in recall. Without these appeals, GPT-4's event extraction output is rather similar: it continues to speculate, but the confidence scores no longer indicate less certain outputs. We observe that GPT-4 is capable of directly indicating its level of confidence, but it needs to be given explicit license to speculate along with ample means to provide its stance to its response. GPT-4 has clearly been subject to significant examination and public scrutiny; it is not an opensource effort, but we speculate that L&amp;O bypasses features imposed onto GPT-4 to guard against embarrassing output. Gaining confidence scores with a useful AUC significantly increases the utility of models such as GPT-4 in developing and extending modest annotation resources for tasks such as event detection.

## 7 Future Work

Though modest in scope, this study presents a promising technique for eliciting useful confidence judgments from GPT-4 while improving F1 in the few-shot setting. Various additional lines of research would expand our understanding of value of this approach. First, additional baselines would be helpful. Evaluating a BERT-like model on the present version of the BETTER Basic event detection task, both few-shot and fine-tuned, would facilitate the assessment of the accuracy of L&amp;O. A baseline for confidence estimation using a white-box technique, such as the use of the logprobabilities from GPT-4 for vertical pipes in output would help contextualize our reported AUC values.

Second, we did not explore the issue of calibration error : many systems express confidence as probability of correctness, and analyses examine the statistical gap between these figures and observed accuracy rates. Instead, we only requested confidence scores 1-5, and we did not attempt to associate each of these scores with specific precision rates. We would also like to explore the use of verbalized confidence (Tian et al., 2023), which could easily be applied to the present task.

Other potential lines of study include the application of L&amp;O to LLMs other than GPT-4 and tasks beyond our variant of BETTER. We observed that prediction in the presence of a label was associated with better F1 and AUC than confidence in the correctness of the response, regardless of its content. Extending this logic to some NLP tasks could be challenging, but we would like to better understand the scope of applicability and relevance of this technique.

## 8 Conclusion

This study presents L&amp;O, which combines a set of prompting techniques for effectively performing the event detection task with GPT-4 in the few-shot setting. More importantly, we provide a strategy for exposing GPT-4's capacity to provide useful confidence scores. This crucially depends upon urging the model to speculate and explain: simply providing the opportunity to explain is not enough. We suspect that GPT-4 has been engineered to default to indicating high certainty, and our appeal lifts this "hold" on the model's functionality. However, the design and breadth of opportunities to explain also impact performance. Eliminating explanations and yes/no questions lowers F1 and AUC. We interpret this, in part, as GPT-4 being designed to avoid "traps": it indicates more of its weaknesses when encouraged to do so. The venues for characterizing uncertainty need to be ample but also well suited to the task presented to the model, as shown with the degradation employing the "conventional" confidence approach.

## 9 Limitations

Section 7 described various limits of this study which could be addressed in later efforts. We also note that we only used GPT-4 and only examined annotation of English texts.

## Ethics Statement

This system enhances F1 and confidence estimation, but many errors remain: users cannot assume that system output is accurate, when marked with high confidence.

## Acknowledgements

This material is based upon work supported by the US Air Force under Contract No. FA8750-22-C0511. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the US Air Force. We also benefited from the feedback and support of Steve Minton, Goran Muric, and Michael Ross.

## References

Fabio Barth. 2022. Few-shot learning for biomedical event extraction with large generative language models.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 84408451, Online. Association for Computational Linguistics.

Shumin Deng, Yubo Ma, Ningyu Zhang, Yixin Cao, and Bryan Hooi. 2023. Information extraction in low-resource scenarios: Survey and perspective.

Jun Gao, Huan Zhao, Wei Wang, Changlong Yu, and Ruifeng Xu. 2024. Eventrl: Enhancing event extraction with outcome supervision for large language models.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks.

Chris Jenkins, Shantanu Agarwal, Joel Barry, Steven Fincke, and Elizabeth Boschee. 2023. Massively multi-lingual event understanding: Extraction, visualization, and search. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages 247-256, Toronto, Canada. Association for Computational Linguistics.

Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024. Think twice before assure: Confidence estimation for large language models through reflection on multiple answers.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words.

Chen Ling, Xujiang Zhao, Xuchao Zhang, Yanchi Liu, Wei Cheng, Haoyu Wang, Zhengzhang Chen, Takao Osaki, Katsushi Matsuda, Haifeng Chen, and Liang Zhao. 2023. Improving open information extraction with large language models: A study on demonstration uncertainty.

Jamie McCusker. 2023. Loke: Linked open knowledge extraction for automated knowledge graph construction.

Timothy Mckinnon and Carl Rubino. 2022. The IARPA BETTER program abstract task four new semantically annotated corpora from IARPA's BETTER program. In Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages 35953600, Marseille, France. European Language Resources Association.

Aniket Kumar Singh, Suman Devkota, Bishal Lamichhane, Uttam Dhakal, and Chandra Dhakal. 2023. The confidence-competence gap in large language models: A cognitive study.

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 5433-5442, Singapore. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.

Qinyuan Wu, Mohammad Aflah Khan, Soumi Das, Vedant Nanda, Bishwamittra Ghosh, Camila Kolling, Till Speicher, Laurent Bindschaedler, Krishna P. Gummadi, and Evimaria Terzi. 2024. Towards reliable latent knowledge estimation in llms: In-context learning vs. prompting based factual knowledge extraction.

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.
Steven Fincke Adrien Bibal Elizabeth Boschee
Granting GPT-4 License and Opportunity: Enhancing Accuracy and Confidence Estimation for Few-Shot Event Detection
Large Language Models (LLMs) such as GPT-4 have shown enough promise in the
few-shot learning context to suggest use in the generation of "silver" data and
refinement of new ontologies through iterative application and review. Such
workflows become more effective with reliable confidence estimation.
Unfortunately, confidence estimation is a documented weakness of models such as
GPT-4, and established methods to compensate require significant additional
complexity and computation. The present effort explores methods for effective
confidence estimation with GPT-4 with few-shot learning for event detection in
the BETTER ontology as a vehicle. The key innovation is expanding the prompt
and task presented to GPT-4 to provide License to speculate when unsure and
Opportunity to quantify and explain its uncertainty (L&O). This approach
improves accuracy and provides usable confidence measures (0.759 AUC) with no
additional machinery.