uTeBC-NLP at SemEval-2024 Task 9: Can LLMs be Lateral Thinkers?
<|reference_start|>uTeBC-NLP at SemEval-2024 Task 9: Can LLMs be Lateral Thinkers?: Inspired by human cognition, Jiang et al.(2023c) create a benchmark for assessing LLMs' lateral thinking-thinking outside the box. Building upon this benchmark, we investigate how different prompting methods enhance LLMs' performance on this task to reveal their inherent power for outside-the-box thinking ability. Through participating in SemEval-2024, task 9, Sentence Puzzle sub-task, we explore prompt engineering methods: chain of thoughts (CoT) and direct prompting, enhancing with informative descriptions, and employing contextualizing prompts using a retrieval augmented generation (RAG) pipeline. Our experiments involve three LLMs including GPT-3.5, GPT-4, and Zephyr-7B-beta. We generate a dataset of thinking paths between riddles and options using GPT-4, validated by humans for quality. Findings indicate that compressed informative prompts enhance performance. Dynamic in-context learning enhances model performance significantly. Furthermore, fine-tuning Zephyr on our dataset enhances performance across other commonsense datasets, underscoring the value of innovative thinking.<|reference_end|>
@article{sadeghi2024utebc-nlp,
    title={uTeBC-NLP at SemEval-2024 Task 9: Can LLMs be Lateral Thinkers?},
    author={Pouya Sadeghi and Amirhossein Abaskohi and Yadollah Yaghoobzadeh},
    journal={arXiv preprint arXiv:2404.02474},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.02474},
    primaryClass={cs.CL cs.AI cs.IR cs.LG}
}