Intelligence Analysis of Language Models
<|reference_start|>Intelligence Analysis of Language Models: In this project, we test the effectiveness of Large Language Models (LLMs) on the Abstraction and Reasoning Corpus (ARC) dataset. This dataset serves as a representative benchmark for testing abstract reasoning abilities, requiring a fundamental understanding of key concepts such as object identification, basic counting, and elementary geometric principles. Tasks from this dataset are converted into a prompt-based format for evaluation. Initially, we assess the models' potential through a Zero-shot approach. Subsequently, we investigate the application of the Chain-of-Thought (CoT) technique, aiming to determine its role in improving model performance. Our results suggest that, despite the high expectations placed on contemporary LLMs, these models still struggle in non-linguistic domains, even when dealing with simpler subsets of the ARC dataset. Our study is the first to concentrate on the capabilities of open-source models in this context. The code, dataset, and prompts supporting this project's findings can be found in our GitHub repository, accessible at: https://github.com/Lianga2000/LLMsOnARC.<|reference_end|>
@article{galanti2024intelligence,
    title={Intelligence Analysis of Language Models},
    author={Liane Galanti and Ethan Baron},
    journal={arXiv preprint arXiv:2407.18968},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.18968},
    primaryClass={cs.AI}
}