Self-supervised Interpretable Concept-based Models for Text Classification
<|reference_start|>Self-supervised Interpretable Concept-based Models for Text Classification: Despite their success, Large-Language Models (LLMs) still face criticism as their lack of interpretability limits their controllability and reliability. Traditional post-hoc interpretation methods, based on attention and gradient-based analysis, offer limited insight into the model's decision-making processes. In the image field, Concept-based models have emerged as explainable-by-design architectures, employing human-interpretable features as intermediate representations. However, these methods have not been yet adapted to textual data, mainly because they require expensive concept annotations, which are impractical for real-world text data. This paper addresses this challenge by proposing a self-supervised Interpretable Concept Embedding Models (ICEMs). We leverage the generalization abilities of LLMs to predict the concepts labels in a self-supervised way, while we deliver the final predictions with an interpretable function. The results of our experiments show that ICEMs can be trained in a self-supervised way achieving similar performance to fully supervised concept-based models and end-to-end black-box ones. Additionally, we show that our models are (i) interpretable, offering meaningful logical explanations for their predictions; (ii) interactable, allowing humans to modify intermediate predictions through concept interventions; and (iii) controllable, guiding the LLMs' decoding process to follow a required decision-making path.<|reference_end|>
@article{de santis2024self-supervised,
    title={Self-supervised Interpretable Concept-based Models for Text
  Classification},
    author={Francesco De Santis, Philippe Bich, Gabriele Ciravegna, Pietro
  Barbiero, Danilo Giordano, Tania Cerquitelli},
    journal={arXiv preprint arXiv:2406.14335},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.14335},
    primaryClass={cs.CL cs.AI}
}