content.raw: Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
content.type: text
content.value: Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
content.type: new_line
content.type: new_line
content.raw: Paper: 
content.type: text
content.value: Paper: 
content.label: Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning (2404.03323)
content.raw: https://huggingface.co/papers/2404.03323
content.id: 2404.03323
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2404.03323
content.type: new_line
content.type: new_line
content.raw: Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 
content.type: text
content.value: Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 
content.type: new_line
content.type: new_line
content.raw: Key concepts:
content.type: text
content.value: Key concepts:
content.type: new_line
content.raw: – Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
content.type: text
content.value: – Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
content.type: new_line
content.raw: – Sparse $\ell_1$ regularization.
content.type: text
content.value: – Sparse $\ell_1$ regularization.
content.type: new_line
content.raw: – Contrastive loss for inner layers of the model.
content.type: text
content.value: – Contrastive loss for inner layers of the model.
content.type: new_line
content.type: new_line
content.raw: Methodology:
content.type: text
content.value: Methodology:
content.type: new_line
content.raw: The approach consists of three main steps:
content.type: text
content.value: The approach consists of three main steps:
content.type: new_line
content.raw: – Create a set of concepts based on the labels of the dataset.
content.type: text
content.value: – Create a set of concepts based on the labels of the dataset.
content.type: new_line
content.raw: – Supply a multi-modal encoder with CBL.
content.type: text
content.value: – Supply a multi-modal encoder with CBL.
content.type: new_line
content.raw: – Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
content.type: text
content.value: – Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
content.type: new_line
content.type: new_line
content.raw: Results and Analysis:
content.type: text
content.value: Results and Analysis:
content.type: new_line
content.raw: The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
content.type: text
content.value: The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
content.type: new_line
content.type: new_line
rawContent: Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
  
  Paper: https://huggingface.co/papers/2404.03323
  
  Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 
  
  Key concepts:
  – Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
  – Sparse $\ell_1$ regularization.
  – Contrastive loss for inner layers of the model.
  
  Methodology:
  The approach consists of three main steps:
  – Create a set of concepts based on the labels of the dataset.
  – Supply a multi-modal encoder with CBL.
  – Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
  
  Results and Analysis:
  The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
  
  
reactions.reaction: 🚀
reactions.users: Andron00e tit33 EffyOsvin
reactions.reaction: 👀
reactions.users: Andron00e EffyOsvin
reactions.reaction: 🔥
reactions.users: EffyOsvin