Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
text
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
new_line
new_line
Paper: 
text
Paper: 
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning (2404.03323)
https://huggingface.co/papers/2404.03323
2404.03323
paper
resource
https://huggingface.co/papers/2404.03323
new_line
new_line
Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 
text
Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 
new_line
new_line
Key concepts:
text
Key concepts:
new_line
– Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
text
– Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
new_line
– Sparse $\ell_1$ regularization.
text
– Sparse $\ell_1$ regularization.
new_line
– Contrastive loss for inner layers of the model.
text
– Contrastive loss for inner layers of the model.
new_line
new_line
Methodology:
text
Methodology:
new_line
The approach consists of three main steps:
text
The approach consists of three main steps:
new_line
– Create a set of concepts based on the labels of the dataset.
text
– Create a set of concepts based on the labels of the dataset.
new_line
– Supply a multi-modal encoder with CBL.
text
– Supply a multi-modal encoder with CBL.
new_line
– Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
text
– Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.
new_line
new_line
Results and Analysis:
text
Results and Analysis:
new_line
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
text
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.
new_line
new_line
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning

Paper: https://huggingface.co/papers/2404.03323

Authors propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs): they introduce a new type of layers known as Concept Bottleneck Layers (CBL), and present three methods for training them: with $\ell_1$-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy. They show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models. 

Key concepts:
– Contrastive Gumbel-Softmax loss: the first contrastive variant of Gumbel-Softmax objective which achieves an inner sparse representation of the Concept Bottleneck Layer activations. 
– Sparse $\ell_1$ regularization.
– Contrastive loss for inner layers of the model.

Methodology:
The approach consists of three main steps:
– Create a set of concepts based on the labels of the dataset.
– Supply a multi-modal encoder with CBL.
– Train this CBL with the picked objective function and train the classifiers head with Cross-Entropy.

Results and Analysis:
The methodology can be applied to the task of interpreted image classification. And the experimental results show the superiority of using sparse hidden representations of concepts.


🚀
Andron00e tit33 EffyOsvin
👀
Andron00e EffyOsvin
🔥
EffyOsvin