Hi, @ptrblck! Thanks for interested in this question.
I’m doing a multi-label classification task, and the label space is about 8900. The classifier needs to make predictions about  what labels the input text corresponds to (generally, an input text might correspond to 5~10 labels).
As described in this paper: https://arxiv.org/pdf/1909.11386.pdf 1, I’m gonna do a per-label mask of the input embedding [B, L, D] (batch size, input length, embed dimension), after the mask, the embedding would become a 4-D tensor [B, L, T, D]. Then I will do convolution.
The original paper suggests that all embedding share the same convolution layer, which means all label embedding should be convolved by the same weights. For simplicity, we could stack the 4-D tensor at the embedding dimension, then it has the shape [B, L, T*D], which is suitable for depthwise convolution.
However, if we directly use 1-D convolution, there will be one unique filter for each label embedding, and there will be 8900 different filters in total, which can be a disaster for GPU memory.  I’m wondering if there is a method to make the filters share the same parameters.