Hi
This is a repost from here 1, as I’m hoping someone in this forum might have some additional expertise in feature fusion.
Is it possible to perform feature fusion using a self-attention mechanism?  I have data distributed over a graph, and multiple feature modalities sampled per node that I would like to “optimally” combine for a node classification problem.  Let’s say that I have a graph $G=(V,E)$ with $N$ nodes.  For each node, we sample $M$ different modalities, such that each node $v$ is characterized by $M$ feature vectors $\in \mathbb{R}^{m_{1}}, \mathbb{R}^{m_{2}}…\mathbb{R}^{m_{M}}$.  For simplicity, assume $m_{1} = m_{2} = … = m_{M} = D$.  I’m interested in “fusing” these features in a  more intelligent way than simply concatenating them and passing them through a linear layer or MLP.
Assuming each node $v$ is characterzed by a feature matrix $x_{i} \in \mathbb{R}^{M \times D}$ (where $M$ is the number of modalities, and $D$ the input dimension), a transformer approach over modalities would yield something like this:
$$
\begin{align}
Q = x_{i}W_{q} \
K = x_{i}W_{k} \
V = x_{i}W_{v}
\end{align}
$$
where $W_{q,k,v} \in \mathbb{R}^{D \times p}.  We then have that
$$
\begin{align}
Z = softmax(\frac{QK^{T}}{\sqrt{D}}, axis=1)V
\end{align}
$$
where $Z \in \mathbb{R}^{M \times D} and $Z_{i}$ is the fusion of modalities with respect to modality $i$ (I think – correct me if I’m wrong, please).  I could then sum over the rows of $Z$ to compute the final combination.  Does this seem correct?  I havn’t found many papers on using self-attention for feature fusion, so any help is much appreciated.
k