Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Transformers.
expansion ratio of FFN, dk and dv are dimensions of key and value, respectively. ViT sets d = dk = dv and r = 4, the cost can be simplified as: The FLOPs of CMT block: where k ≥ 1 is the reduction ratio in LMHSA. Compared to standard Transformer block, the CMT block is more friendly to