Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Small Data.
baseline performance of 67.59 is obtained from DeiT-T with 4 heads, training from scratch with 300 epochs. When removing absolute positional embedding, the performance drops drastically to 58.72. When adopting SOPE and removing absolute position embedding, the performance does not drop so