Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, NLP.
masked, by replacing them with a learned MASK embedding token. For computer vision, the block-wise masking strategy in BEiT is followed, but it masks 60% of the patches instead of 40%. For speech, spans of latent speech representations are masked (wav2vec 2.0). p = 0.065 of all time-steps are