Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, NLP.
sampled to be starting indices and mask the subsequent ten time-steps. This results in approximately 49% of all time-steps to be masked for a typical training sequence. For language, tokens are masked following BERT. 15% of uniformly selected tokens: 80% are replaced by a learned mask token, 10%