@abhilashpal, that’s what it looks like to me. I don’t have any direct experience with the distillation examples, but I took a quick look at the Distillbert and Bert paper and 512 looks like their max token length.