@lewtun Thanks for the quick reply! I did check it out but there is only a code block on how to calculate the loss for pretraining but no other implementation details which are also critical. Do you know whether there is code on that?