ELECTRA actually uses a special pretraining script, which would need to be written during the sprint - if you are very motivated I’m willing to confirm this “single-person” project and give you a TPU VM. Please leave a comment on the official googe sheet in this case: Confirmed teams for Flax/JAX community week - Google Sheets 6