I like this project! Will note it down  Maybe we’ll find some more people later on. Note that this project requires you guys to use a non-transformer model definition: GitHub - kingoflolz/mesh-transformer-jax: Model parallel transformers in JAX and Haiku 8 and model-paralellism in JAX (given the size of the model). Nevertheless I think it’s a feasible project!
Noting it down