Description: We will pretrain a large BART model on The Pile, and measure a performance increase downstream. Potentially we could also add rotary embeddings?
Model: BART (1b+)
Dataset: The Pile
Training scripts: Training scripts will be written as part of the project. Data processing scripts can be taken from GPT-J 6b
Expected result: An adaptable JAX pipeline for training seq2seq models like BART on the pile.