@zuujhyt Maybe try creating different Dataset classes using different DataCollators. You could then use PyTorch Lightning to create a Dataloader w/ multiple Dataset classes like this 68, and train Bart on that. I’ll try doing this.
Let me know if you have any updates on this.