Guidance for continued pre-training of BART with de-noising.
# 🚀 Feature request

An example of continued pre-training of BART with de-noising.

## Motivation

I'm using the run causal LM [script](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py), but it seems on line 340, it's simply copying input to output (learning the identify function).  

1. Would line 340 be the best place to add the de-noising or should I introduce it as part of the collator?
2. Is there any code which implements de-noising in HuggingFace?  BART defines 4-5 main operations which should be easy to reproduce - I just don't want to introduce new code if well-tested code already exists.


Thanks!!!

also it is my questions, thanks <|||||>denoising function is a part of T5 pretraining as well, is there a denoising function implementation in Huggingface repo ? Any advice is appreciated. thanks <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.