distilled bart-large/bart-base
@sshleifer Is there a distilled bart model (not CNN/XSUM) model available?

Thanks!
Hi @kkissmart 
Yes, distilbart models are available [here](https://huggingface.co/sshleifer/distilbart-cnn-6-6)<|||||>@patil-suraj I meant a distill Bart ( a pretrain model) not a summarization model. Do I misunderstand the name?<|||||>No, we don't have that.<|||||>Ohh, AFAIK there is no pre-trained distilbart like distilbert. 
There are two types of distillation 
1. No teacher distillation: which copies alternate layers from the pre-trained model and creates a small student model.
2. With teacher distillation: enforce that the student and teacher produce similar encoder_outputs, logits, and hidden_states

You can easily create a student (no teacher) model using the scripts [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#no-teacher-distillation) , you'll just need to use `bart-large` instead of `bart-large-cnn`.

pre-training distilbart is still not included, however you can train the large model on the down-stream task and then do with teacher distillation for a smaller distilled model.<|||||>Also, consider asking such non-bug questions on the forum https://discuss.huggingface.co/ -:)<|||||>Is there a distilBART base model that does not have pretaining weights?<|||||>There is no distilbart-base model.
There are only distilled models fine-tuned on summarization tasks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@sshleifer @patil-suraj  
```
You can easily create a student (no teacher) model using the scripts here , you'll just need to use bart-large instead of bart-large-cnn.
https://github.com/huggingface/transformers/tree/master/examples/seq2seq#no-teacher-distillation or
https://github.com/huggingface/transformers/tree/master/examples/legacy/seq2seq#no-teacher-distillation ?
```
I want to make a DistilBART model from my japanese BART-large mode. but, No script in this link. Has the script been kept private? I want to see the script.
<|||||>Hi @hisashi-ito 

The seq2seq distillation scripts are now moved under `examples/research_projects/seq2seq-distillation` directory. You can find them here.
https://github.com/huggingface/transformers/tree/master/examples/research_projects/seq2seq-distillation<|||||>Hi @patil-suraj
Thank you for teaching !! 