Can I use a smaller base model than allenai/led-base-16384 for LED?
Hello, I'm trying to fine tune my own Longformer Encoder Decoder following this [notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing#scrollTo=jpUr9QeebZ-n). However, I was wondering if there was a way to consider a base model like 

`allenai/longformer-base-4096`

instead of 

`led-base-16384`?

When I try doing
```python
led = AutoModelForSeq2SeqLM.from_pretrained(
    "allenai/longformer-base-4096",
    config="roberta-base",
    gradient_checkpointing=True,
    use_cache=False,
)
```
but that gives me
```
   led = AutoModelForSeq2SeqLM.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/modeling_auto.py", line 1221, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.longformer.configuration_longformer.LongformerConfig'> for this kind of AutoModel: AutoModelForSeq2SeqLM.
Model type should be one of LEDConfig, BlenderbotSmallConfig, MT5Config, T5Config, PegasusConfig, MarianConfig, MBartConfig, BlenderbotConfig, BartConfig, FSMTConfig, EncoderDecoderConfig, XLMProphetNetConfig, ProphetNetConfig.
```

my `encoder_max_length` is only 2048 since I'm not planning on feeding transcripts of 8k word to summarize but rather transcripts of 2k words. So using the smaller base model would work great. I'm basically trying to replicate the fine tuning for this
https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16
^but I would want to use that model more as a checkpoint to fine tune further on domain specific data. Hence why I'm trying to create:
1.) a model fine tuned on cnn dailymail data(basically replicate the [longformer2roberta](https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16) but a version that we can fine tune further)
2.) using part 1 as a checkpoint, fine tune that on domain specific transcript+summary data

@patrickvonplaten or others in the community, I'd greatly appreciate any advice on this
I'd suggest to still use `allenai/led-base-16384` and just pad the input to a maximum length of only `2048`. Also you could think about reducing `config.attention_window` of the model's config to something like 512 or 256 to make your model more efficient for `2048` input<|||||>E.g. in this notebook: https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing, I just pad the max length of every input to `8192` and use `allenai/led-base-16384` which works very well!<|||||>Sounds great, I'll look into setting`led.config.attention_window=512` instead of 1024 and the `max_input_length=2048` for the encoder. Thank you for your feedback!