Difference in the architecture of openai whisper and huggingface whisper
I was comparing the whisper-medium which I got from OpenAI directly (from https://github.com/openai/whisper.git) with the HuggingFace whisper-medium.
In the decoder part of the model, both has 24 decoder blocks but there is a difference in the block architecture between openai's and huggingFaces.
As it is represented below, In OpenAI whisper decoder, there is a GELU activation function between two linear layer (at mlp block) but in HuggingFace whisper decoder, there isn't that GELU activation function between fc1 and fc2 block.

![image](https://user-images.githubusercontent.com/8229163/224065417-aa2d7a8f-b385-4e18-975b-7638914ffa44.png)

So based on such difference, is the huggignface whisper model works similar to openai's one?
And is it a totally a difference between these two whisper-medium models?
@sanchit-gandhi 


Hey @hannan72!

We register the activation function here (`activation_fn` in the state dict):
https://github.com/huggingface/transformers/blob/fdf84096565b8d2e15de35ac0cd86818c4b12adb/src/transformers/models/whisper/modeling_whisper.py#L468

And then apply it directly after the first feedforward layer (`fc1`):
https://github.com/huggingface/transformers/blob/fdf84096565b8d2e15de35ac0cd86818c4b12adb/src/transformers/models/whisper/modeling_whisper.py#L556

If you check the config for Whisper, you'll see that this activation function defaults to GELU:
https://github.com/huggingface/transformers/blob/fdf84096565b8d2e15de35ac0cd86818c4b12adb/src/transformers/models/whisper/configuration_whisper.py#L106

So the two are entirely equivalent 👍 OpenAI just register the GELU in a sequential block, we register it standalone. But both apply it in the same place.<|||||>Thank you a lot @sanchit-gandhi for your clear answer!