Update FNet Fourier Transform
# What does this PR do?

This PR changes the Fourier Transform according to the [latest updates](https://github.com/google-research/google-research/commit/4f7815fe3aaf71b811a0fb82ac8768a4e8ab0428#diff-4a2be83f4d2b702b206d3f64b2aeb209ac7a2d65fd28382916225463b984bc2c) to the FNet repository.

I will update the configurations of the checkpoints if the changes look fine.

@patrickvonplaten @LysandreJik @sgugger @patil-suraj 
Thanks for the improved version of the FNet Fourier transform! In order to keep backwards compatibility, we should try to leave all existing arguments and simply add new ones - this would definitely be possible :-)<|||||>@sgugger @patrickvonplaten
Does this mean I cannot remove the custom `fftn` function?

Or just the configuration arguments?
If I keep the old config arguments, won't I have to keep the relevant code as well?<|||||>No, the old function can't be removed for backward compatibility, sadly. We can only add new arguments that trigger some new behavior.<|||||>@sgugger I will fix the test according to the new change.

Also, wanted to ask if there is a possiblity of ever deprecating the old version? Just out of curiosity.<|||||>For modeling code, it's tougher to go through the usual deprecation cycle as we might have models on the Hub using the old code. They would then stop working if we ever removed the old code, which is something we try very hard to avoid.<|||||>@sgugger Also, what do you think about [this comment](https://github.com/huggingface/transformers/issues/13684#issuecomment-924472556)?
In his notebook, he makes the `dft_mat_hidden` and `dft_mat_seq` into parameters so they get pushed to the GPU. This might help speed things up?

Also there is one minor change which allows `model.half()`.

Should I also add the changes here?

CC @patrickvonplaten <|||||>@gchhablani the `model.half` support would be cool to have, I was missing it in my last FNet GPU training 😅<|||||>Can you put the fix for `model.half()` in a new PR @gchhablani ? For the other part of making the attributes parameters, I'm not opposed to do it as long as it's backward compatible.<|||||>@sgugger I checked by doing `model.cuda()` on the current model. Refer to [this colab notebook](https://colab.research.google.com/drive/1fB5GOGuH3xClyTXJ_zagR_G-3Qd7QzMQ?usp=sharing) if needed. The buffers are also pushed to the `cuda:0` device. I don't think any change is needed regarding this. Wdyt?<|||||>@stefan-it Can you please explain the issue you face when you do `model.half()`? In the notebook linked in the previous comment, I have tried `model.half()` as well. Except the buffers, everything becomes `torch.float16`. Should the buffers also move to `torch.complex32` or `torch.complex16`?<|||||>If the buffers are properly pushed, then there is no need to change anything I believe.<|||||>Looks good to me as well! @gchhablani - do you think the new version yields a significant speed-up? In this case it might be worth adding a deprecation warning to the current config.<|||||>@patrickvonplaten With the current changes, there is almost negligible modification with `use_latest`. With default conditions (for GPU), we would be using the `torch.fft.fftn` method as before. 

For TPU long sequences (>4096), there is a difference. Instead of using the custom fftn method, the torch fftn method is used. Not sure how much difference is there between the custom fftn (fft in loop) and the torch method, speed-wise. Maybe I can `timeit` on a CPU for long sequences for the different fft methods?

I just thought that we might need to update to the latest changes as on the original repository.<|||||>CC @ontocord<|||||>Ha! Great to see you all are adding some of this stuff... <|||||>Hi @gchhablani , some feedback:

```bash
Traceback (most recent call last):
  File "run_mlm.py", line 552, in <module>
    main()
  File "run_mlm.py", line 501, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/trainer.py", line 1312, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/trainer.py", line 1839, in training_step
    loss = self.compute_loss(model, inputs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/trainer.py", line 1873, in compute_loss
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/models/fnet/modeling_fnet.py", line 792, in forward
    outputs = self.fnet(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/models/fnet/modeling_fnet.py", line 636, in forward
    encoder_outputs = self.encoder(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/models/fnet/modeling_fnet.py", line 334, in forward
    layer_outputs = layer_module(hidden_states)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/models/fnet/modeling_fnet.py", line 293, in forward
    self_fourier_outputs = self.fourier(hidden_states)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/models/fnet/modeling_fnet.py", line 246, in forward
    self_outputs = self.self(hidden_states)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1015, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/turkish-fnet/transformers-fnet-refactoring/src/transformers/models/fnet/modeling_fnet.py", line 225, in forward
    outputs = self.fourier_transform(hidden_states).real
RuntimeError: Unsupported dtype Half
```
Occurs when using the (PyTorch) language modeling example with `--fp16` option - even with the latest `update_fnet` branch in a `nvcr.io/nvidia/pytorch:21.04-py3` container, see [detailed library versions](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_21-04.html#rel_21-04).
<|||||>@ontocord Can you please explain why you think that the DFT matrices need to made into parameters? If it is just to push the params onto the device, using buffers is doing that.

@patrickvonplaten @sgugger Any idea on how can I fix the issue @stefan-it just mentioned?<|||||>They do not convert to cuda or half automatically unless they are in the same parameter list of a module. I made them no_grad Params. 

> On Oct 1, 2021, at 2:26 PM, Gunjan Chhablani ***@***.***> wrote:
> 
> ﻿
> @ontocord Can you please explain why you think that the DFT matrices need to made into parameters?
> 
> @patrickvonplaten @sgugger Any idea on how can I fix the issue @stefan-it just mentioned?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
> Triage notifications on the go with GitHub Mobile for iOS or Android. 
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale