Rename swish to silu to give appropriate credit
The swish was originally coined the "SiLU" in https://arxiv.org/pdf/1606.08415.pdf and https://arxiv.org/abs/1702.03118 long before the swish paper. Renaming other peoples' exact same ideas is unacceptable and huggingface's naming convention implicitly erases the research and work of people outside of Google.
This request inspired by a [discussion](https://www.reddit.com/r/MachineLearning/comments/hkiyir/r_google_has_a_credit_assignment_problem_in/) and a recent [tensorflow issue](https://github.com/tensorflow/tensorflow/issues/41066), but this problem has been brought up every few months for the past few years. In light of recent efforts to make the ML community more equitable and _fair_, this is a no-brainer and long overdue.

**Will this change the current api? How?**
The API would replace the "swish" argument with the "silu" argument and deprecate the swish.
[PyTorch 1.7](https://pytorch.org/docs/1.7.0/generated/torch.nn.SiLU.html?highlight=silu) added the SiLU. Tensorflow added the [SiLU](https://github.com/tensorflow/tensorflow/blob/27d26a8d86bceda282ad9ba3e3116a00759d4ebc/tensorflow/python/ops/nn_impl.py#L517) and should be in the next version. Jax has already added the SiLU;
jax.nn.swish will eventually be deprecated and jax.nn.silu will be added and both of the aforementioned papers will be cited in the documentation.
It is worth mentioning that PyTorch's SiLU op is an optimized implementation: https://github.com/pytorch/pytorch/pull/42976<|||||>You are correct, we should update and deprecate. Do you want to take a stab at a PR?