Make activation functions available from modeling_utils (PyTorch)
* This commit replaces references to PyTorch activation functions/modules by a dict of functions that lives in `modeling_utils`. This ensures that all activation functions are available to all modules, praticularly custom functions such as swish and new_gelu.
* In addition, when available (PT1.2) the native PyTorch gelu function will be used - it supports a CPP/CUDA implementation.

**NOTE** that this replaces all `nn.Module`'s by bare functions except for one which was required for testing to be of the type `nn.Module`. If requested, this can be reverted so that only function calls are replaced by ACT2FN functions, and that existing `nn.Module`s are untouched.

**NOTE** that one would thus also expect that _all_ usages of activation functions are taken from `ACT2FN` for consistency's sake.

**NOTE** since the Module counter-part of PyTorch's GeLU [isn't available (yet)](https://github.com/pytorch/pytorch/pull/20665#issuecomment-536359684), it might be worth waiting to implement this pull, and then use Modules and functions in the right places where one would expect, i.e. `Module` when part of architecture, function when processing other kinds of data.
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=h1) Report
> Merging [#1371](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ae50ad91ea2fedb64ecd2e7c8e2d0d4778dc03aa?src=pr&el=desc) will **increase** coverage by `0.97%`.
> The diff coverage is `85.71%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1371/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1371      +/-   ##
==========================================
+ Coverage   83.76%   84.74%   +0.97%     
==========================================
  Files          84       84              
  Lines       12596    12559      -37     
==========================================
+ Hits        10551    10643      +92     
+ Misses       2045     1916     -129
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX29wZW5haS5weQ==) | `80.41% <100%> (ø)` | :arrow_up: |
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `72.02% <100%> (+0.77%)` | :arrow_up: |
| [transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2dwdDIucHk=) | `83.88% <100%> (-0.11%)` | :arrow_down: |
| [transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3JvYmVydGEucHk=) | `71.01% <100%> (+5.54%)` | :arrow_up: |
| [transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2Rpc3RpbGJlcnQucHk=) | `95.8% <100%> (-0.03%)` | :arrow_down: |
| [transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RyYW5zZm9feGwucHk=) | `75.16% <100%> (ø)` | :arrow_up: |
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `88.41% <100%> (+0.23%)` | :arrow_up: |
| [transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbS5weQ==) | `88.36% <100%> (-0.07%)` | :arrow_down: |
| [transformers/modeling\_transfo\_xl\_utilities.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RyYW5zZm9feGxfdXRpbGl0aWVzLnB5) | `54.16% <37.5%> (+0.27%)` | :arrow_up: |
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `92.57% <90%> (-0.12%)` | :arrow_down: |
| ... and [7 more](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=footer). Last update [ae50ad9...716d783](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Unstale. <|||||>Would feel syntactically cleaner if we could do `ACT2FN.gelu()` instead of a dict (also gives some IDE goodness like autocomplete) (I guess through a class or namespace or something), what do you guys think?<|||||>> Would feel syntactically cleaner if we could do `ACT2FN.gelu()` instead of a dict (also gives some IDE goodness like autocomplete) (I guess through a class or namespace or something), what do you guys think?

Sounds good but note that this is not something I introduced. The ACT2FN dict already existed, but wasn't used consistently it seemed.<|||||>Ah yeah, I see. Would you want to do this change, if you have the time/bandwidth? (+ rebasing on current master so we can merge easily?)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>AFAICT, this has been done by @sshleifer on master. Re-open if necessary!