GPU Usage?
**Question**
> Note the query/issue might not have anything to do with the library as such, just looking for info as to why it will happen. Thanks for understanding.

- Why would the GPU show a usage[verified using `nvidia-smi`] of 420MB/32GB when i import `transformers`?

Note this only happens when i have `tensorflow-gpu 2.0` version in the same enviornment, otherwise it just works normally.

Pytorch v1.3.0
Transformers v2.1.1
tf-gpu v2.0.0
apex 0.1
GPU Tesla V100-SXM2-32GB
NVIDIA-SMI 410.79       
Driver Version: 410.79 
CUDA Version: 10.0
Python 3.7.3

Thanks.

(Sorry for a vague title and a quey)


Extra Outputs when i import transformers

```
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
2019-10-12 18:33:00.840558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-10-12 18:33:00.881658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:b5:00.0
2019-10-12 18:33:00.881762: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881816: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881854: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881892: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881931: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881967: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.882003: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.882015: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-10-12 18:33:00.882328: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-10-12 18:33:00.909789: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400000000 Hz
2019-10-12 18:33:00.916559: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b9ec5e0b90 executing computations on platform Host. Devices:
2019-10-12 18:33:00.916618: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-10-12 18:33:01.606656: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b9ec644220 executing computations on platform CUDA. Devices:
2019-10-12 18:33:01.606754: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-10-12 18:33:01.607078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-12 18:33:01.607115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      
>>> 
```

the same here.

installing last apex code from repository: git clone https://github.com/NVIDIA/apex

it says it's apex-0.1 version, but i think it should say apex-1.0<|||||>I run into this problem while trying to create virtual GPU devices:

```python
import tensorflow as tf
import transformers

devices =  tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_virtual_device_configuration(devices[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)]) 
```
that ends up with error:
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-d51e3d242817> in <module>
----> 1 tf.config.experimental.set_virtual_device_configuration(devices[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])

~/miniconda3/envs/transformers/lib/python3.7/site-packages/tensorflow_core/python/framework/config.py in set_virtual_device_configuration(device, virtual_devices)
    554     virtual_devices: (optional) Need to update
    555   """
--> 556   context.context().set_virtual_device_configuration(device, virtual_devices)

~/miniconda3/envs/transformers/lib/python3.7/site-packages/tensorflow_core/python/eager/context.py in set_virtual_device_configuration(self, dev, virtual_devices)
   1269     if self._context_handle is not None:
   1270       raise RuntimeError(
-> 1271           "Virtual devices cannot be modified after being initialized")
   1272 
   1273     self._virtual_device_map[dev] = virtual_devices

RuntimeError: Virtual devices cannot be modified after being initialized
```
versions:
* Platform Linux-5.0.9-050009-generic-x86_64-with-debian-buster-sid
* Python 3.7.5 (default, Oct 25 2019, 15:51:11) 
* [GCC 7.3.0]
* PyTorch 1.2.0
* Tensorflow 2.0.0

The reason is the use of class variable `dummy_inputs` in `transformers/modeling_tf_utils.py:54` where tensorflow is initialized (and starts using GPU) at import time. I created a [PR](https://github.com/huggingface/transformers/pull/1735) that should fix this.<|||||>Thanks for the PR and figuring it where the issue lied.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
