[LXMERT] Unclear what img_tensorize does with color spaces
## Environment info

- `transformers` version: Not using transformers directly, I'm loading a model "unc-nlp/frcnn-vg-finetuned"
- Platform: MacOS
- Python version: 3.8
- PyTorch version (GPU?): 1.6.0, no GPU
- Tensorflow version (GPU?): don't have
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@eltoto1219 probably

Models: "LXMERT": "unc-nlp/frcnn-vg-finetuned"

Library: https://github.com/huggingface/transformers/tree/master/examples/research_projects/lxmert

## Information

Model I am using (Bert, XLNet ...):  "LXMERT": "unc-nlp/frcnn-vg-finetuned"

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## Problem

I don't get what img_tensorize in utils.py is doing with color spaces. I run the following code to load the model. 

```
# load models and model components
frcnn_cfg = Config.from_pretrained("unc-nlp/frcnn-vg-finetuned")

frcnn = GeneralizedRCNN.from_pretrained("unc-nlp/frcnn-vg-finetuned", config=frcnn_cfg)

image_preprocess = Preprocess(frcnn_cfg)
```
Turns out that frcnn_cfg.input.format is "BGR" so I wanted to know what exactly is going on? Here is where the image is loaded (utils.img_tensorize)

```
def img_tensorize(im, input_format="RGB"):
    assert isinstance(im, str)
    if os.path.isfile(im):
        img = cv2.imread(im)
    else:
        img = get_image_from_url(im)
        assert img is not None, f"could not connect to: {im}"
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    if input_format == "RGB":
        img = img[:, :, ::-1]
    return img
```

See, we seem to be converting the images to RGB, then if it's "RGB" format we flip the blue (?) channel? Is the image ever converted to "BGR"? 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.