Remote code is loaded from `main` even when revision is provided
### System Info

When specifying a branch to load a model with remote code as follows fails because there is no modeling file on `main`. Is this a bug or the expected behaviour? 

### Who can help?

The one and only _**@sgugger**_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
model = transformers.AutoModelForCausalLM.from_pretrained("bigcode/santacoder-fast-inference", revision="main_custom", trust_remote_code=True)
```

The following error shows that the code file is attempted to be loaded from `main` instead of `main_custom` (where a modeling file is present:

```bash
Could not locate the configuration_gpt_bigcode.py inside bigcode/santacoder-fast-inference.
Traceback (most recent call last):
  File "/work/arjunguha-research-group/arjun/venvs/bigcode/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 264, in hf_raise_for_status
    response.raise_for_status()
  File "/shared/centos7/python/3.8.1/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/bigcode/santacoder-fast-inference/resolve/main/configuration_gpt_bigcode.py
```


### Expected behavior

Loading without error.
Will have a look even if you didn't properly tag me :-p <|||||>The PR linked above fixes the config problem (I can load it with `AutoConfig`). The model still won't load however as the auto mpa of that config doesn't contain an entry for `AutoModelForCausalLM`.