`ModelError` when calling SageMaker Endpoint for prediction using the official notebooks
## Environment info
- `transformers` version: 4.6.1
- Platform: Linux-4.14.232-123.381.amzn1.x86_64-x86_64-with-glibc2.10
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger @patil-suraj 

## Information

I have trained and saved a BertForSequenceClassification model to S3. I then used [this notebook](https://github.com/huggingface/notebooks/blob/master/sagemaker/10_deploy_model_from_s3/deploy_transformer_model_from_s3.ipynb) to deploy the model to SageMaker Endpoints. 

I ran:
```python
from sagemaker.huggingface import HuggingFaceModel
import sagemaker 

role = sagemaker.get_execution_role()

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data="s3://XXXXXXXXXXXX/model.tar.gz",  # path to your trained sagemaker model
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version="py36", # python version of the DLC
   env={'HF_TASK':'text-classification'}
)

predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)
```
But when I use the provided snippet:

```python
# example request, you always need to define "inputs"
data = {
   "inputs": "The new Hugging Face SageMaker DLC makes it super easy to deploy models in production. I love it!"
}

# request
predictor.predict(data)
```

I get the following error:

```bash
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "Can\u0027t load config for \u0027/.sagemaker/mms/models/model\u0027. Make sure that:\n\n- \u0027/.sagemaker/mms/models/model\u0027 is a correct model identifier listed on \u0027https://huggingface.co/models\u0027\n\n- or \u0027/.sagemaker/mms/models/model\u0027 is the correct path to a directory containing a config.json file\n\n"
}
". See https://us-east-2.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-XXXXXXXX in account XXXXXXXXXX for more information.
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

cc @philschmid <|||||>Hey @xegulon, 

How have you created your `model.tar.gz` and what does it contain? It looks like that the file structure of it is wrong and the inference toolkit cannot find the `config.json` and `pytorch_model.bin`. 

You can take a look [here](https://huggingface.co/docs/sagemaker/inference#creating-a-model-artifact-modeltargz-for-deployment) at how to properly create a `model.tar.gz`.
https://huggingface.co/docs/sagemaker/inference#creating-a-model-artifact-modeltargz-for-deployment<|||||>Here are the contents of `model.tar.gz`:

![image](https://user-images.githubusercontent.com/74178038/129331926-90568924-9cae-4566-8102-2cd23c3d239b.png)

I used the `save_pretrained` method on the model and tokenizer to get that.

P. S.: after re-checking, I remarked `transformers` version is `4.9.2`<|||||>How have you created this archive? 

and are you sure the structure is not the one below?
```bash
model.tar.gz
    directory
         pytorch_model.bin
```

Could you try creating the archive with the following steps? 
1. cd and create a tar file
```bash
cd {repository}
tar zcvf model.tar.gz *
```
the repository should be the directory where your artifacts are stored

2. Upload model.tar.gz to s3
```Bash
aws s3 cp model.tar.gz <s3://{my-s3-path}>
```
After that, you can use the S3 uri as model_data.



<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.