I think you forgot to add the env task when creating your HuggingFaceModel:
huggingface_model = HuggingFaceModel(
   model_data="s3://xxxxxxxxxxxxxxx/model.tar.gz",  # path to your trained SageMaker model
   role=role,                                            # IAM role with permissions to create an endpoint
   transformers_version="4.6",                           # Transformers version used
   pytorch_version="1.7",                                # PyTorch version used
   py_version='py36',                                    # Python version used
   env={ 'HF_TASK':'conversational' },
)