Hi @benG,
There were some issues with the python sdk. But it is now merged and should be released tomorrow. You can use the new sagemaker-sdk with pip install git+https://github.com/aws/sagemaker-python-sdk.git or you can wait until tomorrow and install pip install sagemaker and then define transformers_version='4.6' (4.6.1).
huggingface_estimator = HuggingFace(entry_point='train.py',
                            source_dir='./scripts',
                            instance_type='ml.p3.2xlarge',
                            instance_count=1,
                            role=role,
                            transformers_version='4.6',
                            pytorch_version='1.7',
                            py_version='py36',
                            hyperparameters = hyperparameters)