ValueError: Source directory does not exist in the repo. (Looks like the link is broken).
I am getting the error above when I am trying to train a text generation model in the sagemaker.
Please see the script and configuration that I am using:
import sagemaker
from sagemaker.huggingface import HuggingFace
git_config = {‘repo’: ‘https://github.com/huggingface/transformers.git’,'branch’: ‘v4.6.1’} # v4.6.1 is referring to the transformers_version you use in the estimator.

gets role for executing training job
role = sagemaker.get_execution_role()
hyperparameters = {
‘model_name_or_path’:‘ktangri/gpt-neo-demo’,
‘output_dir’:’/opt/ml/model’,
‘fp16’: True,
‘train_file’: ‘/opt/ml/input/data/train/train.csv’,
‘validation_file’: ‘/opt/ml/input/data/validation/validation.csv’
# add your remaining hyperparameters
# more info here https://github.com/huggingface/transformers/tree/v4.6.1/examples/language-modeling
}
#configuration for running training on smdistributed Data Parallel
distribution = {‘smdistributed’:{‘dataparallel’:{ ‘enabled’: True}}}

git configuration to download our fine-tuning script
git_config = {‘repo’: ‘https://github.com/huggingface/transformers.git’,'branch’: ‘v4.6.1’}

creates Hugging Face estimator
huggingface_estimator = HuggingFace(
entry_point=‘run_clm.py’,
source_dir=’/examples/language-modeling’,
instance_type=‘ml.p3.16xlarge’,
instance_count=2,
role=role,
git_config=git_config,
transformers_version=‘4.6.1’,
pytorch_version=‘1.7.1’,
py_version=‘py36’,
hyperparameters = hyperparameters
)
huggingface_estimator.fit(
{‘train’: ‘s3://ch-questions-dataset-east1/train/train.csv’,
‘validation’: ‘s3://ch-questions-dataset-east1/validation/validation.csv’}
)
Thank you.