Hi All I am getting the below error msg when trying to train Bert , any help will be great. Bit urgent.
error:-
"Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.
Error for Training job huggingface-sdk-extension-2022-01-24-16-47-13-971: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command “/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32”
LOG:-
2022-01-24 16:53:12,880 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
2022-01-24 16:53:12,904 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2022-01-24 16:53:15,927 sagemaker_pytorch_container.training INFO     Invoking user training script.
2022-01-24 16:53:16,395 sagemaker-training-toolkit INFO     Invoking user script
Training Env:
{
“additional_framework_parameters”: {},
“channel_input_dirs”: {
“test”: “/opt/ml/input/data/test”,
“train”: “/opt/ml/input/data/train”
},
“current_host”: “algo-1”,
“framework_module”: “sagemaker_pytorch_container.training:main”,
“hosts”: [
“algo-1”
],
“hyperparameters”: {
“train_batch_size”: 32,
“model_name”: “distilbert-base-uncased”,
“epochs”: 2
},
“input_config_dir”: “/opt/ml/input/config”,
“input_data_config”: {
“test”: {
“TrainingInputMode”: “File”,
“S3DistributionType”: “FullyReplicated”,
“RecordWrapperType”: “None”
},
“train”: {
“TrainingInputMode”: “File”,
“S3DistributionType”: “FullyReplicated”,
“RecordWrapperType”: “None”
}
},
“input_dir”: “/opt/ml/input”,
“is_master”: true,
“job_name”: “huggingface-sdk-extension-2022-01-24-16-47-13-971”,
“log_level”: 20,
“master_hostname”: “algo-1”,
“model_dir”: “/opt/ml/model”,
“module_dir”: “s3://sagemaker-eu-west-2-352316401451/huggingface-sdk-extension-2022-01-24-16-47-13-971/source/sourcedir.tar.gz”,
“module_name”: “train”,
“network_interface_name”: “eth0”,
“num_cpus”: 8,
“num_gpus”: 1,
“output_data_dir”: “/opt/ml/output/data”,
“output_dir”: “/opt/ml/output”,
“output_intermediate_dir”: “/opt/ml/output/intermediate”,
“resource_config”: {
“current_host”: “algo-1”,
“hosts”: [
“algo-1”
],
“network_interface_name”: “eth0”
},
“user_entry_point”: “train.py”
}
Environment variables:
SM_HOSTS=[“algo-1”]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={“epochs”:2,“model_name”:“distilbert-base-uncased”,“train_batch_size”:32}
SM_USER_ENTRY_POINT=train.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={“current_host”:“algo-1”,“hosts”:[“algo-1”],“network_interface_name”:“eth0”}
SM_INPUT_DATA_CONFIG={“test”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”},“train”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”}}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=[“test”,“train”]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=train
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=8
SM_NUM_GPUS=1
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-eu-west-2-352316401451/huggingface-sdk-extension-2022-01-24-16-47-13-971/source/sourcedir.tar.gz
SM_TRAINING_ENV={“additional_framework_parameters”:{},“channel_input_dirs”:{“test”:"/opt/ml/input/data/test",“train”:"/opt/ml/input/data/train"},“current_host”:“algo-1”,“framework_module”:“sagemaker_pytorch_container.training:main”,“hosts”:[“algo-1”],“hyperparameters”:{“epochs”:2,“model_name”:“distilbert-base-uncased”,“train_batch_size”:32},“input_config_dir”:"/opt/ml/input/config",“input_data_config”:{“test”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”},“train”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”}},“input_dir”:"/opt/ml/input",“is_master”:true,“job_name”:“huggingface-sdk-extension-2022-01-24-16-47-13-971”,“log_level”:20,“master_hostname”:“algo-1”,“model_dir”:"/opt/ml/model",“module_dir”:“s3://sagemaker-eu-west-2-352316401451/huggingface-sdk-extension-2022-01-24-16-47-13-971/source/sourcedir.tar.gz”,“module_name”:“train”,“network_interface_name”:“eth0”,“num_cpus”:8,“num_gpus”:1,“output_data_dir”:"/opt/ml/output/data",“output_dir”:"/opt/ml/output",“output_intermediate_dir”:"/opt/ml/output/intermediate",“resource_config”:{“current_host”:“algo-1”,“hosts”:[“algo-1”],“network_interface_name”:“eth0”},“user_entry_point”:“train.py”}
SM_USER_ARGS=["–epochs",“2”,"–model_name",“distilbert-base-uncased”,"–train_batch_size",“32”]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_TEST=/opt/ml/input/data/test
SM_CHANNEL_TRAIN=/opt/ml/input/data/train
SM_HP_TRAIN_BATCH_SIZE=32
SM_HP_MODEL_NAME=distilbert-base-uncased
SM_HP_EPOCHS=2
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages
Invoking script with the following command:
/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32
2022-01-24 16:53:21,122 - main - INFO -  loaded train_dataset length is: 572
2022-01-24 16:53:21,122 - main - INFO -  loaded test_dataset length is: 144
2022-01-24 16:53:21,457 - filelock - INFO - Lock 140311318218288 acquired on /root/.cache/huggingface/transformers/23454919702d26495337f3da04d1655c7ee010d5ec9d77bdb9e399e00302c0a1.91b885ab15d631bf9cee9dc9d25ece0afd932f2f5130eba28f2055b2220c0333.lock
2022-01-24 16:53:21,790 - filelock - INFO - Lock 140311318218288 released on /root/.cache/huggingface/transformers/23454919702d26495337f3da04d1655c7ee010d5ec9d77bdb9e399e00302c0a1.91b885ab15d631bf9cee9dc9d25ece0afd932f2f5130eba28f2055b2220c0333.lock
2022-01-24 16:53:22,163 - filelock - INFO - Lock 140311212156688 acquired on /root/.cache/huggingface/transformers/9c169103d7e5a73936dd2b627e42851bec0831212b677c637033ee4bce9ab5ee.126183e36667471617ae2f0835fab707baa54b731f991507ebbb55ea85adb12a.lock
2022-01-24 16:53:27,475 - filelock - INFO - Lock 140311212156688 released on /root/.cache/huggingface/transformers/9c169103d7e5a73936dd2b627e42851bec0831212b677c637033ee4bce9ab5ee.126183e36667471617ae2f0835fab707baa54b731f991507ebbb55ea85adb12a.lock
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: [‘vocab_layer_norm.bias’, ‘vocab_projector.weight’, ‘vocab_projector.bias’, ‘vocab_transform.bias’, ‘vocab_layer_norm.weight’, ‘vocab_transform.weight’]

This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: [‘classifier.weight’, ‘pre_classifier.weight’, ‘pre_classifier.bias’, ‘classifier.bias’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2022-01-24 16:53:28,847 - filelock - INFO - Lock 140311146924352 acquired on /root/.cache/huggingface/transformers/0e1bbfda7f63a99bb52e3915dcf10c3c92122b827d92eb2d34ce94ee79ba486c.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock
2022-01-24 16:53:29,484 - filelock - INFO - Lock 140311146924352 released on /root/.cache/huggingface/transformers/0e1bbfda7f63a99bb52e3915dcf10c3c92122b827d92eb2d34ce94ee79ba486c.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock
2022-01-24 16:53:29,811 - filelock - INFO - Lock 140311211720320 acquired on /root/.cache/huggingface/transformers/75abb59d7a06f4f640158a9bfcde005264e59e8d566781ab1415b139d2e4c603.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4.lock
2022-01-24 16:53:30,528 - filelock - INFO - Lock 140311211720320 released on /root/.cache/huggingface/transformers/75abb59d7a06f4f640158a9bfcde005264e59e8d566781ab1415b139d2e4c603.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4.lock
2022-01-24 16:53:31,518 - filelock - INFO - Lock 140311211719928 acquired on /root/.cache/huggingface/transformers/8c8624b8ac8aa99c60c912161f8332de003484428c47906d7ff7eb7f73eecdbb.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79.lock

2022-01-24 16:53:38 Uploading - Uploading generated training model2022-01-24 16:53:31,850 - filelock - INFO - Lock 140311211719928 released on /root/.cache/huggingface/transformers/8c8624b8ac8aa99c60c912161f8332de003484428c47906d7ff7eb7f73eecdbb.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79.lock
[2022-01-24 16:53:36.715 algo-1:26 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2022-01-24 16:53:36.867 algo-1:26 INFO profiler_config_parser.py:102] User has disabled profiler.
[2022-01-24 16:53:36.868 algo-1:26 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.
[2022-01-24 16:53:36.869 algo-1:26 INFO hook.py:201] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.
[2022-01-24 16:53:36.870 algo-1:26 INFO hook.py:255] Saving to /opt/ml/output/tensors
[2022-01-24 16:53:36.871 algo-1:26 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.
#015Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]#015Downloading: 100%|██████████| 483/483 [00:00<00:00, 460kB/s]
#015Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]#015Downloading:   2%|▏         | 4.08M/268M [00:00<00:06, 40.8MB/s]#015Downloading:   3%|▎         | 8.71M/268M [00:00<00:06, 42.3MB/s]#015Downloading:   5%|▌         | 13.5M/268M [00:00<00:05, 43.8MB/s]#015Downloading:   7%|▋         | 18.4M/268M [00:00<00:05, 45.2MB/s]#015Downloading:   9%|▊         | 23.3M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  10%|█         | 28.0M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  12%|█▏        | 32.6M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  14%|█▍        | 37.7M/268M [00:00<00:04, 47.5MB/s]#015Downloading:  16%|█▌        | 42.7M/268M [00:00<00:04, 48.4MB/s]#015Downloading:  18%|█▊        | 47.8M/268M [00:01<00:04, 49.1MB/s]#015Downloading:  20%|█▉        | 52.8M/268M [00:01<00:04, 49.2MB/s]#015Downloading:  22%|██▏       | 57.8M/268M [00:01<00:04, 49.6MB/s]#015Downloading:  23%|██▎       | 62.9M/268M [00:01<00:04, 50.0MB/s]#015Downloading:  25%|██▌       | 68.0M/268M [00:01<00:03, 50.3MB/s]#015Downloading:  27%|██▋       | 73.2M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  29%|██▉       | 78.2M/268M [00:01<00:03, 50.5MB/s]#015Downloading:  31%|███       | 83.3M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  33%|███▎      | 88.5M/268M [00:01<00:03, 50.9MB/s]#015Downloading:  35%|███▍      | 93.6M/268M [00:01<00:03, 51.1MB/s]#015Downloading:  37%|███▋      | 98.8M/268M [00:02<00:03, 51.3MB/s]#015Downloading:  39%|███▉      | 104M/268M [00:02<00:03, 50.5MB/s] #015Downloading:  41%|████      | 109M/268M [00:02<00:03, 50.5MB/s]#015Downloading:  43%|████▎     | 114M/268M [00:02<00:03, 50.7MB/s]#015Downloading:  44%|████▍     | 119M/268M [00:02<00:02, 50.8MB/s]#015Downloading:  46%|████▋     | 124M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  48%|████▊     | 130M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  50%|█████     | 135M/268M [00:02<00:02, 51.2MB/s]#015Downloading:  52%|█████▏    | 140M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  54%|█████▍    | 145M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  56%|█████▌    | 150M/268M [00:03<00:02, 51.4MB/s]#015Downloading:  58%|█████▊    | 155M/268M [00:03<00:02, 50.9MB/s]#015Downloading:  60%|█████▉    | 160M/268M [00:03<00:02, 51.0MB/s]#015Downloading:  62%|██████▏   | 166M/268M [00:03<00:02, 51.2MB/s]#015Downloading:  64%|██████▎   | 171M/268M [00:03<00:01, 51.5MB/s]#015Downloading:  66%|██████▌   | 176M/268M [00:03<00:01, 52.1MB/s]#015Downloading:  68%|██████▊   | 181M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  70%|██████▉   | 187M/268M [00:03<00:01, 52.9MB/s]#015Downloading:  72%|███████▏  | 192M/268M [00:03<00:01, 52.0MB/s]#015Downloading:  74%|███████▎  | 198M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  76%|███████▌  | 203M/268M [00:04<00:01, 52.9MB/s]#015Downloading:  78%|███████▊  | 208M/268M [00:04<00:01, 49.9MB/s]#015Downloading:  80%|███████▉  | 213M/268M [00:04<00:01, 50.8MB/s]#015Downloading:  82%|████████▏ | 219M/268M [00:04<00:00, 51.4MB/s]#015Downloading:  84%|████████▎ | 224M/268M [00:04<00:00, 52.0MB/s]#015Downloading:  86%|████████▌ | 229M/268M [00:04<00:00, 52.4MB/s]#015Downloading:  88%|████████▊ | 235M/268M [00:04<00:00, 52.7MB/s]#015Downloading:  90%|████████▉ | 240M/268M [00:04<00:00, 52.8MB/s]#015Downloading:  92%|█████████▏| 245M/268M [00:04<00:00, 52.9MB/s]#015Downloading:  94%|█████████▎| 251M/268M [00:04<00:00, 53.0MB/s]#015Downloading:  96%|█████████▌| 256M/268M [00:05<00:00, 53.1MB/s]#015Downloading:  98%|█████████▊| 261M/268M [00:05<00:00, 52.7MB/s]#015Downloading: 100%|█████████▉| 267M/268M [00:05<00:00, 51.6MB/s]#015Downloading: 100%|██████████| 268M/268M [00:05<00:00, 50.9MB/s]
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: [‘vocab_layer_norm.bias’, ‘vocab_projector.weight’, ‘vocab_projector.bias’, ‘vocab_transform.bias’, ‘vocab_layer_norm.weight’, ‘vocab_transform.weight’]

This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: [‘classifier.weight’, ‘pre_classifier.weight’, ‘pre_classifier.bias’, ‘classifier.bias’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
#015Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]#015Downloading:  16%|█▌        | 36.9k/232k [00:00<00:00, 242kB/s]#015Downloading:  82%|████████▏ | 190k/232k [00:00<00:00, 321kB/s] #015Downloading: 100%|██████████| 232k/232k [00:00<00:00, 755kB/s]
#015Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]#015Downloading:  20%|█▉        | 92.2k/466k [00:00<00:00, 610kB/s]#015Downloading:  73%|███████▎  | 338k/466k [00:00<00:00, 750kB/s] #015Downloading: 100%|██████████| 466k/466k [00:00<00:00, 1.52MB/s]
#015Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]#015Downloading: 100%|██████████| 28.0/28.0 [00:00<00:00, 24.3kB/s]
#015  0%|          | 0/36 [00:00<?, ?it/s]Traceback (most recent call last):
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 699, in convert_to_tensors
tensor = as_tensor(value)
ValueError: too many dimensions ‘str’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “train.py”, line 83, in 
trainer.train()
File “/opt/conda/lib/python3.6/site-packages/transformers/trainer.py”, line 1246, in train
for step, inputs in enumerate(epoch_iterator):
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 444, in next
(data, worker_id) = self._next_data()
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 526, in _next_data
2022-01-24 16:53:37,616 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
Command “/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32”
#015Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]#015Downloading: 100%|ââââââââââ| 483/483 [00:00<00:00, 460kB/s]
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
#015Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]#015Downloading:   2%|â         | 4.08M/268M [00:00<00:06, 40.8MB/s]#015Downloading:   3%|â         | 8.71M/268M [00:00<00:06, 42.3MB/s]#015Downloading:   5%|â         | 13.5M/268M [00:00<00:05, 43.8MB/s]#015Downloading:   7%|â         | 18.4M/268M [00:00<00:05, 45.2MB/s]#015Downloading:   9%|â         | 23.3M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  10%|â         | 28.0M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  12%|ââ        | 32.6M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  14%|ââ        | 37.7M/268M [00:00<00:04, 47.5MB/s]#015Downloading:  16%|ââ        | 42.7M/268M [00:00<00:04, 48.4MB/s]#015Downloading:  18%|ââ        | 47.8M/268M [00:01<00:04, 49.1MB/s]#015Downloading:  20%|ââ        | 52.8M/268M [00:01<00:04, 49.2MB/s]#015Downloading:  22%|âââ       | 57.8M/268M [00:01<00:04, 49.6MB/s]#015Downloading:  23%|âââ       | 62.9M/268M [00:01<00:04, 50.0MB/s]#015Downloading:  25%|âââ       | 68.0M/268M [00:01<00:03, 50.3MB/s]#015Downloading:  27%|âââ       | 73.2M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  29%|âââ       | 78.2M/268M [00:01<00:03, 50.5MB/s]#015Downloading:  31%|âââ       | 83.3M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  33%|ââââ      | 88.5M/268M [00:01<00:03, 50.9MB/s]#015Downloading:  35%|ââââ      | 93.6M/268M [00:01<00:03, 51.1MB/s]#015Downloading:  37%|ââââ      | 98.8M/268M [00:02<00:03, 51.3MB/s]#015Downloading:  39%|ââââ      | 104M/268M [00:02<00:03, 50.5MB/s] #015Downloading:  41%|ââââ      | 109M/268M [00:02<00:03, 50.5MB/s]#015Downloading:  43%|âââââ     | 114M/268M [00:02<00:03, 50.7MB/s]#015Downloading:  44%|âââââ     | 119M/268M [00:02<00:02, 50.8MB/s]#015Downloading:  46%|âââââ     | 124M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  48%|âââââ     | 130M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  50%|âââââ     | 135M/268M [00:02<00:02, 51.2MB/s]#015Downloading:  52%|ââââââ    | 140M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  54%|ââââââ    | 145M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  56%|ââââââ    | 150M/268M [00:03<00:02, 51.4MB/s]#015Downloading:  58%|ââââââ    | 155M/268M [00:03<00:02, 50.9MB/s]#015Downloading:  60%|ââââââ    | 160M/268M [00:03<00:02, 51.0MB/s]#015Downloading:  62%|âââââââ   | 166M/268M [00:03<00:02, 51.2MB/s]#015Downloading:  64%|âââââââ   | 171M/268M [00:03<00:01, 51.5MB/s]#015Downloading:  66%|âââââââ   | 176M/268M [00:03<00:01, 52.1MB/s]#015Downloading:  68%|âââââââ   | 181M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  70%|âââââââ   | 187M/268M [00:03<00:01, 52.9MB/s]#015Downloading:  72%|ââââââââ  | 192M/268M [00:03<00:01, 52.0MB/s]#015Downloading:  74%|ââââââââ  | 198M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  76%|ââââââââ  | 203M/268M [00:04<00:01, 52.9MB/s]#015Downloading:  78%|ââââââââ  | 208M/268M [00:04<00:01, 49.9MB/s]#015Downloading:  80%|ââââââââ  | 213M/268M [00:04<00:01, 50.8MB/s]#015Downloading:  82%|âââââââââ | 219M/268M [00:04<00:00, 51.4MB/s]#015Downloading:  84%|âââââââââ | 224M/268M [00:04<00:00, 52.0MB/s]#015Downloading:  86%|âââââââââ | 229M/268M [00:04<00:00, 52.4MB/s]#015Downloading:  88%|âââââââââ | 235M/268M [00:04<00:00, 52.7MB/s]#015Downloading:  90%|âââââââââ | 240M/268M [00:04<00:00, 52.8MB/s]#015Downloading:  92%|ââââââââââ| 245M/268M [00:04<00:00, 52.9MB/s]#015Downloading:  94%|ââââââââââ| 251M/268M [00:04<00:00, 53.0MB/s]#015Downloading:  96%|ââââââââââ| 256M/268M [00:05<00:00, 53.1MB/s]#015Downloading:  98%|ââââââââââ| 261M/268M [00:05<00:00, 52.7MB/s]#015Downloading: 100%|ââââââââââ| 267M/268M [00:05<00:00, 51.6MB/s]#015Downloading: 100%|ââââââââââ| 268M/268M [00:05<00:00, 50.9MB/s]
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: [‘vocab_layer_norm.bias’, ‘vocab_projector.weight’, ‘vocab_projector.bias’, ‘vocab_transform.bias’, ‘vocab_layer_norm.weight’, ‘vocab_transform.weight’]
This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: [‘classifier.weight’, ‘pre_classifier.weight’, ‘pre_classifier.bias’, ‘classifier.bias’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
#015Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]#015Downloading:  16%|ââ        | 36.9k/232k [00:00<00:00, 242kB/s]#015Downloading:  82%|âââââââââ | 190k/232k [00:00<00:00, 321kB/s] #015Downloading: 100%|ââââââââââ| 232k/232k [00:00<00:00, 755kB/s]
#015Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]#015Downloading:  20%|ââ        | 92.2k/466k [00:00<00:00, 610kB/s]#015Downloading:  73%|ââââââââ  | 338k/466k [00:00<00:00, 750kB/s] #015Downloading: 100%|ââââââââââ| 466k/466k [00:00<00:00, 1.52MB/s]
#015Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]#015Downloading: 100%|ââââââââââ| 28.0/28.0 [00:00<00:00, 24.3kB/s]
#015  0%|          | 0/36 [00:00<?, ?it/s]Traceback (most recent call last):
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 699, in convert_to_tensors
tensor = as_tensor(value)
ValueError: too many dimensions ‘str’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “train.py”, line 83, in 
trainer.train()
File “/opt/conda/lib/python3.6/site-packages/transformers/trainer.py”, line 1246, in train
for step, inputs in enumerate(epoch_iterator):
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 444, in next
(data, worker_id) = self._next_data()
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 526, in _next_data
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch
return self.collate_fn(data)
File “/opt/conda/lib/python3.6/site-packages/transformers/data/data_collator.py”, line 123, in call
return_tensors=“pt”,
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 2680, in pad
return BatchEncoding(batch_outputs, tensor_type=return_tensors)
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 204, in init
self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch
return self.collate_fn(data)
File “/opt/conda/lib/python3.6/site-packages/transformers/data/data_collator.py”, line 123, in call
return_tensors=“pt”,
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 2680, in pad
return BatchEncoding(batch_outputs, tensor_type=return_tensors)
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 204, in init
self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 716, in convert_to_tensors
"Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.
#015  0%|          | 0/36 [00:00<?, ?it/s]
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 716, in convert_to_tensors
"Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.
#015  0%|          | 0/36 [00:00<?, ?it/s]


2022-01-24 16:54:35 Failed - Training job failed
ProfilerReport-1643042834: Stopping
UnexpectedStatusException                 Traceback (most recent call last)
 in 
10
11 # starting the train job with our uploaded datasets as input
—> 12 huggingface_estimator.fit({‘train’: training_input_path, ‘test’: test_input_path})
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
690         self.jobs.append(self.latest_training_job)
691         if wait:
 → 692             self.latest_training_job.wait(logs=logs)
693
694     def _compilation_job_name(self):
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
1665         # If logs are requested, call logs_for_jobs.
1666         if logs != “None”:
 → 1667             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
1668         else:
1669             self.sagemaker_session.wait_for_job(self.job_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
3783
3784         if wait:
 → 3785             self._check_job_status(job_name, description, “TrainingJobStatus”)
3786             if dot:
3787                 print()
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
3341                 ),
3342                 allowed_statuses=[“Completed”, “Stopped”],
 → 3343                 actual_status=status,
3344             )
3345
UnexpectedStatusException: Error for Training job huggingface-sdk-extension-2022-01-24-16-47-13-971: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command “/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32”
Downloading: 100%|ââââââââââ| 483/483 [00:00<00:00, 460kB/s]
Downloading:  18%|ââ        | 47.8M/268M [00:01<00:04, 49.1MB/s]