Text Summarization, Generative Ai Tools, Artificial Intelligence, Machine Learning Ai, Hugging Face.
args.pad_to_max_length: # If we did not pad to max length, we need to pad the labels too labels = accelerator.pad_across_processes(batch["labels"], dim=1, pad_index=tokenizer.pad_token_id) generated_tokens, labels = accelerator.gather((generated_tokens, labels)) generated_tokens =