text: Title: Using PEFT causes model to not predict EOS
  Body: ### System Info
  
  > peft version: 0.9.0
  > accelerate version: 0.27.2
  > transformers version: 4.37.0
  > trl version: 0.7.12.dev0
  > base model: openai-community/gpt2
  > hardware: 2xA100
  
  I'm doing a LORA peft of GPT2 through [trl](https://github.com/huggingface/trl) and have noticed that my trained model assigns very low probability to the EOS token which causes it to alway generate the maximum number of tokens.
  
  After trying a few different fixes I ran the code without the PEFT option and just used the base model. The problem resolved immediately. 
  
  To make the comparison clear I created a toy case with a dataset that contains the same datapoint ("Hello <|endoftext|>") repeatedly. I then overfit on this dataset with a small batch size for a few dozen iterations. To see the effect on the probability of generating the eos_token I inserted the following code fragment in my `compute_metrics` method:
  
  >   logits, labels = eval_preds
  >   eos_indices = np.where(labels==tokenizer.eos_token_id)
  >   model_distribution = torch.softmax(torch.tensor(logits),dim=-1).numpy()
  >   eos_probs = model_distribution[eos_indices[0],eos_indices[1],-1]
  >   eos_probs = [format(x*100,'.3f') for x in eos_probs.tolist()]
  >   print('eos probs:',eos_probs)
  
  The basic full finetuning results in the EOS token probability converging to 1 almost immediately as the model memorizes the location of the EOS tokens. However if I just use TRL's code for a LORA PEFT the printed values remain close to zero and don't increase at all.
  
  I've seen some references online suggesting that this could be caused by LORA not updating the model's embedding matrix. So I added the following change to the peft_config: `peft_config.modules_to_save = ["wte"]`. This doesn't have any effect on the results. I'm also doubtful this is the cause as when I run the supervised finetuning I don't see any change in the embedding matrix but get the desired results anyway.
  
  Any help would be appreciated as I would like to avoid a full finetuning but right now have no way of getting a functional model with a PEFT.
  
  ### Who can help?
  
  _No response_
  
  ### Information
  
  - [ ] The official example scripts
  - [X] My own modified scripts
  
  ### Tasks
  
  - [ ] An officially supported task in the `examples` folder
  - [X] My own task or dataset (give details below)
  
  ### Reproduction
  
  Use the following model_config (note the PEFT parameters) and training arguments:
  
  > ModelConfig(model_name_or_path='openai-community/gpt2', model_revision='main', torch_dtype=None, trust_remote_code=False, attn_implementation=None, use_peft=True, lora_r=64, lora_alpha=16, lora_dropout=0.05, lora_target_modules=None, lora_modules_to_save=None, load_in_8bit=False, load_in_4bit=False, bnb_4bit_quant_type='nf4', use_bnb_nested_quant=False)
  
  > TrainingArguments(
  > _n_gpu=1,
  > adafactor=False,
  > adam_beta1=0.9,
  > adam_beta2=0.999,
  > adam_epsilon=1e-08,
  > auto_find_batch_size=False,
  > bf16=False,
  > bf16_full_eval=False,
  > data_seed=None,
  > dataloader_drop_last=False,
  > dataloader_num_workers=0,
  > dataloader_persistent_workers=False,
  > dataloader_pin_memory=True,
  > ddp_backend=None,
  > ddp_broadcast_buffers=None,
  > ddp_bucket_cap_mb=None,
  > ddp_find_unused_parameters=None,
  > ddp_timeout=1800,
  > debug=[],
  > deepspeed=None,
  > disable_tqdm=False,
  > dispatch_batches=None,
  > do_eval=True,
  > do_predict=False,
  > do_train=False,
  > eval_accumulation_steps=1,
  > eval_delay=0,
  > eval_steps=10,
  > evaluation_strategy=steps,
  > fp16=False,
  > fp16_backend=auto,
  > fp16_full_eval=False,
  > fp16_opt_level=O1,
  > fsdp=[],
  > fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
  > fsdp_min_num_params=0,
  > fsdp_transformer_layer_cls_to_wrap=None,
  > full_determinism=False,
  > gradient_accumulation_steps=1,
  > gradient_checkpointing=True,
  > gradient_checkpointing_kwargs={'use_reentrant': False},
  > greater_is_better=None,
  > group_by_length=False,
  > half_precision_backend=auto,
  > hub_always_push=False,
  > hub_model_id=None,
  > hub_private_repo=False,
  > hub_strategy=every_save,
  > hub_token=<HUB_TOKEN>,
  > ignore_data_skip=False,
  > include_inputs_for_metrics=False,
  > include_num_input_tokens_seen=False,
  > include_tokens_per_second=False,
  > jit_mode_eval=False,
  > label_names=None,
  > label_smoothing_factor=0.0,
  > learning_rate=1.41e-05,
  > length_column_name=length,
  > load_best_model_at_end=False,
  > local_rank=0,
  > log_level=passive,
  > log_level_replica=warning,
  > log_on_each_node=True,
  > logging_dir=/scratch/km3888/gcode_peft/_/runs/Apr23_14-18-54_gh004.hpc.nyu.edu,
  > logging_first_step=False,
  > logging_nan_inf_filter=True,
  > logging_steps=1.0,
  > logging_strategy=steps,
  > lr_scheduler_kwargs={},
  > lr_scheduler_type=linear,
  > max_grad_norm=1.0,
  > max_steps=20000,
  > metric_for_best_model=None,
  > mp_parameters=,
  > neftune_noise_alpha=None,
  > no_cuda=False,
  > num_train_epochs=1.0,
  > optim=adamw_torch,
  > optim_args=None,
  > output_dir=/scratch/km3888/gcode_peft/_,
  > overwrite_output_dir=False,
  > past_index=-1,
  > per_device_eval_batch_size=4,
  > per_device_train_batch_size=4,
  > prediction_loss_only=False,
  > push_to_hub=True,
  > push_to_hub_model_id=None,
  > push_to_hub_organization=None,
  > push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
  > ray_scope=last,
  > remove_unused_columns=True,
  > report_to=['wandb'],
  > resume_from_checkpoint=None,
  > run_name=/scratch/km3888/gcode_peft/_,
  > save_on_each_node=False,
  > save_only_model=False,
  > save_safetensors=True,
  > save_steps=500,
  > save_strategy=steps,
  > save_total_limit=None,
  > seed=42,
  > skip_memory_metrics=True,
  > split_batches=False,
  > tf32=None,
  > torch_compile=False,
  > torch_compile_backend=None,
  > torch_compile_mode=None,
  > torchdynamo=None,
  > tpu_metrics_debug=False,
  > tpu_num_cores=None,
  > use_cpu=False,
  > use_ipex=False,
  > use_legacy_prediction_loop=False,
  > use_mps_device=False,
  > warmup_ratio=0.0,
  > warmup_steps=0,
  > weight_decay=0.0,
  > )
  > 
  
  Create dataset:
  
  > import copy
  > dummy_data = [{"text":"Hello <|endoftext|>"} for _ in range(1000)]
  > with open("dummy_data.json","w") as f:
  >     json.dump(dummy_data,f)
  > full_dataset = load_dataset('json', data_files="dummy_data.json",split='train')
  > full_dataset = full_dataset.map(lambda x: {'text':add_eos(x['text'])})
  > split_data = full_dataset.train_test_split(test_size=0.05)
  > train_dataset = split_data['train'].shuffle()
  > eval_dataset = copy.deepocpy(train_dataset)
  > tokenizer = AutoTokenizer.from_pretrained(model_config.model_name_or_path, token=access_token,use_fast=True, add_eos=True)
  
  Set up custom evaluation function:
  
  > def compute_metrics(eval_preds):
  >     metric = evaluate.load("accuracy",training_args.output_dir.split('/')[-1])
  >     logits, labels = eval_preds
  >     eos_indices = np.where(labels==tokenizer.eos_token_id)
  >     model_distribution = torch.softmax(torch.tensor(logits),dim=-1).numpy()
  >     eos_probs = model_distribution[eos_indices[0],eos_indices[1],-1]
  >     eos_probs = [format(x*100,'.3f') for x in eos_probs.tolist()]
  >     print('eos probs:',eos_probs)
  >     predictions = np.argmax(logits,axis=-1)
  >     predictions = np.reshape(predictions.astype(np.int32),-1)
  >     labels = np.reshape(labels.astype(np.int32),-1)
  >     return metric.compute(predictions=predictions, references=labels)
  
  Instantiate and run [SFTTrainer](https://github.com/huggingface/trl/blob/main/trl/trainer/sft_trainer.py)
  > trainer = SFTTrainer(
  >     model=model_config.model_name_or_path,
  >     model_init_kwargs=model_kwargs,
  >     args=training_args,
  >     train_dataset=train_dataset,
  >     eval_dataset=eval_dataset,
  >     dataset_text_field="text",
  >     max_seq_length=args.max_seq_length,
  >     tokenizer=tokenizer,
  >     packing=False,
  >     peft_config=get_peft_config(model_config),
  >     compute_metrics=compute_metrics,
  >     dataset_num_proc=20)
  > 
  >     trainer.train()
  
  The  `eos_probs` printed in `compute_metrics` will be near-zero
  
  ### Expected behavior
  
  I would expect the above code to result in `eos_probs` values being nearly 1 after a few training iterations.