LLAMA for sequence classification
### System Info

@ArthurZucker and @younesbelkada

I am trying to perform sequence classification for text using LLAMA 7B model leveraging LORA training. I have 2 classes.
Tokeniser and models are loading fine. But loss is zero after the first batch; when I check the logits, of model outputs, they are nan. 

```
tokenizer = LlamaTokenizer.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
if getattr(tokenizer, "pad_token_id") is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id
print("Loading LLAMA model...")
model = LlamaForSequenceClassification.from_pretrained(
        PATH_TO_CONVERTED_WEIGHTS,
        num_labels=2,
        output_attentions=False,
        output_hidden_states=False,
        # torch_dtype=getattr(torch, 'float16'),
        load_in_8bit=True,
        low_cpu_mem_usage=True,  
    )
peft_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    target_modules=target_modules,
    inference_mode=False,
    r=lora_rank, lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    modules_to_save=mod_to_save)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

device = torch.device('cuda')
model.cuda()
```
This is my training loop.

```
for e in range(epochs):
        train_loss = 0
        train_acc = 0

        model.train()

        optim = optimizer(model.parameters(), lr=lr, eps=1e-8)

        scheduler = get_linear_schedule_with_warmup(
            optim,
            num_warmup_steps=0,
            num_training_steps=len(train_dataloader) * epochs
        )

        for batch in train_dataloader:
            b_input_ids = batch[0].to(device)
            b_input_mask = batch[1].to(device)
            b_labels = batch[2].to(device)
            model.zero_grad()

            output = model(
                b_input_ids,
                # token_type_ids=None,
                attention_mask=b_input_mask,
                labels=b_labels
            )

            loss = output['loss']
            if torch.isnan(loss):
                print("!!!! Loss is NaN")
                break
            preds = output['logits'].detach().cpu().numpy()
            labels = b_labels.to('cpu').numpy()
            print(f"Loss :{loss.item()}")
            train_loss += loss.item()
            train_acc += accuracy(preds, labels)
            loss.backward()

            torch.nn.utils.clip_grad_value_(model.parameters(), 5.0)

            optim.step()
            scheduler.step()
        avg_train_loss = train_loss / len(train_dataloader)
        avg_train_acc = train_acc / len(train_dataloader)

        print('average training loss for epoch: {}'.format(avg_train_loss))
        print('average training accuracy for epoch: {}'.format(avg_train_acc))
```
I am getting ‘NaN’ loss after the first batch.

Experiments tried (but did not work):

- Tried clip grad value and clip grad norm (values from 1.0 to 5.0)
  ```torch.nn.utils.clip_grad_value_(model.parameters(), 5.0)```
- Tried changing lr too
- loading in 8bit and float16

Any help would be greatly appreciated. Thanks

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Starting seq classification training using the above code.

### Expected behavior

I see the loss calculated for the first batch only. From the next batch the logits become NaN and in turn loss and everything else is NaN
hi @Lathashree01 
thanks for reporting, can you try to load the model in `bfloat16`, also what is the GPU hardware you are using? <|||||>Hi @younesbelkada , 
I am using Quadro RTX 6000 node with 8 GPUs of 24GB memory.

When I run using - bfloat16, I am getting below error:
`TypeError: Got unsupported ScalarType BFloat16`
    @  preds = output['logits'].detach().cpu().numpy()
<|||||>Can you replace the lines that causing that error to:
```python
            preds = output['logits'].detach().cpu().float().numpy()
            labels = b_labels.to('cpu').float().numpy()
```
from what I can they are used to compute the accuracy only so it should be fine
Also can you share your bitsandbytes version?<|||||>Hi @younesbelkada ,

I tried removing the above lines and ran with bfloat16; I see loss values normally, hope everything works as expected. 
Thank you so much. I was stuck in this and was trying out so many other things. 

Also, I changed the accuracy calculation to:
```
            preds = output['logits'].detach().cpu().to(torch.float16)
            labels = b_labels.to('cpu').numpy()
```

My bitsandBytes version is  - bitsandbytes             0.39.0
However, I do see some errors when I run `python -m bitsandbytes`. 

UserWarning: /home/anaconda3/envs/finetuneenv did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searchingfurther paths...
..
things related to posixpath
....
raise RuntimeError('Something when wrong when trying to find file. Maybe you do not have a linux system?')
RuntimeError: Something when wrong when trying to find file. Maybe you do not have a linux system?<|||||>Thanks ! Does training works with PEFT + int8 as showed in the script you shared? i.e. do you get that error only if you do `python -m bitsandbytes` ?
Also, there should be no need to call `model.cuda()` after you have quantized the model<|||||>No, the training does not work fine with int8; logits and loss goes to NaN.

Above mentioned error about bitsandbytes appears when I start training (even in bfloat16)  and when I do `python -m bitsandbytes`.

Since I am not using any components on bnb and I am also not loading the model in int8, I ignored the error while training with bfloat16. Is that fine? Please let me know what I can do if that's a problem.

<|||||>I see now thanks ! 
for the bnb issue probably your CUDA + bnb installation might be broken, you can post the issue on the bitsandbytes repository by stating what hardware and operating system you are using
Regarding your hotfix, which is to fine-tune in bf16 I think it is fine to do so, bfloat16 training is recommended over float16 training<|||||>> Regarding your hotfix, which is to fine-tune in bf16 I think it is fine to do so, bfloat16 training is recommended over float16 training

Oh, that's relieving. 

Thank you, I will check on bitsandbytes error.<|||||>I have the same errors.<|||||>已经收到您的邮件。<|||||>> I have the same errors.

Mine got resolved when trained with bf16. 
If you could please elaborate on where your error is or maybe post your error trace. It will be helpful for the team or others to provide any suggestions.
