Problem saving and/or loading fine-tuned model
# ❓ Questions & Help
I have finetuned bert-base-cased for multi-class classification with BertForSequenceClassification and got great results (about 88% accuracy on my dev data). However, when I save the finetuned model, load it and run the evaluation on the exact same dev data, I got awful results (about 0.17 accuracy).

At first glance, it seems that either I am wrongly saving the fine-tuned model OR wrongly loading it after training. Would it be possible that save_pretrained only save the weights of the BERT model without the ones of the classifier above ?  @patrickvonplaten 

## Details
Here is how I save the fine-tuned model after training:
`model_to_save = model.module if hasattr(model, "module") else model # Take care of distributed/parallel training`
 `model_to_save.save_pretrained(args.output_dir)`
 `tokenizer.save_pretrained(args.output_dir)`

And here is how I load the fine-tuned model for running evaluation:
`model = BertForSequenceClassification.from_pretrained(
                         args.model_name_or_path, 
                         num_labels = args.num_labels, 
                         output_attentions = False, 
                         output_hidden_states = False, 
                         cache_dir = args.cache_dir)`
where `args.model_name_or_path` is the path of my .bin checkpoint, and args.num_labels stays unchanged during all process.

## Full code

```python
def train(args, model, tokenizer, dataset, tb_writer, categories):

    # Load training dataset.
    if args.do_eval and args.eval_filepath is None:
        print("No validation file given: splitting dataset to train/test datasets...\n")
        train_dataset, validation_dataset = split_data(dataset, args.test_percent, args.seed)
    else:
        train_dataset = dataset

    print("Creating training dataloader...\n")
    train_data, train_sampler, train_dataloader = create_dataloader(train_dataset, args.batch_size, training_data=True)
                                                       
    # Setting up Optimizer & Learning Rate Scheduler.
    optimizer = AdamW(model.parameters(),
                  lr = args.learning_rate,
                  eps = args.adam_epsilon
                )
    total_steps = len(train_dataloader) * args.num_epochs
    scheduler = get_linear_schedule_with_warmup(optimizer, 
                                                num_warmup_steps = 0, # Default value in run_glue.py
                                                num_training_steps = total_steps)
    # Init some useful variables.
    global_step = 0
    tr_loss, logging_loss = 0.0, 0.0

    # For each epoch...
    t = time.time()
    for epoch_i in range(0, args.num_epochs):
        
        # Perform one full pass over the training set.
        print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, args.num_epochs))
        print('Training...')

        # Measure how long the training epoch takes.
        t0 = time.time()

        # Put the model into training mode. Don't be mislead--the call to 
        # `train` just changes the *mode*, it doesn't *perform* the training.
        # `dropout` and `batchnorm` layers behave differently during training
        # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
        model.train()

        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):

            # Unpack this training batch from our dataloader. 
            # As we unpack the batch, we'll also copy each tensor to the GPU using the `to` method.
            # `batch` contains three pytorch tensors:
            #   [0]: input ids 
            #   [1]: attention masks
            #   [2]: labels 
            b_input_ids = batch[0].to(args.device)
            b_input_mask = batch[1].to(args.device)
            b_labels = batch[2].to(args.device)

            # Always clear any previously calculated gradients before performing a backward pass. 
            # PyTorch doesn't do this automatically because accumulating the gradients is "convenient while training RNNs". 
            # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
            model.zero_grad()        

            # Perform a forward pass (evaluate the model on this training batch).
            # This will return the loss (rather than the model output) because we have provided the `labels`.
            # The documentation for this `model` function is here: 
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            outputs = model(b_input_ids, 
                        token_type_ids=None, 
                        attention_mask=b_input_mask, 
                        labels=b_labels)

            # The call to `model` always returns a tuple, so we need to pull the loss value out of the tuple.
            loss = outputs[0]
            
            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel training

            # Accumulate the training loss over all of the batches so that we can calculate the average loss at the end. 
            # `loss` is a Tensor containing a single value; the `.item()` function just returns the Python value from the tensor.
            tr_loss += loss.item()

            # Perform a backward pass to calculate the gradients.
            loss.backward()

            # Clip the norm of the gradients to 1.0. This is to help prevent the "exploding gradients" problem.
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and take a step using the computed gradient.
            # The optimizer dictates the "update rule"--how the parameters are modified based on their gradients, the learning rate, etc.
            optimizer.step()

            # Update the learning rate.
            scheduler.step()
            
            # Update global step.
            global_step += 1
            
            # Progress update every 'logging_steps' batches.
            if args.logging_steps > 0 and step != 0 and step % args.logging_steps == 0:
                
                # Calculate elapsed time in minutes.
                elapsed = format_time(time.time() - t0)
                
                # Compute average training loss over the last 'logging_steps'. Write it to Tensorboard.
                loss_scalar = (tr_loss - logging_loss) / args.logging_steps
                tb_writer.add_scalar('Train/Loss', loss_scalar, global_step)
                logging_loss = tr_loss
                
                # Print the log.
                print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.    Training loss: {:.2f}'.format(step, len(train_dataloader), elapsed, loss_scalar))

        print("  Training epoch took: {:}\n".format(format_time(time.time() - t0)))
        
        if args.do_eval and args.eval_filepath is None:
            print("Running Validation...")
            # After the completion of each training epoch, measure our performance on our validation set.
            t0 = time.time()
            result, df_wrong, df_right = evaluate(args, model, validation_dataset, categories)
            
            # Write results to tensorboard.
            tb_writer.add_scalar('Test/Accuracy', result[0], epoch_i + 1)
            tb_writer.add_scalar('Test/Recall', result[1], epoch_i + 1)
            tb_writer.add_scalar('Test/Precision', result[2], epoch_i + 1)
            tb_writer.add_scalar('Test/F1 score', result[3], epoch_i + 1)
            tb_writer.add_scalar('Test/MCC', result[4], epoch_i + 1)
            
            # Plot confusion matrix.
            plot_confusion_matrix(result[5], categories, args.output_dir)
            
            # Save dataframes of wrong and right predictions for further analysis.
            df_wrong.to_csv(os.path.join(args.output_dir, 'preds_wrong.csv'))
            df_right.to_csv(os.path.join(args.output_dir, 'preds_right.csv'))
            
            print("  Validation took: {:}\n".format(format_time(time.time() - t0)))
            
    print("Training complete!  Took: {}\n".format(format_time(time.time() - t)))
        
    print("Saving model to {}...\n.".format(args.output_dir))
    model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training
    model_to_save.save_pretrained(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)
    torch.save(args, os.path.join(args.output_dir, 'training_args.bin'))  # Good practice: save your training arguments together with the trained model
    return


def evaluate(args, model, validation_dataset, categories):

    # Creating validation dataloader.
    validation_data, validation_sampler, validation_dataloader = create_dataloader(validation_dataset, args.batch_size, training_data=False)
    
    # Get validation sentences.
    validation_sentences = validation_dataset[3]
    
    # Tracking variables
    nb_eval_steps = 0
    preds = None
    out_label_ids = None
    
    # Put the model in evaluation mode--the dropout layers behave differently during evaluation.
    model.eval()

    # Evaluate data for one epoch
    for batch in validation_dataloader:

        # Add batch to GPU.
        b_input_ids, b_input_mask, b_labels = tuple(t.to(args.device) for t in batch)

        # Telling the model not to compute or store gradients, saving memory and speeding up validation
        with torch.no_grad():        

            # Forward pass, calculate logit predictions.
            # This will return the logits rather than the loss because we have not provided labels.
            # token_type_ids is the same as the "segment ids", which differentiates sentence 1 and 2 in 2-sentence tasks.
            outputs = model(b_input_ids, 
                            token_type_ids=None, 
                            attention_mask=b_input_mask)

        # Get the "logits" output by the model. The "logits" are the output values prior to applying an activation function like the softmax.
        logits = outputs[0]

        # Move logits and labels to CPU and store them.
        if preds is None:
            preds = logits.detach().cpu().numpy()
            out_label_ids = b_labels.detach().cpu().numpy()
        else:
            preds = np.append(preds, logits.detach().cpu().numpy(), axis=0)
            out_label_ids = np.append(out_label_ids, b_labels.detach().cpu().numpy(), axis=0)
                
        # Track the number of batches
        nb_eval_steps += 1
            
    # Take the max predicitions.
    preds = np.argmax(preds, axis=1)
    
    # Report results.
    result = compute_metrics(preds, out_label_ids, categories)
    print("  * Accuracy: {0:.4f}".format(result[0]))
    print("  * Recall: {0:.4f}".format(result[1]))
    print("  * Precision: {0:.4f}".format(result[2]))
    print("  * F1 score: {0:.4f}".format(result[3]))
    print("  * MCC: {0:.4f}".format(result[4]))

    # Get wrong and right predictions.
    df_wrong, df_right = analyze_predictions(preds, out_label_ids, validation_sentences)
    
    return result, df_wrong, df_right

def main(args):

    # Create tensorboard summarywriter.
    tb_writer = SummaryWriter()
    
    # Create output dir if none mentioned.
    if args.output_dir is None:
        model_name = os.path.splitext(os.path.basename(args.model_name_or_path))[0]
        args.output_dir = "./output/" + model_name + '/'
    if not os.path.exists(args.output_dir):
        os.makedirs(args.output_dir)
        
    # Set the seed value all over the place to make this reproducible.
    set_seed(args.seed)
    
    print("\n========================================")
    print('               Load model                 ')
    print("========================================\n")
    print("Loading BertForSequenceClassification model...\n")
    model = BertForSequenceClassification.from_pretrained(
        args.model_name_or_path, # Use the 12-layer BERT model, with an cased vocab.
        num_labels = args.num_labels, # The number of output labels
        output_attentions = False, # Whether the model returns attentions weights.
        output_hidden_states = False, # Whether the model returns all hidden-states.
        cache_dir = args.cache_dir,
    )
    #model = BertForSequenceClassification.from_pretrained(args.model_name_or_path)
    print('Loading BertTokenizer...\n')
    tokenizer = BertTokenizer.from_pretrained(args.model_name_or_path, do_lower_case=False)
    
    print("Setting up CUDA & GPU...")
    if torch.cuda.is_available():
        if args.gpu_id:
            torch.cuda.set_device(args.gpu_id)
            args.n_gpu = 1
            print("-> GPU training available! As '--gpu_id' was set, only GPU {} {} will be used (no parallel training).\n".format(torch.cuda.get_device_name(args.gpu_id), args.gpu_id))
        else:
            args.n_gpu = torch.cuda.device_count()
            gpu_ids = list(range(0, args.n_gpu))
            if args.n_gpu > 1:
                model = torch.nn.DataParallel(model, device_ids=gpu_ids, output_device=gpu_ids[-1])
            print("-> GPU training available! Training will use GPU(s) {}\n".format(gpu_ids))
        args.device = torch.device("cuda")
    else:
        args.device = torch.device("cpu")
        args.n_gpu = 0
        print("-> No GPU available, using the CPU instead.\n")
    model.to(args.device)  # Tell pytorch to run the model on the device.
    
    
    print("\n========================================")
    print('               Processing data            ')
    print("========================================\n")
    df, categories = load_data(args)
        
    print("Tokenizing sentences...")
    tokenized = tokenize_sentences(tokenizer, df)
    attention_masks = create_masks(tokenized)
    
    dataset = (tokenized, df.Class_id.values, attention_masks, df.Sentence.values)
    if args.do_train:
        print("\n========================================")
        print('            Launching training            ')
        print("========================================\n")
        train(args, model, tokenizer, dataset, tb_writer, categories)
    
    elif args.do_eval and args.eval_filepath is not None:
        print("\n========================================")
        print('            Launching validation          ')
        print("========================================\n")
        result, df_wrong, df_right = evaluate(args, model, dataset, categories)
        
        # Save dataframes of wrong and right predictions for further analysis.
        df_wrong.to_csv(os.path.join(args.output_dir, 'wrong_preds.csv'))
        df_right.to_csv(os.path.join(args.output_dir, 'right_preds.csv'))
```
  
Hi @antoilouis,

Thanks for posting this. 
A quick wild guess might be that after training you save the model in `args.output_dir` but then load it from a different `args.model_name_or_path` (also since you said that if you evaluate the model right after training you get good results. 

My advice to solve this wolud be the following: 
Train for 1 epoch and a tiny part of the dataset. Print out / Save some model weights, then save load it again and check whether the weights are equal. They should be equal. <|||||>Oh and also please post your environment information here. Make sure that you have the newest version of `transformers`. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I meet the same question . do you have any advice now? Thanks!<|||||>I am currently facing the same issue with the `BertForTokenClassification` model, the model behaves differently each time it is loaded. Are there solutions?