bug in movement-pruning
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.2 
- Platform: linux
- Python version: 3.8 
- PyTorch version (GPU?): 1.7 
- Tensorflow version (GPU?): - 
- Using GPU in script?: - 
- Using distributed or parallel set-up in script?: - 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@VictorSanh, @JetRunner

## Information

Model I am running the `masked_run_glue.py` on RTE datasets (I see similar results also on WNLI). I added one line to subsample the dataset for faster iteration of debugging as below:
```
 if args.max_train_samples is not None:
            sampled_indices = list(range(len(train_dataset)))
            random.shuffle(sampled_indices)
            sampled_indices = sampled_indices[:args.max_train_samples]
            train_dataset = torch.utils.data.Subset(train_dataset, sampled_indices)
```

Here is the command I run:
```
python masked_run_glue.py   --task_name rte   --do_train   --do_eval   --max_seq_length 128   --num_train_epochs 3   --output_dir test  --do_lower_case   --model_type masked_bert     --model_name_or_path bert-base-uncased     --warmup_steps 500     --learning_rate 3e-5 --mask_scores_learning_rate 1e-2     --initial_threshold 1 --final_threshold 0.15     --initial_warmup 1 --final_warmup 2     --pruning_method topK --mask_init constant --mask_scale 0.    --data_dir  data/RTE/    --overwrite_output_dir    --per_gpu_train_batch_size 64  --max_train_samples 100 
```

No matter how many examples I set, the results are always the `exact` the same as below:
```
09/21/2021 18:27:44 - INFO - __main__ - ***** Eval results  *****
09/21/2021 18:27:44 - INFO - __main__ -   acc = 0.5270758122743683
09/21/2021 18:27:44 - INFO - __main__ -   eval_avg_entropy = 1.8548108
```

To me, there must be a bug causing it. 
Thanks for your help in advance @VictorSanh  

## Expected behavior
The code needs to return different results for different train_datasets given.
I believe you are getting a random performance (RTE is a binary classification dataset) which means the loss has diverged during the training. Could you increase *significantly* the number of steps you are training for? You can take inspiration from these [hyper-parameters](https://docs.google.com/spreadsheets/d/17JgRq_OFFTniUrz6BZWW_87DjFkKXpI1kYDSsseT_7g/edit?usp=sharing).
In my experience, if you prune too fast (like you are doing if you don't have enough steps of fine-pruning), the loss will likely diverge at some point.
<|||||>Hi @VictorSanh 
Thank you for coming back to me, I confirm I ran the experiment for num_steps = 2000 and this is getting the same results and equal to the one posted above for both num_samples=100 and num_samples=1000. I greatly appreciate your help and any suggestions on this. thanks <|||||>Hi @VictorSanh 
I run the method for a very large number of steps (30K steps), and still sampling for N=100 and N=1000 both gives the exact same results posted above. I think there might be a bug causing it, I appreciate a lot your help. 

- Also for the commands given in the repo, you have set initial_threshold=0  final_threshold=0.1 for soft-pruning, but this is 1 and 0.15 for topK one, basically the final one is less than the start for TopK while this is reversed for soft-version. I am confused how to set the threshold value, could you give me some intuition how one can do it? Maybe this helps to solve the issue thanks <|||||>> Hi @VictorSanh
> Thank you for coming back to me, I confirm I ran the experiment for num_steps = 2000 and this is getting the same results and equal to the one posted above for both num_samples=100 and num_samples=1000. I greatly appreciate your help and any suggestions on this. thanks

Could you look at the tensorboard and more particularly the training loss? and then report back<|||||>> * Also for the commands given in the repo, you have set initial_threshold=0  final_threshold=0.1 for soft-pruning, but this is 1 and 0.15 for topK one, basically the final one is less than the start for TopK while this is reversed for soft-version. I am confused how to set the threshold value, could you give me some intuition how one can do it? Maybe this helps to solve the issue thanks

For topK, the threshold value is the percentage of remaining weights (100% at the beginning, 15% at the end).
For the soft version, the threshold corresponds to the \lambda_{mvp} described in the paper.<|||||>Dear @VictorSanh 
Thank you for the advice, I will try with it, do you mind also giving me some advice on how one can set initial warmup and final_warmup? as for the threshold values, although this does not make sense perhaps, but I see soft pruning gives a better results with setting initial threshold to 1. I see the final results in the doc you shared, could you also kindly share the range of values you tried, this is hard for me to understand how to set warmup and threshold for different methods. I appreciate any input on this. 
thanks <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.