[Performance] Tracking open Issues and PRs (pytorch transformers)
Let's use this Issue to track performance issues and enhancement requests, so it's easier to prioritize the work.

**This is for pytorch `transformers`**

Also I will label it as a `Good Difficult Issue` in case someone is ready for a challenging but rewarding experience of figuring things out. If you do want to take the challenge comment in the corresponding Issue/PR that resonates with you so others would know you're working on it.

If I missed any other relevant open performance-related Issues/PRs that need attention please comment below.

## Regression:
- [ ] https://github.com/huggingface/transformers/pull/11218 Regression after Bart-like refactoring - need to compare the original Bart refactoring PR since most likely the regression happened there.
- [ ]

## Odd slowness:
- [ ] https://github.com/huggingface/transformers/issues/10816 figuring out why eval with --fp16_full_eval is 25% slower
- [ ]

## Fused kernels possibilities:
- [ ] https://github.com/huggingface/transformers/issues/11368 Megatron fused CUDA kernels to improve Hugging Face model classes' scalability 
- [ ] research pytorch kernels?
- [ ] I know Deepspeed has various kernels that we might be able to use

## Faster / leaner startup / module loading
- [ ] https://github.com/huggingface/transformers/issues/12274 - skip storage allocation which gets dropped for pretrained weights

## Faster optimizers
- [ ] https://github.com/huggingface/transformers/issues/12084 - a proposal to port `MemoryEfficientFP16Optimizer` from fairseq
- [ ] https://github.com/huggingface/transformers/issues/9965 - `torch.optim._multi_tensor` faster optimizers - having some bottleneck in the test script - need to profile

## Scalability
- [ ] https://github.com/huggingface/transformers/issues/10321 Tensor Parallelism

## Deepspeed-specific features
- [ ] https://github.com/huggingface/transformers/issues/9606 a list of features that can be integrated
- [ ] https://github.com/huggingface/transformers/issues/12273 - make `from_pretrained` loading faster

## Tests
- [ ] No issue yet, but we really need to add performance regression tests


@stas00  If I want to work on this issue, should I pick one of those issues to keep track of its performance?
Can you also tell me how I can keep track of the performances? Can you give me some guidance?
<|||||>Hi @JuheonChu, this is not an Issue to work on. As the title says this is a collection of pointers to track other Issues. It's dated but many issues that it links to are still valid. So you can click on the issue that resonates with you and discuss the details there - not here. 

I hope this addresses your question.