Training a TimeSFormer for video classification 
My input data are the feature maps, instead of raw images. and have the form : (4,50,1,1,256)
mini_batch=4 / frames=50 / channels=1 / H=1 / W= 256
The parameters of the TimeSformer are : 
TimeSformer(
    dim = 128,
    image_size = 256,
    patch_size = 16,
    num_frames = 50,
    num_classes = 2,
    depth = 12,
    heads = 8,
    dim_head =  32,
    attn_dropout = 0.,
    ff_dropout = 0.
) 
In order to check if my network is working, I have tried to make it overfit by using only 6 training data and 2 validation data of the same shape as before (4,50,1,1,256).
But the accuracy I'm getting is in oscillation and never reaches a value >80% and my training loss is not decreasing it's always around 0.6900 - 06950
My training function and parameters are:
![Capture1](https://user-images.githubusercontent.com/24354915/115587902-7c75f500-a2ce-11eb-9a6c-88266f3c233a.PNG)
![Capture2](https://user-images.githubusercontent.com/24354915/115588104-ae875700-a2ce-11eb-81f0-d6e31e177d5d.PNG)
![Capture3](https://user-images.githubusercontent.com/24354915/115588124-b47d3800-a2ce-11eb-82bc-c80e87670bd3.PNG)
I would appreciate any suggestion.
thank you

Hello! Is this a `transformers` issue? Where does `transformers` come into play?<|||||>> 
> 
> Hello! Is this a `transformers` issue? Where does `transformers` come into play?
Hi !! Actually, here I am showing only the issues I'm getting when training the TimeSFormer model,
you can find more information about the model in this link: https://github.com/lucidrains/TimeSformer-pytorch <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.