Has anyone been able to get Deepspeed working for inference with GPT-Neo, on a finetuned model?
As per this GitHub issue:
  

      github.com/Xirider/finetune-gpt2xl
  

  
    
  
	  
  

  
    
      Gpt-neo inference with Deepspeed: IndexError: Dimension out of range 14
    

    
      
        opened 
        
          
        
        Aug 10, 2021
      
      


      
        
          
          kingpalethe
        
      
    

    
    
  


  
    Thanks for this useful repository. I was able to follow it to train a gtp-neo 2.…7B model. 

Inference on the model works well for me, using less than 8GB of Vram, so fits on consumer-level gpus, however, I'm not yet able to get the inference working with Deepspeed. 

To be clear... 

I am using the code from here:

https://github.com/Xirider/finetune-gpt2xl/blob/main/README.md#generate-text-with-a-gpt-neo-27-billion-parameters-model

And it works well, if I comment out this line:

`deepspeed.init_inference(model, mp_size=1, dtype=torch.half, replace_method='auto')`

If I retain the line, then the inference fails with this error message:

```
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 374, in forward
    output = DeepSpeedSelfAttentionFunction.apply(
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 312, in forward
    output, key_layer, value_layer, context_layer = selfAttention_fp()
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 270, in selfAttention_fp
    qkv_out = qkv_func(input,
IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)
```

I'm actually a bit vague on whether Deepspeed actually should be used with inference for GTP-NEO, as far.

Huggingface says....

https://huggingface.co/transformers/main_classes/deepspeed.html

> DeepSpeed ZeRO-2 is primarily used only for training, as its features are of no use to inference.

But, Microsoft has a guide which shows the usage of Deepspeed for inference with this model...

https://github.com/microsoft/DeepSpeed/blob/master/docs/_tutorials/inference-tutorial.md#end-to-end-gpt-neo-27b-inference
  

  

  
    
    
  

  


…I am finding that inference with Deepspeed works well on the un-finetuned model, “EleutherAI/gpt-neo-2.7B”
… But after I fine tune the model, inference with Deepspeed fails with this error message:
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 374, in forward
    output = DeepSpeedSelfAttentionFunction.apply(
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 312, in forward
    output, key_layer, value_layer, context_layer = selfAttention_fp()
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 270, in selfAttention_fp
    qkv_out = qkv_func(input,
IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)