regisss

/

llama2-70b-fused-qkv-mlperf

like 1

Problem

When we try to use this module for a fine-tuning project on our cluster, there is an issue with the check for flash_attn.

Workaround

A workaround used by phi-1.5 is to wrap the imports in try-except block. We can implement a similar one here till there is a proper fix from the transformers library( Please refer to the issue above for an expanded context)

Phi-1.5 code:
https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_phi.py#L50