taniks/blackwell_xformers_flash_attn_win64

Flash Attention Windows Build Guide

Modified from https://huggingface.co/lldacing/flash-attention-windows-wheel to add input for python path for a virtual environment.

Build Configuration

Reset Workers

If you want to reset workers, edit WindowsWhlBuilder_cuda.bat and modify:

set MAX_JOBS=1

Basic Build

WindowsWhlBuilder_cuda.bat

Build with Specific CUDA Architecture (sm80 and sm120)

WindowsWhlBuilder_cuda.bat CUDA_ARCH="80;120"

Enable cxx11abi with Custom Python Path

WindowsWhlBuilder_cuda.bat CUDA_ARCH="80;120" FORCE_CXX11_ABI=TRUE PYTHON_PATH="RootOfVENVHere"

Windows Long Paths Setup

Two requirements for enabling long paths on Windows:

1. Registry Configuration

Run in PowerShell (as Administrator):

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name LongPathsEnabled -Type DWord -Value 1

2. Git Configuration

git config --system core.longpaths true

Post Install

"..venv\Lib\site-packages\xformers\ops\fmha\flash.py"

    FLASH_VERSION = flash_attn.__version__
    FLASH_VER_MIN = (2, 7, 1)
    FLASH_VER_LAST = (2, 8, 1)  # last supported, inclusive

FLASH_VER_LAST to 2,8,1 instead of 2,8,0