i try torch.cuda.empty_cache() function to resolve my problem,
before status:
1-25: 0.4s
25-50: 2-3s
after add torch.cuda.empty_cache() after 1-25 computing done
1-25: 0.4s
torch.cuda.empty_cache()  2-3s
25-50: 0.4s
even forward time is faster, but empty_cache cost too much time 
i guess maybe 1-25 has cover all memory of gpu, so without empty_cache(), gpu will check and free memory for new forward. if i use empty_cache, gpu free all memory. my purpose is make it more faster. anyone know how to resolve it?