Thank you for your reply.  In network perimeter training , the gpu0 were fully used, but gpu1,2,3 half but it works.  when network architecture searching, it also create oom. In another words:
epoch < 5 :Data (4,3,224,224) GPU0:12GB GPU1~3 5GB
epoch >=5:  Data(4,3,224,224) and Architect[(20,8),(12,4,3)]
How to do that or relax the gpu0 burden into others?