Solved by nobody in post #6 
              
              
                Thank you for your answer above. I found the reason. The optimizer uses different gradient calculation strategies for one-dimensional parameters, which requires more GPU memory space.