Thanks for your reply!
I modify the code and use the time.time() function to record the time
the code is as follows:
    last_layer=list(net.children())[-1]
    for param in net.parameters():
        param.requires_grad = False 
    for param in last_layer.parameters():
        param.requires_grad = True

    torch.cuda.synchronize()
    start_time=time.time()
    for i in range(len(inputs)):
        optimizer.zero_grad()
        loss[i].backward(retain_graph=True)
        last_grad_norm=0.0
        for name, para in last_layer.named_parameters():
            if 'weight' in name:
                last_grad_norm += para.grad.norm().cpu().item() 
        # print("last grad norm", last_grad_norm) 
        result.append(last_grad_norm)
    torch.cuda.synchronize()
    print("the time for computing the gradient for the last layer:",time.time()-start_time)

    for param in net.parameters():
        param.requires_grad = True
    torch.cuda.synchronize()
    start_time=time.time()
    for i in range(len(inputs)):
        optimizer.zero_grad()
        loss[i].backward(retain_graph=True)
        all_grad_norm=0.0
        for name, para in last_layer.named_parameters():
            if 'weight' in name:
                all_grad_norm += para.grad.norm().cpu().item() 
        result.append(all_grad_norm)
    torch.cuda.synchronize()   
    print("the time for computing the gradient for all layers:",time.time()-start_time)

the result is as follows:
image1618×320 153 KB
The cost time is almost similar.
Besides, because I am a new learner of Pytorch, I don’t know how to compute the gradient of loss about the input of the last layer, which include the softmax,bn and the last layer- conv. Could you give me an example or reference. Thanks so much!