Update for example 5 with @googlebot explanation. Still far from what I would expect…
with Timer("kernel complete"):
    input_cpu = torch.rand(32, 512, 512).pin_memory()
    parameter_cpu = torch.rand(32, 512, 512, requires_grad=True).pin_memory()
    stddev = torch.ones_like(input_cpu, device="cuda")

    with Timer("kernel launch"):
        for i in tqdm(range(N_ITERATIONS)):
            input_gpu = input_cpu.to("cuda", non_blocking=True)
            parameter_gpu = parameter_cpu.to("cuda", non_blocking=True)

            result = Normal(input_gpu, stddev).log_prob(parameter_gpu)

    torch.cuda.synchronize()

With N_ITERATIONS = 512
DURATION: kernel launch 4.855967344999954
DURATION: kernel complete 7.614574080998864

With N_ITERATIONS = 2048
DURATION: kernel launch 23.592640275999656
DURATION: kernel complete 26.401106097000593