Thanks for the reply.
I found the function “cudaDeviceSynchronize();” in “cuda_runtime.h” (libs: -lcudart).
When I add it after the forward function, I get the following results:

20:46:37: Debugging starts

 Loop nbr.:  0
T_tensorDeclaration:  76694 ns
T_forward:  4744557 ns

 Loop nbr.:  1
T_tensorDeclaration:  23076 ns
T_forward:  4170827 ns

 Loop nbr.:  2
T_tensorDeclaration:  17030 ns
T_forward:  4189053 ns

 Loop nbr.:  3
T_tensorDeclaration:  15676 ns
T_forward:  4167333 ns

 Loop nbr.:  4
T_tensorDeclaration:  16317 ns
T_forward:  4172503 ns

 Loop nbr.:  5
T_tensorDeclaration:  15665 ns
T_forward:  4190116 ns

 Loop nbr.:  6
T_tensorDeclaration:  15595 ns
T_forward:  4163890 ns

 Loop nbr.:  7
T_tensorDeclaration:  15042 ns
T_forward:  4166625 ns

 Loop nbr.:  8
T_tensorDeclaration:  15177 ns
T_forward:  4180314 ns

 Loop nbr.:  9
T_tensorDeclaration:  15237 ns
T_forward:  4167773 ns

 Loop nbr.:  10
T_tensorDeclaration:  14767 ns
T_forward:  4189780 ns

 Loop nbr.:  11
T_tensorDeclaration:  15009 ns
T_forward:  4188130 ns

 Loop nbr.:  12
T_tensorDeclaration:  14792 ns
T_forward:  4169861 ns

 Loop nbr.:  13
T_tensorDeclaration:  14547 ns
T_forward:  4186025 ns

 Loop nbr.:  14
T_tensorDeclaration:  14807 ns
T_forward:  4198113 ns

 Loop nbr.:  15
T_tensorDeclaration:  14582 ns
T_forward:  4192522 ns

 Loop nbr.:  16
T_tensorDeclaration:  14601 ns
T_forward:  4183756 ns

 Loop nbr.:  17
T_tensorDeclaration:  14733 ns
T_forward:  4204688 ns

 Loop nbr.:  18
T_tensorDeclaration:  14592 ns
T_forward:  4167611 ns

 Loop nbr.:  19
T_tensorDeclaration:  14630 ns
T_forward:  4174918 ns

 Loop nbr.:  20
T_tensorDeclaration:  14659 ns
T_forward:  4156959 ns

 Loop nbr.:  21
T_tensorDeclaration:  14422 ns
T_forward:  4178084 ns

 Loop nbr.:  22
T_tensorDeclaration:  14625 ns
T_forward:  4192556 ns

 Loop nbr.:  23
T_tensorDeclaration:  14531 ns
T_forward:  4175173 ns

 Loop nbr.:  24
T_tensorDeclaration:  14580 ns
T_forward:  4166659 ns

 Loop nbr.:  25
T_tensorDeclaration:  14485 ns
T_forward:  4177613 ns

 Loop nbr.:  26
T_tensorDeclaration:  14479 ns
T_forward:  4179809 ns

 Loop nbr.:  27
T_tensorDeclaration:  14505 ns
T_forward:  4178439 ns

 Loop nbr.:  28
T_tensorDeclaration:  14496 ns
T_forward:  4184789 ns

 Loop nbr.:  29
T_tensorDeclaration:  14618 ns
T_forward:  4177106 ns

 Loop nbr.:  30
T_tensorDeclaration:  14547 ns
T_forward:  4182303 ns

 Loop nbr.:  31
T_tensorDeclaration:  14641 ns
T_forward:  4173239 ns

 Loop nbr.:  32
T_tensorDeclaration:  19249 ns
T_forward:  4176845 ns

 Loop nbr.:  33
T_tensorDeclaration:  14245 ns
T_forward:  4128455 ns

 Loop nbr.:  34
T_tensorDeclaration:  14577 ns
T_forward:  4151471 ns

 Loop nbr.:  35
T_tensorDeclaration:  14191 ns
T_forward:  4157208 ns

 Loop nbr.:  36
T_tensorDeclaration:  14156 ns
T_forward:  4157328 ns

 Loop nbr.:  37
T_tensorDeclaration:  14135 ns
T_forward:  4164124 ns

 Loop nbr.:  38
T_tensorDeclaration:  14024 ns
T_forward:  4149202 ns

 Loop nbr.:  39
T_tensorDeclaration:  14068 ns
T_forward:  4148399 ns

 Loop nbr.:  40
T_tensorDeclaration:  14095 ns
T_forward:  4149947 ns

 Loop nbr.:  41
T_tensorDeclaration:  14166 ns
T_forward:  4134731 ns

 Loop nbr.:  42
T_tensorDeclaration:  13995 ns
T_forward:  4130745 ns

 Loop nbr.:  43
T_tensorDeclaration:  14047 ns
T_forward:  4157489 ns

 Loop nbr.:  44
T_tensorDeclaration:  17812 ns
T_forward:  4145059 ns

 Loop nbr.:  45
T_tensorDeclaration:  14840 ns
T_forward:  4145376 ns

 Loop nbr.:  46
T_tensorDeclaration:  14524 ns
T_forward:  4144078 ns

 Loop nbr.:  47
T_tensorDeclaration:  14292 ns
T_forward:  4156597 ns

 Loop nbr.:  48
T_tensorDeclaration:  14604 ns
T_forward:  4130630 ns

 Loop nbr.:  49
T_tensorDeclaration:  14345 ns
T_forward:  4156479 ns

 Loop nbr.:  50
T_tensorDeclaration:  14376 ns
T_forward:  4137532 ns

 Loop nbr.:  51
T_tensorDeclaration:  14162 ns
T_forward:  4138844 ns

 Loop nbr.:  52
T_tensorDeclaration:  14233 ns
T_forward:  4138394 ns

 Loop nbr.:  53
T_tensorDeclaration:  14245 ns
T_forward:  4160527 ns

 Loop nbr.:  54
T_tensorDeclaration:  16532 ns
T_forward:  4154455 ns

 Loop nbr.:  55
T_tensorDeclaration:  15135 ns
T_forward:  4151513 ns

 Loop nbr.:  56
T_tensorDeclaration:  14605 ns
T_forward:  4159401 ns

 Loop nbr.:  57
T_tensorDeclaration:  15114 ns
T_forward:  4146790 ns

 Loop nbr.:  58
T_tensorDeclaration:  15189 ns
T_forward:  4163448 ns

 Loop nbr.:  59
T_tensorDeclaration:  14600 ns
T_forward:  4154543 ns

 Loop nbr.:  60
T_tensorDeclaration:  14391 ns
T_forward:  4152786 ns

 Loop nbr.:  61
T_tensorDeclaration:  14449 ns
T_forward:  4139992 ns

 Loop nbr.:  62
T_tensorDeclaration:  14358 ns
T_forward:  4136887 ns

 Loop nbr.:  63
T_tensorDeclaration:  14809 ns
T_forward:  4138758 ns

 Loop nbr.:  64
T_tensorDeclaration:  14358 ns
T_forward:  4128357 ns

 Loop nbr.:  65
T_tensorDeclaration:  14318 ns
T_forward:  4138889 ns

 Loop nbr.:  66
T_tensorDeclaration:  14135 ns
T_forward:  4138168 ns

 Loop nbr.:  67
T_tensorDeclaration:  14288 ns
T_forward:  4158496 ns

 Loop nbr.:  68
T_tensorDeclaration:  14304 ns
T_forward:  4144945 ns

 Loop nbr.:  69
T_tensorDeclaration:  14286 ns
T_forward:  4137468 ns

 Loop nbr.:  70
T_tensorDeclaration:  14182 ns
T_forward:  4163196 ns

 Loop nbr.:  71
T_tensorDeclaration:  15629 ns
T_forward:  4154694 ns

 Loop nbr.:  72
T_tensorDeclaration:  14497 ns
T_forward:  4119437 ns

 Loop nbr.:  73
T_tensorDeclaration:  14230 ns
T_forward:  4150738 ns

 Loop nbr.:  74
T_tensorDeclaration:  14308 ns
T_forward:  4382309 ns

 Loop nbr.:  75
T_tensorDeclaration:  15805 ns
T_forward:  4121531 ns

 Loop nbr.:  76
T_tensorDeclaration:  15074 ns
T_forward:  4116742 ns

 Loop nbr.:  77
T_tensorDeclaration:  14275 ns
T_forward:  4126637 ns

 Loop nbr.:  78
T_tensorDeclaration:  14278 ns
T_forward:  4127533 ns

 Loop nbr.:  79
T_tensorDeclaration:  14324 ns
T_forward:  4128542 ns

 Loop nbr.:  80
T_tensorDeclaration:  14327 ns
T_forward:  4118499 ns

 Loop nbr.:  81
T_tensorDeclaration:  14270 ns
T_forward:  4138547 ns

 Loop nbr.:  82
T_tensorDeclaration:  14196 ns
T_forward:  4130813 ns

 Loop nbr.:  83
T_tensorDeclaration:  18730 ns
T_forward:  4100290 ns

 Loop nbr.:  84
T_tensorDeclaration:  14288 ns
T_forward:  4127821 ns

 Loop nbr.:  85
T_tensorDeclaration:  14240 ns
T_forward:  4113148 ns

 Loop nbr.:  86
T_tensorDeclaration:  14323 ns
T_forward:  4118545 ns

 Loop nbr.:  87
T_tensorDeclaration:  14405 ns
T_forward:  4117402 ns

 Loop nbr.:  88
T_tensorDeclaration:  14367 ns
T_forward:  4108369 ns

 Loop nbr.:  89
T_tensorDeclaration:  14503 ns
T_forward:  4139753 ns

 Loop nbr.:  90
T_tensorDeclaration:  14343 ns
T_forward:  4128485 ns

 Loop nbr.:  91
T_tensorDeclaration:  14376 ns
T_forward:  4128973 ns

 Loop nbr.:  92
T_tensorDeclaration:  14353 ns
T_forward:  4126810 ns

 Loop nbr.:  93
T_tensorDeclaration:  14201 ns
T_forward:  4123998 ns

 Loop nbr.:  94
T_tensorDeclaration:  15228 ns
T_forward:  4122794 ns

 Loop nbr.:  95
T_tensorDeclaration:  14555 ns
T_forward:  4106747 ns

 Loop nbr.:  96
T_tensorDeclaration:  14585 ns
T_forward:  4130666 ns

 Loop nbr.:  97
T_tensorDeclaration:  14393 ns
T_forward:  4126877 ns

 Loop nbr.:  98
T_tensorDeclaration:  14084 ns
T_forward:  4134650 ns

 Loop nbr.:  99
T_tensorDeclaration:  14464 ns
T_forward:  4138943 ns

 Loop nbr.:  100
T_tensorDeclaration:  14557 ns
T_forward:  4120151 ns
20:46:44: Debugging has finished

Does it means that the real forward time of the network is around 4ms ? It seems quite slow for a small network like this, isn’t it ?
When the data are feed by batch in the training, it’s going much faster.