Alright I got it !
BTW, there was a mistake is the previous snippet -> I forgot to synchronize the kernels 
Here is the correct snippet (PS: I removed the t2 since it throws an error):
import timeit

setup = """

import torch
import torch.nn.functional as F

device = "cuda"

x = torch.rand(200, 3, 200, device=device)
weight = torch.rand(256, 3, 3, device=device)
eps = 1e-5
eps_t = torch.full((len(weight), ), eps, device=device)

"""

t1 = "F.conv1d(x, weight) + eps;torch.cuda.synchronize()"
t3 = "F.conv1d(x, weight, torch.full((len(weight), ), eps, device=device));torch.cuda.synchronize()"
t4 = "F.conv1d(x, weight, eps_t);torch.cuda.synchronize()"

number = 100

print("%d ms" % round(1000 * timeit.timeit(stmt=t1, setup=setup, number=number)))
print("%d ms" % round(1000 * timeit.timeit(stmt=t3, setup=setup, number=number)))
print("%d ms" % round(1000 * timeit.timeit(stmt=t4, setup=setup, number=number)))

And of course, the results are different ! Now, t3 is always faster than t1 . If you store your eps in a tensor, it can give a slight improvement (about 5%) -> I measured it through t4.