daiwx2002:

x=torch.randn(1000,1000)
y=x**2
Can the operation of **2 speed up by GPU?


Yes. But if you just want to perform a single or simple operations on a GPU, you will lose all the speed advantage via transfer speeds.
In [1]: import torch                                                            

In [2]: x = torch.randn(1000, 1000)                                             

In [3]: %timeit x**2                                                            
7.33 ms ± 1.46 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [4]: x_gpu = x.to(torch.device('cuda:1'))                                                                                                                                         

In [5]: %timeit x_gpu**2                                                        
48.8 µs ± 591 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

EDIT: And if you just care about simple linear algebra operations on the CPU, I would recommend using NumPy over PyTorch.
In [6]: import numpy as np                                                      

In [7]: x_np = np.array(x)                                                      

In [8]: %timeit x_np**2                                                         
364 µs ± 9.05 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)