Thank you @patrickwilliams3
A convolution layer computes the inner product along the 1 dimension. A fully connected layer can be implemented using 1x1 convolution. Take a segmentation network as an example. The last layer in a segmentation network is usually implemented using 1x1 convolution.
What it does in fact is computing the inner product of weights in the last layer with the embeddings of each pixel to obtain logits (class predictions) of each pixel.
There is no question that we can implemented a fully connected layer using 1x1 convolutions. What I don’t understand is why the results in the above codes are different.
This might be a dump hypothesis: Perhaps the order of operations in convolution implementation is different? Perhaps something is happening during permutation? More likely, there is a bug in the above code and I just don’t see it. Even more likely, I am entirely 
P.S: You can compare the results of 1x1 convolution with tensordot operation as well.