The code looks alright.
Note that you would have to add nn.Flatten before the linear layer in case your conv layer returns more then a single output channel.
In the current setup it should be working. 
What do you mean by: “examine the weights, they don’t look like I understood/expected them to be”?