You can check the official code https://github.com/pathak22/context-encoder/blob/master/train.lua#L103 309. In fact, there is no channel-wise fc layer and the details of that is the conv2d+convTranspose2d.