I custom PyTorch word language model (my code https://github.com/ttpro1995/custom_word_language_model 35)
I add convolution layer before LSTM. (see file class ModelWrapper in https://github.com/ttpro1995/custom_word_language_model/blob/master/model/model_wrapper.py 10)
In main.py 3, I freeze the encoder (embedding) and train convolution, lstm, decoder
    for p in model.conv_module.parameters():
        p.data.add_(-lr, p.grad.data)

    for p in model.rnn.parameters():
        p.data.add_(-lr, p.grad.data)

    for p in model.decoder.parameters():
        p.data.add_(-lr, p.grad.data)

I got test ppl 3.70, which is too small.
Run command
python main.py --cuda --emsize 300 --nhid 168 --dropout 0.5 --epochs 40 --noglove
log file https://gist.github.com/e7644ad05836b6a147cb243e3764ff1f 21
Please tell me if anything go wrong.