Try to use more batches, the first one will be always slower, because lot of time will be spent allocating memory that will be cached for subsequent runs. Are the batch sizes and model parameters exactly the same as in tensorflow? I can try to run your model tomorrow, but it would be helpful if you could also provide me with the code for tensorflow implementation.