I’ve migrated to PyTorch from Chainer for the library of deep learning,
and found PyTorch is a little slower than Chainer at test time with convolutional networks.
I’ve noticed this when implementing convolutional networks for segmentation:
% ./speedtest.py
==> Running on GPU: 0 to evaluate 1000 times
==> Testing FCN32s with Chainer
Elapsed time: 52.03 [s / 1000 evals]
Hz: 19.22 [hz]
==> Testing FCN32s with PyTorch
Elapsed time: 58.78 [s / 1000 evals]
Hz: 17.01 [hz]

I expected that PyTorch is faster than Chainer,
because it use C extension to make computations faster in most functional implementations.
Is this a known result?
I’ve checked convnet-benchmarks 45, but couldn’t find result of PyTorch.
(I tried with torch.backends.cudnn.benchmark = True and it shows ~22Hz in PyTorch, but I heard it limits input tensor size, and not same condition with Chainer.)
Speed Test

  
      github.com
  
  
    wkentaro/pytorch-fcn/blob/master/examples/voc/speedtest.py 67
#!/usr/bin/env python

import argparse
import time

import numpy as np
import six


def bench_chainer(gpu, times, dynamic_input=False):
    import chainer
    import fcn
    print('==> Testing FCN32s with Chainer')
    chainer.cuda.get_device(gpu).use()

    chainer.config.train = False
    chainer.config.enable_backprop = False

    if dynamic_input:
        x_data = np.random.random((1, 3, 480, 640)).astype(np.float32)


  This file has been truncated. show original

  
  
    
    
  
  


PyTorch implementation

  
      github.com
  
  
    wkentaro/pytorch-fcn/blob/master/torchfcn/models/fcn32s.py 43
import os.path as osp

import fcn
import numpy as np
import torch
import torch.nn as nn


# https://github.com/shelhamer/fcn.berkeleyvision.org/blob/master/surgery.py
def get_upsampling_weight(in_channels, out_channels, kernel_size):
    """Make a 2D bilinear kernel suitable for upsampling"""
    factor = (kernel_size + 1) // 2
    if kernel_size % 2 == 1:
        center = factor - 1
    else:
        center = factor - 0.5
    og = np.ogrid[:kernel_size, :kernel_size]
    filt = (1 - abs(og[0] - center) / factor) * \
           (1 - abs(og[1] - center) / factor)
    weight = np.zeros((in_channels, out_channels, kernel_size, kernel_size),


  This file has been truncated. show original

  
  
    
    
  
  


Chainer implementation

  
      github.com
  
  
    wkentaro/fcn/blob/master/fcn/models/fcn32s.py 25
import os.path as osp

import chainer
import chainer.functions as F
import chainer.links as L
import numpy as np

from .. import data
from .. import initializers


class FCN32s(chainer.Chain):

    pretrained_model = osp.expanduser(
        '~/data/models/chainer/fcn32s_from_caffe.npz')

    def __init__(self, n_class=21):
        self.n_class = n_class
        kwargs = {
            'initialW': chainer.initializers.Zero(),


  This file has been truncated. show original