See this:

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Libtorch index API perform poorly under repetitive usage 31
    

    
      
        opened 
        
          
        
        Sep 22, 2020
      
      


      
        
          
          linjiapro
        
      
    
  



  🐛 Bug
I have a Libtorch project that has some execution speed issues. I have spent half a day to debug it....





  
  
    
    
  
  


Accessor is much faster but still slower than vector. Regarding why index is so slow, look at this:

  
      github.com
  
  
    pytorch/pytorch/blob/1494005cfdf228f8441c1a7aecdfe1c3dfc77de6/aten/src/ATen/TensorIndexing.cpp 5
#include <ATen/TensorIndexing.h>

#include <c10/util/Exception.h>

namespace at {
namespace indexing {

const EllipsisIndexType Ellipsis = EllipsisIndexType();

std::ostream& operator<<(std::ostream& stream, const Slice& slice) {
  stream << slice.start() << ":" << slice.stop() << ":" << slice.step();
  return stream;
}

std::ostream& operator<<(std::ostream& stream, const TensorIndex& tensor_index) {
  if (tensor_index.is_none()) {
    stream << "None";
  } else if (tensor_index.is_ellipsis()) {
    stream << "...";
  } else if (tensor_index.is_integer()) {


  This file has been truncated. show original

  
  
    
    
  
  


Tensor Tensor::index(ArrayRefat::indexing::TensorIndex indices) const {
TORCH_CHECK(indices.size() > 0, “Passing an empty index list to Tensor::index() is not valid syntax”);
OptionalDeviceGuard device_guard(device_of(*this));
return at::indexing::get_item(*this, indices);
}
I suspect the device guard is heavy. Further, the get_item call is here:

  
      github.com
  
  
    pytorch/pytorch/blob/09660896c0dd2bec888857300a7be9edb52dd05d/aten/src/ATen/TensorIndexing.h 4
#pragma once

#include <c10/util/Optional.h>
#include <ATen/core/TensorBody.h>
#include <ATen/ExpandUtils.h>
#include <ATen/Functions.h>

namespace at {
namespace indexing {

const int64_t INDEX_MAX = std::numeric_limits<int64_t>::max();
const int64_t INDEX_MIN = std::numeric_limits<int64_t>::min();

enum class TensorIndexType { None, Ellipsis, Integer, Boolean, Slice, Tensor };

constexpr c10::nullopt_t None{c10::nullopt_t::init()};

struct CAFFE2_API EllipsisIndexType final { EllipsisIndexType() {} };
CAFFE2_API extern const EllipsisIndexType Ellipsis;



  This file has been truncated. show original

  
  
    
    
  
  


self_device is even pass around. My guess is that once CPU or GPU device stuff is involved. It is going to be expensive.