title: Parallel scan requests to HBase in Java and Python have different performance
body: <h1>Statement</h1>
  <p>We have 10 machines HBase cluster and billions of rows inside. Every row consists of one column family and ~20 columns. We need perform frequent scan requests which contains start row prefix and end row prefix. Usually every scan returns abount 100 - 10000 rows.</p>
  <p>Because requests can come very often (up to several requests per minute), so performance is preoritized. Due to system's architecture we want to realize our solution in Python instead of current Java code. The problem is with Python we obtain 5x-10x worse performance than in Java.</p>
  <h1>What now works</h1>
  <p>We have Java code which perform scan requests to HBase. It uses ususal HBase Java API:</p>
  <pre><code>public List&lt;String&gt; getNumber(Number key) {
      List&lt;String&gt; res = new ArrayList&lt;&gt;();
  
      String start_key = key.getNumber();
      String next_key = key.getNumber() + &quot;1&quot;;
      byte[] prefix_begin = Bytes.toBytes(start_key);
      byte[] prefix_end = Bytes.toBytes(next_key);
      Scan scan = new Scan(prefix_begin, prefix_end);
      ResultScanner scanner = table.getScanner(scan);
      for (Result result : scanner) {
          byte[] row = result.getRow();
          res.add(Bytes.toString(row));
      }
      
      return res;
  }
  </code></pre>
  <p>These queries parallelized with the help of <code>Callable</code> interface and <code>ScheduledThreadPoolExecutor</code>. The <code>call()</code> method of every callable just run <code>getNumber(Number key)</code>.</p>
  <pre><code>public List&lt;String&gt; getNumbers(List&lt;Number&gt; keys) {
      List&lt;String&gt; res = new ArrayList&lt;String&gt;();
  
      List&lt;Callables.CallingCallable&gt; callables = new ArrayList();
      for (Number source : keys) {
          callables.add(new Callables.CallingCallable(this, source));
      }
  
      Object futures = new ArrayList();
      ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(24);
  
      try {
          futures = executor.invokeAll(callables);
      } catch (InterruptedException ex) {
      }
  
      executor.shutdown();
  }
  </code></pre>
  <p>This works pretty good and allows achieve following performance:</p>
  <ul>
  <li><strong>1.5 - 2.0 sec</strong> per single scan and</li>
  <li><strong>5.0 - 8.0 sec</strong> per 100 parallelized scans</li>
  </ul>
  <h1>What we try</h1>
  <p>We try to implement similar solution in Python with the help of <a href="https://happybase.readthedocs.org/en/latest/#" rel="nofollow noreferrer">Happybase</a> library:</p>
  <pre><code>@staticmethod
  def execute_query(key, table_name, con_pool):
          items = []
          with con_pool.connection() as connection:
              table = happybase.Table(table_name, connection)
              [row_start, row_end] = get_start_and_end_row(key)
              selected_rows = table.scan(row_start=row_start, row_stop=row_end)
              for key, data in selected_rows:
                  items.append(Item(data))
          return items
  
  @staticmethod
  def execute_in_parallel(table_name, hbase_host, hbase_port, keys):
          pool = ThreadPool(24)
          con_pool = happybase.ConnectionPool(size=24, host=hbase_host, port=hbase_port)
          execute_query_partial = partial(execute_query, table_name=table_name, con_pool=con_pool)
          result_info = pool.map_async(execute_query_partial, keys, chunksize=1)
          result = result_info.get()
  </code></pre>
  <p>Achieved performance:</p>
  <ul>
  <li><strong>2.0 - 3.0 sec</strong> per single scan and</li>
  <li><strong>30 - 55 sec</strong> per 100 parallelized scans</li>
  </ul>
  <p>As we can see performance of single scan is very similar. But parallelized tasks  in Python are much slower.</p>
  <p>Any ideas why does it happen? Maybe some issues with our Python/Happybase code? Or performance of HBase Thrift server (which HappyBase uses to connect to HBase)?</p>