title: H2OAutoML throws libgomp exception in train step
body: <p>I run H2O on a docker image using Python 3.6.3 and H2O 3.26.0.3.</p>
  
  <pre><code>import h2o
  from h2o.automl import H2OAutoML
  h2o.init()
  </code></pre>
  
  <p>In this step, initialization is successful and it prints the following information.</p>
  
  <pre><code>H2O cluster uptime: 01 secs
  H2O cluster timezone:   Europe/Istanbul
  H2O data parsing timezone:  UTC
  H2O cluster version:    3.26.0.3
  H2O cluster version age:    9 days
  H2O cluster name:   H2O_from_python_96273_8m5wyj
  H2O cluster total nodes:    1
  H2O cluster free memory:    26.67 Gb
  H2O cluster total cores:    72
  H2O cluster allowed cores:  72
  H2O cluster status: accepting new members, healthy
  H2O connection url: http://127.0.0.1:54321
  H2O connection proxy:   None
  H2O internal security:  False
  H2O API Extensions: Amazon S3, XGBoost, Algos, AutoML, Core V3, Core V4
  Python version: 3.6.3 final
  </code></pre>
  
  <p>Now, I will run AutoML but it is problematic.</p>
  
  <pre><code>hf = h2o.H2OFrame(x_train)
  aml = H2OAutoML(max_runtime_secs=600)
  aml.train(x = list(df.columns[:-1]), y = df.columns[-1], training_frame = hf)
  </code></pre>
  
  <p>I have the following error</p>
  
  <blockquote>
    <p>ConnectionResetError                      Traceback (most recent call
    last) ~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py
    in urlopen(self, method, url, body, headers, retries, redirect,
    assert_same_host, timeout, pool_timeout, release_conn, chunked,
    body_pos, **response_kw)
        599                                                   body=body, headers=headers,
    --> 600                                                   chunked=chunked)
        601 </p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in
    _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
        383                     # otherwise it looks like a programming error was the cause.
    --> 384                     six.raise_from(e, None)
        385         except (SocketTimeout, BaseSSLError, SocketError) as e:</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/packages/six.py in
    raise_from(value, from_value)</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in
    _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
        379                 try:
    --> 380                     httplib_response = conn.getresponse()
        381                 except Exception as e:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py in
    getresponse(self)    1330             try:
    -> 1331                 response.begin()    1332             except ConnectionError:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py in
    begin(self)
        296         while True:
    --> 297             version, status, reason = self._read_status()
        298             if status != CONTINUE:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py in
    _read_status(self)
        257     def _read_status(self):
    --> 258         line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
        259         if len(line) > _MAXLINE:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/socket.py in
    readinto(self, b)
        585             try:
    --> 586                 return self._sock.recv_into(b)
        587             except timeout:</p>
    
    <p>ConnectionResetError: [Errno 104] Connection reset by peer</p>
    
    <p>During handling of the above exception, another exception occurred:</p>
    
    <p>ProtocolError                             Traceback (most recent call
    last) ~/.local/lib/python3.6/site-packages/requests/adapters.py in
    send(self, request, stream, timeout, verify, cert, proxies)
        448                     retries=self.max_retries,
    --> 449                     timeout=timeout
        450                 )</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in
    urlopen(self, method, url, body, headers, retries, redirect,
    assert_same_host, timeout, pool_timeout, release_conn, chunked,
    body_pos, **response_kw)
        637             retries = retries.increment(method, url, error=e, _pool=self,
    --> 638                                         _stacktrace=sys.exc_info()[2])
        639             retries.sleep()</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/util/retry.py in
    increment(self, method, url, response, error, _pool, _stacktrace)
        367             if read is False or not self._is_method_retryable(method):
    --> 368                 raise six.reraise(type(error), error, _stacktrace)
        369             elif read is not None:</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/packages/six.py in
    reraise(tp, value, tb)
        684         if value.<strong>traceback</strong> is not tb:
    --> 685             raise value.with_traceback(tb)
        686         raise value</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in
    urlopen(self, method, url, body, headers, retries, redirect,
    assert_same_host, timeout, pool_timeout, release_conn, chunked,
    body_pos, **response_kw)
        599                                                   body=body, headers=headers,
    --> 600                                                   chunked=chunked)
        601 </p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in
    _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
        383                     # otherwise it looks like a programming error was the cause.
    --> 384                     six.raise_from(e, None)
        385         except (SocketTimeout, BaseSSLError, SocketError) as e:</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/packages/six.py in
    raise_from(value, from_value)</p>
    
    <p>~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in
    _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
        379                 try:
    --> 380                     httplib_response = conn.getresponse()
        381                 except Exception as e:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py in
    getresponse(self)    1330             try:
    -> 1331                 response.begin()    1332             except ConnectionError:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py in
    begin(self)
        296         while True:
    --> 297             version, status, reason = self._read_status()
        298             if status != CONTINUE:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py in
    _read_status(self)
        257     def _read_status(self):
    --> 258         line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
        259         if len(line) > _MAXLINE:</p>
    
    <p>/opt/rh/rh-python36/root/usr/lib64/python3.6/socket.py in
    readinto(self, b)
        585             try:
    --> 586                 return self._sock.recv_into(b)
        587             except timeout:</p>
    
    <p>ProtocolError: ('Connection aborted.', ConnectionResetError(104,
    'Connection reset by peer'))</p>
    
    <p>During handling of the above exception, another exception occurred:</p>
    
    <p>ConnectionError                           Traceback (most recent call
    last) ~/.local/lib/python3.6/site-packages/h2o/backend/connection.py
    in request(self, endpoint, data, json, filename, save_to)
        404                                     headers=headers, timeout=self._timeout, stream=stream,
    --> 405                                     auth=self._auth, verify=self._verify_ssl_cert, proxies=self._proxies)
        406             self._log_end_transaction(start_time, resp)</p>
    
    <p>~/.local/lib/python3.6/site-packages/requests/api.py in
    request(method, url, **kwargs)
         59     with sessions.Session() as session:
    ---> 60         return session.request(method=method, url=url, **kwargs)
         61 </p>
    
    <p>~/.local/lib/python3.6/site-packages/requests/sessions.py in
    request(self, method, url, params, data, headers, cookies, files,
    auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert,
    json)
        532         send_kwargs.update(settings)
    --> 533         resp = self.send(prep, **send_kwargs)
        534 </p>
    
    <p>~/.local/lib/python3.6/site-packages/requests/sessions.py in
    send(self, request, **kwargs)
        645         # Send the request
    --> 646         r = adapter.send(request, **kwargs)
        647 </p>
    
    <p>~/.local/lib/python3.6/site-packages/requests/adapters.py in
    send(self, request, stream, timeout, verify, cert, proxies)
        497         except (ProtocolError, socket.error) as err:
    --> 498             raise ConnectionError(err, request=request)
        499 </p>
    
    <p>ConnectionError: ('Connection aborted.', ConnectionResetError(104,
    'Connection reset by peer'))</p>
    
    <p>During handling of the above exception, another exception occurred:</p>
    
    <p>H2OConnectionError                        Traceback (most recent call
    last)  in 
    ----> 1 aml.train(x = list(df.columns[:-1]), y = df.columns[-1], training_frame = hf)</p>
    
    <p>~/.local/lib/python3.6/site-packages/h2o/automl/autoh2o.py in
    train(self, x, y, training_frame, fold_column, weights_column,
    validation_frame, leaderboard_frame, blending_frame)
        443         poll_updates = ft.partial(self._poll_training_updates, verbosity=self._verbosity, state={})
        444         try:
    --> 445             self._job.poll(poll_updates=poll_updates)
        446         finally:
        447             poll_updates(self._job, 1)</p>
    
    <p>~/.local/lib/python3.6/site-packages/h2o/job.py in poll(self,
    poll_updates)
         55             pb = ProgressBar(title=self._job_type + " progress", hidden=hidden)
         56             if poll_updates:
    ---> 57                 pb.execute(self._refresh_job_status, print_verbose_info=ft.partial(poll_updates, self))
         58             else:
         59                 pb.execute(self._refresh_job_status)</p>
    
    <p>~/.local/lib/python3.6/site-packages/h2o/utils/progressbar.py in
    execute(self, progress_fn, print_verbose_info)
        169                 # Query the progress level, but only if it's time already
        170                 if self._next_poll_time &lt;= now:
    --> 171                     res = progress_fn()  # may raise StopIteration
        172                     assert_is_type(res, (numeric, numeric), numeric)
        173                     if not isinstance(res, tuple):</p>
    
    <p>~/.local/lib/python3.6/site-packages/h2o/job.py in
    _refresh_job_status(self)
         92     def _refresh_job_status(self):
         93         if self._poll_count &lt;= 0: raise StopIteration("")
    ---> 94         jobs = h2o.api("GET /3/Jobs/%s" % self.job_key)
         95         self.job = jobs["jobs"][0] if "jobs" in jobs else jobs["job"][0]
         96         self.status = self.job["status"]</p>
    
    <p>~/.local/lib/python3.6/site-packages/h2o/h2o.py in api(endpoint, data,
    json, filename, save_to)
        102     # type checks are performed in H2OConnection class
        103     _check_connection()
    --> 104     return h2oconn.request(endpoint, data=data, json=json, filename=filename, save_to=save_to)
        105 
        106 </p>
    
    <p>~/.local/lib/python3.6/site-packages/h2o/backend/connection.py in
    request(self, endpoint, data, json, filename, save_to)
        413             else:
        414                 self._log_end_exception(e)
    --> 415                 raise H2OConnectionError("Unexpected HTTP error: %s" % e)
        416         except requests.exceptions.Timeout as e:
        417             self._log_end_exception(e)</p>
    
    <p>H2OConnectionError: Unexpected HTTP error: ('Connection aborted.',
    ConnectionResetError(104, 'Connection reset by peer'))</p>
  </blockquote>
  
  <p>I suspect that proxy might be the reason of this exception. When I add the proxy information to the path, then exception message would be "HTTP 500 INKApi Error"</p>
  
  <pre><code>import os
  os.environ['http_proxy']= ...
  os.environ['https_proxy']= ...
  </code></pre>
  
  <p>JVM stdout log file dumps the following exception.</p>
  
  <blockquote>
    <p>[thread 140335217821440 also had an error][thread 140335320467200 also
    had an error] [thread 140335207294720 also had an error]</p>
    
    <p>[thread 140335316256512 also had an error]# A fatal error has been detected by the Java Runtime Environment:</p>
    
    <p>[thread 140335202031360 also had an error]
      SIGSEGV (0xb) at pc=0x00007fa3276cdb8d, pid=51986, tid=0x00007fa2575f5700</p>
    
    <p>JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
     Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
     Problematic frame:
     [thread 140335231506176 also had an error] C  [libc.so.6+0x39b8d][thread 140335341520640 also had an error]</p>
  </blockquote>
  
  <p>JVM stderr log file contains interesting logs</p>
  
  <blockquote>
    <p>libgomp: Thread creation failed: Resource temporarily unavailable
    <strong>* Error in `/usr/bin/java': free(): corrupted unsorted chunks: 0x00007efe342f0240 *</strong></p>
    
    <p>libgomp: Thread creation failed: Resource temporarily unavailable</p>
  </blockquote>
  
  <p>Funny but It runs successfully when I run same code on my local machine. I suspect that it might be because of docker configuration.</p>