<p dir="auto"><strong>Apache Airflow version</strong>: 1.10.10 and 1.10.9 has been verified</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):<br>
no Kube</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Others</strong>: docker environment with java 1.8</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">The existing function to extract a job id from the monitoring page is no longer compatible with new beam 2.20:</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/apache/airflow/blob/96697180d79bfc90f6964a8e99f9dd441789177c/airflow/contrib/hooks/gcp_dataflow_hook.py#L142-L148">airflow/airflow/contrib/hooks/gcp_dataflow_hook.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 142 to 148
      in
      <a data-pjax="true" class="commit-tease-sha" href="/apache/airflow/commit/96697180d79bfc90f6964a8e99f9dd441789177c">9669718</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L142" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="142"></td>
          <td id="LC142" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">def</span> <span class="pl-en">_extract_job</span>(<span class="pl-s1">line</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L143" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="143"></td>
          <td id="LC143" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-c"># Job id info: https://goo.gl/SE29y9.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L144" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="144"></td>
          <td id="LC144" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">job_id_pattern</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">compile</span>( </td>
        </tr>

        <tr class="border-0">
          <td id="L145" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="145"></td>
          <td id="LC145" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s">br'.*console.cloud.google.com/dataflow.*/jobs/([a-z|0-9|A-Z|\-|\_]+).*'</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L146" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="146"></td>
          <td id="LC146" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">matched_job</span> <span class="pl-c1">=</span> <span class="pl-s1">job_id_pattern</span>.<span class="pl-en">search</span>(<span class="pl-s1">line</span> <span class="pl-c1">or</span> <span class="pl-s">''</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L147" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="147"></td>
          <td id="LC147" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">if</span> <span class="pl-s1">matched_job</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L148" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="148"></td>
          <td id="LC148" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-k">return</span> <span class="pl-s1">matched_job</span>.<span class="pl-en">group</span>(<span class="pl-c1">1</span>).<span class="pl-en">decode</span>() </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">The monitoring page includes the zone as part of the URL causing the regex to extract always the <code class="notranslate">zone</code> instead of the <code class="notranslate">job_id</code></p>
<p dir="auto">A new regex compatible with the new and old monitoring url needs to be added to the operator:</p>
<p dir="auto"><a href="https://github.com/apache/beam/blob/v2.20.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java#L192-L195">https://github.com/apache/beam/blob/v2.20.0/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java#L192-L195</a></p>
<p dir="auto">As per the been new configuration region is extracted instead of the job id</p>
<p dir="auto"><strong>Apache Airflow 1.10.10</strong>:</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
NAME="CentOS Linux"<br>
VERSION="7 (Core)"<br>
ID="centos"<br>
ID_LIKE="rhel fedora"<br>
VERSION_ID="7"<br>
PRETTY_NAME="CentOS Linux 7 (Core)"<br>
ANSI_COLOR="0;31"<br>
CPE_NAME="cpe:/o:centos:centos:7"<br>
HOME_URL="<a href="https://www.centos.org/" rel="nofollow">https://www.centos.org/</a>"<br>
BUG_REPORT_URL="<a href="https://bugs.centos.org/" rel="nofollow">https://bugs.centos.org/</a>"</li>
</ul>
<p dir="auto">CENTOS_MANTISBT_PROJECT="CentOS-7"<br>
CENTOS_MANTISBT_PROJECT_VERSION="7"<br>
REDHAT_SUPPORT_PRODUCT="centos"<br>
REDHAT_SUPPORT_PRODUCT_VERSION="7"</p>
<ul dir="auto">
<li>
<p dir="auto"><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):<br>
Linux mid1-t029nifi-1 3.10.0-327.28.3.el7.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="69689814" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/1" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/1/hovercard" href="https://github.com/apache/airflow/pull/1">#1</a> SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</p>
</li>
<li>
<p dir="auto"><strong>Install tools</strong>:<br>
pip, yum</p>
</li>
<li>
<p dir="auto"><strong>Others</strong>:</p>
</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">When dag serialisation is active, If I manually start an operator, the 1st one works fine, the next will fail with this error:</p>
<p dir="auto">Could not queue task instance for execution, dependencies not met: Trigger Rule: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'skipped': Decimal('0'), 'successes': Decimal('0'), 'failed': Decimal('0'), 'upstream_failed': Decimal('0'), 'done': 0L, 'total': 1}, upstream_task_ids=set([u'query']</p>
<p dir="auto">Settings dag serialisation to false the problem does not arise.</p>
<p dir="auto">please note : <em>Scheduler works fine</em>.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I expected to start manually all the dag's tasks from the 1st one to the last.</p>

<p dir="auto">Code is not able to correctly find the task's status that is before the one I'm restarting.<br>
If I start the 1st operator, anything works fine.</p>
<p dir="auto">You can reproduce it following these steps:</p>
<ol start="0" dir="auto">
<li>enable dag serialisation</li>
<li>put the DAG in pause ( so that the scheduler won't touch it )</li>
<li>start the 1st operator and wait it completes and it's successful</li>
<li>start the 2nd operator....</li>
</ol>
<p dir="auto">op1 &gt;&gt; op2</p>
<p dir="auto"><strong>Anything else we need to know</strong>:</p>
<p dir="auto">This happens every time.<br>
Mysql 5.7.x, Python 2.7</p>