title: Spark History UI is available but tracking UI errors for in-progress applications
body: <p>I've inherited a cluster that uses knox and am trying to figure out why the Spark history server is available for completed Spark jobs but the Spark UI is not available for in-progress Spark applications. </p>
  
  <p>In this yarn UI (which is exposed via Knox) there are 5 completed yarn applications and 1 in-progress yarn application. All are spark applications:
  <a href="https://i.stack.imgur.com/7yhok.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7yhok.png" alt="yarn UI exposed via knox"></a></p>
  
  <p>In the Tracking UI columns the available links are:</p>
  
  <ul>
  <li><a href="https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0001" rel="nofollow noreferrer">https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0001</a></li>
  <li><a href="https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0002" rel="nofollow noreferrer">https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0002</a></li>
  <li><a href="https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0003" rel="nofollow noreferrer">https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0003</a></li>
  <li><a href="https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0004" rel="nofollow noreferrer">https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0004</a></li>
  <li><a href="https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0005" rel="nofollow noreferrer">https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0005</a></li>
  <li><a href="https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0006" rel="nofollow noreferrer">https://my-knox-endpoint/gateway/my-cluster/yarn/proxy/application_1580137635209_0006</a></li>
  </ul>
  
  <p>The five links pertaining to the completed jobs all successfully bring up the Spark History server UI for those jobs. If I issue <code>cat ${GATEWAY_HOME}/logs/gateway-audit.log</code> I can see the following appear when I hit any of those five links:</p>
  
  <blockquote>
    <p>20/01/27 15:50:55 ||55bef3f3-a52f-4790-97d0-bd6e5076a293|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.0.229|YARNUI||||access|uri|/gateway/my-cluster-name/yarn/proxy/application_1580137635209_0001|unavailable|Request method: GET<br>
    20/01/27 15:50:55 ||55bef3f3-a52f-4790-97d0-bd6e5076a293|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.0.229|YARNUI||||dispatch|uri|<a href="http://my-cluster-name-m:8088/proxy/application_1580137635209_0001|unavailable|Request" rel="nofollow noreferrer">http://my-cluster-name-m:8088/proxy/application_1580137635209_0001|unavailable|Request</a> method: GET<br>
    <strong>20/01/27 15:50:55 ||55bef3f3-a52f-4790-97d0-bd6e5076a293|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.0.229|YARNUI||||dispatch|uri|<a href="http://my-cluster-name-m:8088/proxy/application_1580137635209_0001|success|Response" rel="nofollow noreferrer">http://my-cluster-name-m:8088/proxy/application_1580137635209_0001|success|Response</a> status: 302<br>
    20/01/27 15:50:55 |||audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.0.229|YARNUI||||access|uri|/gateway/my-cluster-name/yarn/proxy/application_1580137635209_0001|success|Response status: 302</strong><br>
    20/01/27 15:50:55 ||f7617e15-3bf4-4a8c-9701-9785894d7884|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.0.234|SPARKHISTORYUI||||access|uri|/gateway/my-cluster-name/sparkhistory/history/application_1580137635209_0001/1|unavailable|Request method: GET
    20/01/27 15:50:55 ||f7617e15-3bf4-4a8c-9701-9785894d7884|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.0.234|SPARKHISTORYUI||||dispatch|uri|<a href="http://my-cluster-name-m:18080/history/application_1580137635209_0001/1/|unavailable|Request" rel="nofollow noreferrer">http://my-cluster-name-m:18080/history/application_1580137635209_0001/1/|unavailable|Request</a> method: GET<br>
    20/01/27 15:50:55 ||f7617e15-3bf4-4a8c-9701-9785894d7884|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.0.234|SPARKHISTORYUI||||dispatch|uri|<a href="http://my-cluster-name-m:18080/history/application_1580137635209_0001/1/|success|Response" rel="nofollow noreferrer">http://my-cluster-name-m:18080/history/application_1580137635209_0001/1/|success|Response</a> status: 30</p>
  </blockquote>
  
  <p>and lots and lots of other log records for Spark History UI resources. All good. Notice the 302 record (redirect)</p>
  
  <p>However, if I hit the link for the in-progress application I get sent to <a href="http://my-cluster-name-m:18080/history/application_1580137635209_0006/1" rel="nofollow noreferrer">http://my-cluster-name-m:18080/history/application_1580137635209_0006/1</a> which is the cluster master node, and the following displayed:
  <a href="https://i.stack.imgur.com/snw59.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/snw59.png" alt="enter image description here"></a></p>
  
  <p>In the logs I see:</p>
  
  <blockquote>
    <p>20/01/27 15:58:38 ||aec261d3-7ecc-43a7-8815-d7185ee13833|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.1.130|YARNUI||||access|uri|/gateway/my-cluster-name/yarn/proxy/application_1580137635209_0006|unavailable|Request method: GET<br>
    20/01/27 15:58:38 ||aec261d3-7ecc-43a7-8815-d7185ee13833|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.1.130|YARNUI||||dispatch|uri|<a href="http://my-cluster-name-m:8088/proxy/application_1580137635209_0006|unavailable|Request" rel="nofollow noreferrer">http://my-cluster-name-m:8088/proxy/application_1580137635209_0006|unavailable|Request</a> method: GET<br>
    20/01/27 15:58:38 ||aec261d3-7ecc-43a7-8815-d7185ee13833|audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.1.130|YARNUI||||dispatch|uri|<a href="http://my-cluster-name-m:8088/proxy/application_1580137635209_0006|success|Response" rel="nofollow noreferrer">http://my-cluster-name-m:8088/proxy/application_1580137635209_0006|success|Response</a> status: 200<br>
    20/01/27 15:58:38 |||audit|109.231.200.210, 165.225.80.109, 34.102.220.138, 130.211.1.130|YARNUI||||access|uri|/gateway/my-cluster-name/yarn/proxy/application_1580137635209_0006|success|Response status: 200  </p>
  </blockquote>
  
  <p>Notice there are no 302 records there. </p>
  
  <p>Edit: Since originally posting this I have noticed that if i click on the Tracking UI link <strong>immediately</strong> after the application starts then I am taken to the details of the yarn application:</p>
  
  <p><a href="https://i.stack.imgur.com/EGfR9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EGfR9.png" alt="yarn app details"></a></p>
  
  <p>A few seconds later clicking on the same link will take me to the error as shown above. </p>
  
  <p>I'm a bit lost at this point. Can anyone help explain why I can't view the Spark UI for in-progress applications? Any pointers as to how I can diagnose would be welcomed.</p>