title: Scala Spark Application is not exiting with proper status
body: <p>I am working on a spark stand alone application , which has a below code structure:</p>
  <p><strong>code structure:</strong></p>
  <pre><code>jobStatus=Pass
  try
  {
  calls rest API based on status it throws exception or success
  }
  catch{ 
  catch all exceptions
  jobStatus =Fail
  }
  finally
  {
  jobStatus match {
          //case Pass =&gt; System.exit(0)
          case Pass =&gt; return(0) 
          //case Fail =&gt; System.exit(1)
          case Fail =&gt; return(1) 
        }
  }
  </code></pre>
  <p>So,if i use System.exit (0) or (1) the  spark application always exit with Failed Status(even though there is no exceptions).</p>
  <p>If i use return(0) or (1) the spark application always ends with Success status(even though there is exception)</p>
  <p><strong>Versions used :</strong></p>
  <p>Scala Version : 2.11.0</p>
  <p>Spark version: 2.2.0</p>
  <p><strong>Spark log (return(0) or return(1) case:</strong></p>
  <p><em>INFO SparkContext: Successfully stopped SparkContext</em></p>
  <p><em>INFO ApplicationMaster: Unregistering ApplicationMaster with <strong>SUCCEEDED</strong></em></p>
  <p><em>INFO AMRMClientImpl: Waiting for application to be successfully unregistered.</em></p>
  <p><em>INFO ShutdownHookManager: Shutdown hook called</em></p>
  <p><strong>Spark log (System.exit(0) or System.exit(1) case:</strong></p>
  <p><em>INFO SparkContext: Successfully stopped SparkContext</em></p>
  <p><em>INFO ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)</em></p>
  <p><em>INFO ApplicationMaster: Unregistering ApplicationMaster with <strong>FAILED</strong> (diag message: Shutdown hook called before final status was reported.)</em></p>
  <p><em>INFO AMRMClientImpl: Waiting for application to be successfully unregistered</em></p>
  <p>I am looking for suggestions or  inputs to fix this scenario to  end the application with proper status. Let me know if you need any other details on this.</p>