title: boto3 glue get_job_runs - check execution with certain date exists in the response object
body: <p>I am trying to fetch glue job executions that got failed previous day using 'get_job_runs' function available through boto3's glue client.
  <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_job_runs" rel="nofollow noreferrer">https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_job_runs</a>.</p>
  <p>The request syntax, does not have an option to filter executions or job runs by date/status -</p>
  <pre><code>response = client.get_job_runs(
      JobName='string',
      NextToken='string',
      MaxResults=123
  )
  </code></pre>
  <p>The response I receive back looks something like below -</p>
  <pre><code>{
    &quot;JobRuns&quot;: [
      {
        &quot;Id&quot;: &quot;jr_89bfa55b544f7eec4f6ea574dfb0345345uhi4df65e59869e93c5d8f5efef989&quot;,
        &quot;Attempt&quot;: 0,
        &quot;JobName&quot;: &quot;GlueJobName&quot;,
        &quot;StartedOn&quot;: datetime.datetime(2021, 1, 27, 4, 32, 47, 718000, tzinfo=tzlocal()),
        &quot;LastModifiedOn&quot;: datetime.datetime(2021, 1, 27, 4, 36, 14, 975000, tzinfo=tzlocal()),
        &quot;CompletedOn&quot;: datetime.datetime(2021, 1, 27, 4, 36, 14, 975000, tzinfo=tzlocal()),
        &quot;JobRunState&quot;: &quot;FAILED&quot;,
        &quot;Arguments&quot;: {
          &quot;--additional-python-modules&quot;: &quot;awswrangler&quot;,
          &quot;--conf&quot;: &quot;spark.executor.memory=40g&quot;,
          &quot;--conf &quot;: &quot;spark.driver.memory=40g&quot;,
          &quot;--enable-spark-ui&quot;: &quot;true&quot;,
          &quot;--extra-py-files&quot;: &quot;s3://GlueJobName/lttb.py&quot;,
          &quot;--job-bookmark-option&quot;: &quot;job-bookmark-disable&quot;,
          &quot;--spark-event-logs-path&quot;: &quot;s3://GlueJobName/glue-script/spark-event-logs&quot;
        },
        &quot;ErrorMessage&quot;: &quot;MemoryError: Unable to allocate xxxxx&quot;,
        &quot;PredecessorRuns&quot;: [],
        &quot;AllocatedCapacity&quot;: 8,
        &quot;ExecutionTime&quot;: 199,
        &quot;Timeout&quot;: 2880,
        &quot;MaxCapacity&quot;: 8.0,
        &quot;WorkerType&quot;: &quot;G.2X&quot;,
        &quot;NumberOfWorkers&quot;: 4,
        &quot;LogGroupName&quot;: &quot;/aws-glue/jobs&quot;,
        &quot;GlueVersion&quot;: &quot;2.0&quot;
      }
    ],
    &quot;NextToken&quot;: &quot;string&quot;
  }
  </code></pre>
  <p>So, what I am doing now is looping through the response object to check if the &quot;CompletedOn&quot; date matches with yesterday's date using prev_day calculated using datetime and timedelta and I am doing this in a while loop to fetch last 10000 executions, as the 'get_job_runs' single call is insufficient.</p>
  <pre><code>import boto3
  from datetime import datetime, timedelta
  
  logger = logging.getLogger()
  logger.setLevel(logging.INFO)
  
  glue_client = boto3.client(&quot;glue&quot;)
  
  def filter_failed_exec_prev_day(executions, prev_day) -&gt; list:
      filtered_resp = []
      for execution in executions['JobRuns']:
          if execution['JobRunState'] == 'FAILED' and execution['CompletedOn'].date() == prev_day:
              filtered_resp.append(execution)
      return filtered_resp
  
  
  def get_final_executions() -&gt; list:
      final_job_runs_list = []
      MAX_EXEC_SEARCH_CNT = 10000
      prev_day = (datetime.utcnow() - timedelta(days=1)).date()
      buff_exec_cnt = 0
      l_job = 'GlueJobName'
  
      response = glue_client.get_job_runs(
          JobName=l_job
      )
      resp_count = len(response['JobRuns'])
  
      if resp_count &gt; 0:
          buff_exec_cnt += resp_count
          filtered_resp = filter_failed_exec_prev_day(response, prev_day)
          final_job_runs_list.extend(filtered_resp)
  
          while buff_exec_cnt &lt;= MAX_EXEC_SEARCH_CNT:
              if 'NextToken' in response:
                  response = glue_client.get_job_runs(
                      JobName=l_job
                  )
                  buff_exec_cnt += len(response['JobRuns'])
                  filtered_resp = filter_failed_exec_prev_day(response, prev_day)
                  final_job_runs_list.extend(filtered_resp)
              else:
                  logger.info(f&quot;{job} executions list: {final_job_runs_list}&quot;)
                  break
      return final_job_runs_list
  </code></pre>
  <p>Here, I am using a while loop to break the call after hitting 10K executions, this is triple the amount of executions we see each day on the job.
  <br/>Now, I am hoping to break the while loop after I encounter execution that belongs to <code>prev_day - 1</code>, so is it possible to search the entire response dict for <code>prev_day - 1</code> to make sure all prev day's executions are covered considering the datetime.datetime object we receive from boto3 for <code>CompletedOn</code> attribute?</p>
  <p>Appreciate reading through.</p>
  <p>Thank you</p>