title: AWS Glue pushdown predicate not working properly
body: <p>I'm trying to optimize my Glue/PySpark job by using push down predicates.</p>
  
  <pre><code>start = date(2019, 2, 13) 
  end = date(2019, 2, 27) 
  print("&gt;&gt;&gt; Generate data frame for ", start, " to ", end, "... ")
  relaventDatesDf = spark.createDataFrame([
      Row(start=start, stop=end)
  ])
  relaventDatesDf.createOrReplaceTempView("relaventDates")
  
  relaventDatesDf = spark.sql("SELECT explode(generate_date_series(start, stop)) AS querydatetime FROM relaventDates")
  relaventDatesDf.createOrReplaceTempView("relaventDates")
  print("===LOG:Dates===")
  relaventDatesDf.show()
  
  flightsGDF = glueContext.create_dynamic_frame.from_catalog(database = "xxx", table_name = "flights", transformation_ctx="flights", push_down_predicate="""
      querydatetime BETWEEN '%s' AND '%s'
      AND querydestinationplace IN (%s)
  """ % (start.strftime("%Y-%m-%d"), today.strftime("%Y-%m-%d"), ",".join(map(lambda s: str(s), arr))))
  </code></pre>
  
  <p>However it appears, that Glue still attempts to read data outside the specified date range? </p>
  
  <pre><code>INFO S3NativeFileSystem: Opening 's3://.../flights/querydestinationplace=12191/querydatetime=2019-03-01/part-00045-6cdebbb1-562c-43fa-915d-93b125aeee61.c000.snappy.parquet' for reading
  INFO FileScanRDD: Reading File path: s3://.../flights/querydestinationplace=12191/querydatetime=2019-03-10/part-00021-34a13146-8fb2-43de-9df2-d8925cbe472d.c000.snappy.parquet, range: 0-11797922, partition values: [12191,17965]
  WARN S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
  INFO S3NativeFileSystem: Opening 's3://.../flights/querydestinationplace=12191/querydatetime=2019-03-10/part-00021-34a13146-8fb2-43de-9df2-d8925cbe472d.c000.snappy.parquet' for reading
  WARN S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
  </code></pre>
  
  <p>Notice the <code>querydatetime=2019-03-01</code> and <code>querydatetime=2019-03-10</code> its outside the specified range of <code>2019-02-13 - 2019-02-27</code>. Is that why there's the next line "aborting HTTP connection" tho? It goes on to say "This is likely an error and may result in sub-optimal behavior" is something wrong?  </p>
  
  <p>I wonder if the problem is because it does not support BETWEEN inside the predicate or IN? </p>
  
  <hr>
  
  <p>The table create DDL</p>
  
  <pre><code>CREATE EXTERNAL TABLE `flights`(
    `id` string, 
    `querytaskid` string, 
    `queryoriginplace` string, 
    `queryoutbounddate` string, 
    `queryinbounddate` string, 
    `querycabinclass` string, 
    `querycurrency` string, 
    `agent` string, 
    `quoteageinminutes` string, 
    `price` string, 
    `outboundlegid` string, 
    `inboundlegid` string, 
    `outdeparture` string, 
    `outarrival` string, 
    `outduration` string, 
    `outjourneymode` string, 
    `outstops` string, 
    `outcarriers` string, 
    `outoperatingcarriers` string, 
    `numberoutstops` string, 
    `numberoutcarriers` string, 
    `numberoutoperatingcarriers` string, 
    `indeparture` string, 
    `inarrival` string, 
    `induration` string, 
    `injourneymode` string, 
    `instops` string, 
    `incarriers` string, 
    `inoperatingcarriers` string, 
    `numberinstops` string, 
    `numberincarriers` string, 
    `numberinoperatingcarriers` string)
  PARTITIONED BY ( 
    `querydestinationplace` string, 
    `querydatetime` string)
  ROW FORMAT SERDE 
    'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
  STORED AS INPUTFORMAT 
    'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' 
  OUTPUTFORMAT 
    'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
  LOCATION
    's3://pinfare-glue/flights/'
  TBLPROPERTIES (
    'CrawlerSchemaDeserializerVersion'='1.0', 
    'CrawlerSchemaSerializerVersion'='1.0', 
    'UPDATED_BY_CRAWLER'='pinfare-parquet', 
    'averageRecordSize'='19', 
    'classification'='parquet', 
    'compressionType'='none', 
    'objectCount'='623609', 
    'recordCount'='4368434222', 
    'sizeKey'='86509997099', 
    'typeOfData'='file')
  </code></pre>