title: AWS Glue Incremental crawl of continually arriving data on S3
body: <p>I'm looking for a way to set-up an incremental Glue crawler for S3 data, where data arrives continuously and is partitioned by the date it was captured (so the S3 paths within the include path contain <em>date=yyyy-mm-dd</em>). My concern is, that if I run the crawler in the course of a day, the partition for it will be created, and will not be re-visited in subsequent crawls. Is there a way to force a given partition, that I know might still be receiving updates, to be crawled while running the crawler incrementally and not wasting resources on historic data?</p>