Enterprise application data typically is stored in one or more databases, such as those in a relational database management system. To perform a full text search of data stored in an enterprise application, data to be crawled by a search engine typically is stored for retrieval at a later point in time. A search engine crawler, i.e., a computer or program that systematically retrieves and indexes data for searching, accesses the stored information to prepare a search index. To do so, stored enterprise application data are extracted from a database and provided to one or more search crawlers for indexing. Depending on the lifecycle of the enterprise application, various types of data access methodologies (“crawls”) may be used. For example, where existing data is present in the enterprise application before it's accessed by the search engine, the entire set of existing data may be crawled into a search engine. This may be referred to as a “full crawl.” Once a full crawl has been performed, “incremental crawls” may be performed to extract only data which has changed since the previous crawl. Different crawl techniques may require different amounts and types of resources from the associated enterprise application. For example, a full crawl may require a large amount of data access, but little processing time since all the data in the application can be quickly provided by a single lookup command. Similarly, an incremental crawl may require more processing resources to identify the relevant data in the application.
Different crawl methodologies also may be used. Third-party and web-based search engines commonly use synchronous crawling techniques. In a synchronous crawl, a search crawler requests information to be stored and indexed for use by the search engine. Typically, the crawler interacts with an application or server containing the information to be indexed using HTTP or a similar protocol. In response to the request, one or more applications containing the requested data process the request and serve the appropriate data to the crawler. In contrast, an asynchronous technique may use a file system to pre-process enterprise application data into a regular file. The search engine crawler then processes data from the regular file instead of directly from the enterprise application as occurs in a synchronous search technique.