  Analyzing the increasingly large volumes of data that are available today,
possibly including the application of custom machine learning models, requires
the utilization of distributed frameworks. This can result in serious
productivity issues for "normal" data scientists. This paper introduces AFrame,
a new scalable data analysis package powered by a Big Data management system
that extends the data scientists' familiar DataFrame operations to efficiently
operate on managed data at scale. AFrame is implemented as a layer on top of
Apache AsterixDB, transparently scaling out the execution of DataFrame
operations and machine learning model invocation through a parallel,
shared-nothing big data management system. AFrame incrementally constructs
SQL++ queries and leverages AsterixDB's semistructured data management
facilities, user-defined function support, and live data ingestion support. In
order to evaluate the proposed approach, this paper also introduces an
extensible micro-benchmark for use in evaluating DataFrame performance in both
single-node and distributed settings via a collection of representative
analytic operations. This paper presents the architecture of AFrame, describes
the underlying capabilities of AsterixDB that efficiently support modern data
analytic operations, and utilizes the proposed benchmark to evaluate and
compare the performance and support for large-scale data analyses provided by
alternative DataFrame libraries.

Databases