text:
\section{Introduction}\label{sec:intro}
Datasets are not set in stone; they have to be refreshed periodically in many areas (e.g., sales, census, or marketing). For example, many demographic data and surveys (leveraged in geodemography\footnote{Geodemography is an area of market research, specializing in profiling economic and demographic characteristics of geographical areas~\cite{goss_1995_geodem}.}) are updated annually by national census organizations
or primary research companies. These data, in turn, get ingested by companies around the globe to improve business decisions. The data ingestion leads to dataset refresh, where the previous version of the dataset is replaced by a newer vintage.
Typically, during the refresh, one needs to apply value added methods like imputation, reconciliations, or reprojections to the raw data before loading these data into a business intelligence system. When working with data, there is always a possibility that the raw data have flaws or the data transformations have unintended consequences, leading to loading incorrect data into production systems.

The consequences of such errors vary. For example, if the transformed dataset does not have a required variable, the software doing data analysis on this transformed data may fail as it would be unable to find the variable. Such an error would be detected fairly early in the testing of software systems. However, an error may be more subtle: all the variables would be present, but the values of these variables are incorrect, leading to incorrect results generated by the business intelligence software.

In one example, the value of the variable would have no physical meaning: a sample report obtained from the new vintage of data may suggest that the average price of a house for area A is $-\$500$K. This value is meaningless and can be easily captured by checking the dataset for negative values.

In another example, a sample report may suggest that the average price of a house for area B is $\$50$K while for area C it is $\$10$M. Both numbers are extreme, but not outside of the realm of possibility. Thus, an analyst may need to manually verify both numbers and must have some context to appropriately assess the resulting numbers. This manual verification is arduous.

If upon verification, the suspicious results are deemed erroneous, then the data team has to fix the defect, recreate the dataset as necessary, and reload it into the production system. This process repeats itself until all the data-related defects are detected and eliminated. Then the dataset is loaded into a production database, and the product is made available to a customer. This process is laborious and time-consuming. A single iteration (from detecting a defect in the data, to fixing and reloading the dataset), based on the authors' experience, may take multiple days, significantly delaying the release of the product to a customer.

Thus, our \textbf{goal} is to detect data-related defects automatically, streamline data refresh schedules, and reduce the cost of detecting and fixing the defects.

\subsection{Existing solutions}\label{sec:existing_solutions}
There exists a significant amount of test frameworks for testing database engines and business logic that alters the data in the databases~\cite{kapfhammer2003family,maule2008impact,haraty2002regression,nanda2011regression,haftmann2005efficient}. In addition, some automated database testing frameworks~\cite{DbFit,DbUnit,NDbUnit,DBTest,SQL} are developed to make sure that the previously captured analytics SQL queries execute successfully in the current version.

The problem we aim to solve is complementary, because in our case  --- once the dataset is loaded into the database --- the data stay constant (due to the nature of the analytics workloads, end-users do not alter the data in databases). To reach our goal, we need to examine the changes that happen in the data preparation step (before the data are loaded into the database) and flag the erroneous records and variables.

Essentially, we are interested in verifying and validating the data, rather than
\begin{enumerate*}
    \item validating business logic~\cite{kapfhammer2003family,maule2008impact,haraty2002regression,nanda2011regression,haftmann2005efficient} of an application that uses the database as persistent storage or
    \item checking if database engine itself performs correctly~\cite{hintz1996method,cialini2007method}.
    \end{enumerate*}
That is, we are interested in
\begin{enumerate*}
    \item detecting the changes in the dataset itself and
    \item deciding if a given change is expected or not.
\end{enumerate*}

\subsection{Contributions}
In this paper, we present an automated REgreSsion Testing tool fOR datasEts abbreviated as RESTORE for data regression testing. Regression testing is a type of testing which ensures that the existing functionality of a software product is not broken with new changes, i.e., the functionality does not regress~\cite{huizinga2007automated,lewis2000software}. In our case the functionality is data-centric.

The tool works by comparing the previous vintage of a dataset with a new one and reporting potential issues. The tool leverages statistical analysis and offers a set of comprehensive testing rules for dataset testing. The tool is written in R language~\cite{r_core} (popular in the data science and statistical communities) and is released as an open-source R package on GitHub~\cite{restore_repo}.

We also provide a validation case study based on a sample hierarchical  (i.e., tree-like) geodemographic dataset. The study shows that RESTORE can efficiently speed up the testing procedure and reduce the cost of testing. The usefulness of RESTORE is also supported by results of an anonymous survey of 15 data scientists who are now using the tool.
RESTORE exhibits the following characteristics.

\begin{enumerate}
    \item \textbf{Efficiency}: Reduces the amount of time and effort required to create a new vintage of dataset, by automatically detecting data discrepancies before the dataset is loaded into a database of a business intelligence system.
    \item \textbf{Variability}: Supports both flat and hierarchical  data structures.
    \item \textbf{Scalability}: Can process medium size datasets (tested on datasets comprised of 600+ variables with $\approx$~1.5 million observations)
    and can be scaled up to larger datasets.
    \item \textbf{Simplicity}: Encapsulates a batch of relatively complex testing rules into a simple single-function interface. One can easily use it by providing two datasets (read either from R data frame or from an external file) and a small amount of metadata (see Section~\ref{sec:interface} for details).
    \item \textbf{Flexibility}: Allows to add new tests and alter the existing ones.
\end{enumerate}

The rest of the paper is structured as follows. Section~\ref{sec:regression} introduces background information and our proposed method. Section~\ref{sec:tests} presents the regression tests. Section~\ref{sec:restore} discusses the interface of RESTORE. Section~\ref{sec:related} depicts related work. Finally, Section~\ref{sec:conclusion} concludes the paper.

\section{Regression testing for datasets}\label{sec:regression}
In this section, we first discuss how the geodemographic data are processed and tested in Section~\ref{sec:background}; then we present details of our method in Section~\ref{sec:method}.

\subsection{Background}\label{sec:background}
Data accuracy, integrity, and quality are becoming more and more crucial to data analytic solutions~\cite{delen2013}. However, it is challenging to verify and validate modern datasets because of the large amount and diversity of data. Thus, it is increasingly critical to develop an efficient and effective solution for data testing to assure the accuracy, integrity, and quality of the data.

It is important to understand how the data are developed and tested in a typical data analytics / data science ``shop''. We provide a concrete example to illustrate a typical procedure of data development and testing, based on the process adopted by Environics Analytics, Toronto, Canada (abbreviated to EA in this paper). EA specializes in geodemography and marketing analytics and builds standard and custom data-driven solutions for their clients. Many of their data and services are provided using Software-as-a-Service (SaaS) approach~\cite{mell_2011_nist}. These services require datasets to be refreshed multiple times per year. During the refresh, anomalies in new vintages of datasets may introduce defects in services. The quality assurance team used to manually test the datasets to detect and eliminate the defects, but this process was time- and human-resource-consuming.

EA's SaaS platform hosts data built and maintained by EA, as well as data supplied by EA's partners or clients. Data supplied by partners or clients can be in a variety of formats. Their most common data structure is a tabular dataset rolled out to some or all levels of geography. This data architecture has its advantages because it allows for the necessary flexibility to work with different types and sizes of data. In this paper, we assume that all datasets are in tabular format (or can be converted to this format).

Before an automated data testing tool is adopted, the dataset development process in EA is as follows.

\begin{enumerate}
    \item The data team creates a dataset (either a new one or a refresh of an existing one).
    \item This dataset is loaded into a staging database by the data team.
    \item The software development and quality assurance / testing teams execute a mixture of automated and manual test workloads (against the application) mimicking customers' behaviour (e.g., select a particular geographic area and then run house prices report). Under the hood, the software layer issues analytic (read-only) queries to the staging database. As part of the software testing, data in analytics reports are assessed, resulting in possible data errors to be uncovered.
    \item If failures (such as the ones discussed in Section~\ref{sec:intro}) in the analytics reports are observed during the execution of the workloads, a bug report is issued for further investigation. Data bugs may exist in a variety of forms: e.g., errors in raw data, errors in calculation of ``constructed'' variables as part of the load into the staging database, and errors generated by how the application handles the data.
\end{enumerate}

Once the data team fixes the defects, this team recreates the dataset as necessary, reloads it into the staging database, and hands it over to the software development team for testing (basically, rerunning the above process). This process repeats itself until all the data-related defects are eliminated. Then the dataset is loaded into a production database, and the product is made available to a customer.

As mentioned in Section~\ref{sec:intro}, the process is time-consuming and may significantly delay the release (from days to weeks) of the product to customers.

\subsection{Our method}\label{sec:method}
Our goal is to detect data defects as early as possible within the development process with a minimal amount of effort so that we can reduce processing time, thereby creating a more optimized and efficient workflow. Namely, we strive to detect anomalies in datasets in the very first step of the process described in Section~\ref{sec:background}.

We propose to detect the erroneous data early (hence reduction of rework) by introducing a novel approach for automated regression testing of the data. To the best of our knowledge, such a data regression testing framework does not exist. This regression testing will help to detect problems automatically (reducing the amount of manual data testing) and early in the development process (reducing the likelihood of data or metadata defects loaded into the staging database). Such early automated detection of the defects would free resources to focus on more complex workloads and scenarios, thus,

\begin{enumerate}
    \item Improving overall product quality (as teams will have more time and resources to identify complex defects that otherwise would be ``masked'' by simpler defects, which can be caught by the automated regression testing~\cite{vanMegen_1995,rafi2012}), and
    \item Reducing development costs (the savings will manifest themselves because the cost of creating, maintaining, and executing test cases will be lower than the cost of manual testing).
\end{enumerate}

\begin{figure}[t]
    \centering
    \includegraphics[width=\columnwidth]{structure.png}
    \caption{Comparison of old and new vintages of the dataset.}
    \label{fig:structure}
\end{figure}

These steps of automated data regression testing are graphically depicted in Figure~\ref{fig:structure}, which shows the following process (discussed in details in Sections~\ref{sec:tests} and \ref{sec:restore}).

\begin{enumerate}
    \item Load both old and new vintages of the dataset into RESTORE.
    \item Apply a set of test to verify and validate the integrity of the new vintage.
    \item Generate the final report which is exported in a human- and machine-readable formats.
\end{enumerate}

Let us now look at the details of the tests used to compare the vintages.

\section{Tests' description}\label{sec:tests}
There exist automated database testing frameworks~\cite{DbFit,DbUnit,NDbUnit,DBTest,SQL} to make sure that the analytic statement (captured in one of the previous releases) executes successfully (in the release under test). However, these will typically be inapplicable to our case. As discussed in Section~\ref{sec:existing_solutions}, the new vintage of a dataset differs from the old one by construction in our case. Thus, the recordsets generated by the analytics database queries will differ from the old vintage to the new vintage.

To address our problem (i.e., regression testing of the modified dataset), a set of tests performs an approximate comparison (rather than exact comparison as done by~\cite{DbFit}) of the datasets. Based on the discussions with EA data scientists, we create ten groups of tests, which will be discussed in details below. The data scientists found these tests to be helpful in practice, i.e., the tests were able to reliably detect defects in the data. The tests are also computationally inexpensive\footnote{We provide details of performance measurements in Section~\ref{sec:validation}. In a nutshell, RESTORE can compare two vintages of a dataset containing $>500$ variables and $>60,000$ observations in less than four minutes on commodity hardware.}, which helps to preserve scalability and enable fast verification of changes to a dataset (in a test-driven-development manner~\cite{beck2003test}, where a regression test suite can be executed quickly to make sure that no new errors were injected with the latest changes).

We also need to define success criteria for these tests. The criteria are defined based on the practical experience of EA data scientists who find that these values provide a large number of true data defects while keeping the number of false defects low. We cannot guarantee that these values are optimal for any dataset, rather they can be treated as a set of good starting values and adjusted based on a particular use-case and the needs of a data tester.

Below we give details of our tests grouped into three categories: high-level tests dealing with metadata, tests of paired observations, and tests leveraging the results of the paired tests (which we deem higher-order tests). These tests are discussed in Sections \ref{sec:np-test}, \ref{sec:p-test}, and \ref{sec:h-test}, respectively. Finally, we discuss the power, usage, and limitations of these tests in Section~\ref{sec:discussion}.

\subsection{High-level testing of vintages}\label{sec:np-test}
An example of a dataset vintage with $N$ variables\footnote{The term variable is synonymous to column or feature, depending on the reader's background.} and $M$ observations is given in Table~\ref{tab:vintage_ex}. Note that the `Key' values are not necessarily numeric. The only constraint is that the 2-tuple of `Key' and `Hierarchy Level' should be unique for every row (i.e., observation).

\begin{table}
\caption{Example of a dataset vintage; $v_{\cdot}$ is a variable name. }
\label{tab:vintage_ex}
\begin{tabular}{r|l|r|r|r}
\toprule
Key      & Hierarchy Level & $v_1$    & $\ldots$ & $v_N$    \\ \midrule
1        & National        & 100      & $\ldots$ & 500      \\
2        & City            & 3        & $\ldots$ & 16       \\
$\ldots$ & $\ldots$        & $\ldots$ & $\ldots$ & $\ldots$ \\
$M$      & National        & 12       & $\ldots$ & 124      \\ \bottomrule
\end{tabular}
\end{table}

We now perform three groups of high-level tests assessing the characteristics of the vintages as follows:
\begin{enumerate*}
    \item comparing attributes of the variables,
    \item checking the variables for missing observations, and
    \item counting discarded observations.
\end{enumerate*}

\subsubsection{Variables' attributes comparison}\label{sec:diff-test}
\underline{Rationale}: We perform a set of ``sanity-check'' tests, comparing high-level characteristics of the datasets (i.e., metadata), such as the number of rows and columns. If the numbers do not match, this may be a cause for concern.

\underline{Method}: We obtain three items for each of the vintages, old and new: the number of variables, the names of variables, and the number of observations in the dataset. We then compare these items.

\underline{Success criteria}: If the number of variables, the names of the variables, and the number of observations are the same, then the test passes. If the number of variables or the number of observations are not identical between the old vintage and the new vintage of a dataset, the test fails and this mismatch gets reported. If a variable, present in the old vintage, is not present in the new vintage (or vice versa), then it is also considered a failure and the name of the variable is added to the report.

\subsubsection{Missing (NA) observations}\label{sec:na-test}
\underline{Rationale}: Typically, a clean dataset should not have missing observations in a variable.

\underline{Method}: Thus, for each `Variable Name' (e.g., for each $v_i$ in $v_1, \ldots, v_N$ in Table~\ref{tab:vintage_ex}) and `Hierarchy Level', we search for missing observations (in R such observations are marked as NA). This process is done individually for old and new vintages of the dataset.

\underline{Success criteria}: A `Variable Name' and `Hierarchy Level' pair that has zero missing observations passes the test; otherwise, it gets reported.

\subsubsection{Discarded observation count}\label{sec:disc_cnt}

Now we can join the old and new vintages of the dataset, so that we can perform pairwise tests for each variable (as will be discussed in Section~\ref{sec:p-test}). We perform the inner join (in the relational algebra sense of the term~\cite{maier_1983}) on the `Key' and `Hierarchy Level' columns (shown in Table~\ref{tab:vintage_ex}) of the old and new vintages, discarding the observations that are present in only one of the vintages. Before moving to the pairwise tests, we will perform one last metadata test, based on the count of discarded observations.

\underline{Rationale}:  EA data scientists found out that paired observations are more valuable for detecting defects than the non-paired ones (as they contain more information about changes to the dataset). However, the observations that did not make it into the inner join of the old and new vintage may indicate a defect in the data preparation process.

\underline{Method}: Count the number of observation present in the old and absent in the new vintage, deemed $c_1$, as well as the number of observation present in the new and absent in the old vintage, deemed $c_2$. Note that we already compared the count of observations of the vintages in Section~\ref{sec:diff-test}. However, in this section we pair the observations, which brings additional information. If we denote  the count of observations in the old vintage as $c_o$, in the new vintage --- as $c_n$, and in the join of old and new vintage as $c_u$, then $c_1 = c_o - c_u$ and  $c_2 = c_n - c_u$. If all the observations are paired via the inner join, then $c_o = c_n = c_u$.

\underline{Success criteria}: The test passes if $c_1 = 0$ and $c_2 =0$; otherwise the test fails, and the values of $c_1$ and $c_2$ are reported.  A tester can then assess if the discarded observations appeared in the data are due to normal data churn or because of a defect in data preparation.

\subsection{Paired testing}\label{sec:p-test}
Once we join the old and new vintage (using the approach discussed in Section~\ref{sec:disc_cnt}), we can conduct the following sets of pairwise tests (performing comparisons for a given variable and hierarchy level):
\begin{enumerate*}
    \item the magnitude ratio test,
    \item the mean relative error test,
    \item the correlation test, and
    \item the distribution test.
\end{enumerate*}
These tests are discussed below.

\subsubsection{Magnitude ratios}\label{sec:magn}
\underline{Rationale}: We compare the magnitudes for the minimum, maximum, sum, mean, and median values for each level of hierarchy between the old and new vintages. The expectation is that extreme points of the distribution, as well as the central points, pairwise, should be in the same ballpark, which we will assess by comparing the order of magnitudes.

\underline{Method}: Let us denote a metric for the $i$-th variable and $j$-th hierarchy level of an old vintage as $m_{i,j,o}$ and for $i$-th variable and $j$-th hierarchy level of a new vintage as $m_{i,j,n}$, respectively. Then the magnitude ratio $R_{i,j}$ is computed as follows:

\begin{equation}
	R_{i,j} =
	\begin{cases}
		1, &\text{if $m_{i,j,o} = 0$ and $m_{i,j,n} = 0$;} \\
		\text{undefined}, &\text{if $m_{i,j,o} = 0$ or $m_{i,j,n} = 0$;} \\
		m_{i,j,o} / m_{i,j,n}, &\text{otherwise.}
	\end{cases}
\end{equation}

\underline{Success criteria}: We compute the value of $R_{i,j}$ for each pair of the metrics (min of the old and new vintage, max of the old and new vintage, etc.). If $0.1 < R_{i,j} < 10$ then both values are of the same magnitude and the test succeeds, otherwise --- fails and gets reported. Note that we have two special cases. If $m_{i,j,o} = 0$ and $m_{i,j,n} = 0$, then we assume that the magnitudes are identical --- setting $R_{i,j}=1$. If $m_{i,j,o} = 0$ or $m_{i,j,n} = 0$, then we cannot credibly assess magnitude difference; in this case we emit a warning asking an analyst to assess the magnitude difference manually.

Note that given the pairwise nature of the comparison, the ratios of sums and averages will yield identical results. However, we retain both for a practical reason: the sums help an analyst to compare the values of variables at different levels of hierarchies (as, typically, the sum of observations at a lower hierarchy level aggregate to the value at a higher level of the hierarchy) hence the decision to keep the sum values.

\subsubsection{Mean relative error}\label{sec:mre}
\underline{Rationale}: The previous test (comparing min, max, etc.) assesses statistics that discard information about pairwise relations of individual observations. Given that we pair observations in the old and new vintage, we can compare each observation using mean relative error. We prefer the mean relative error over the mean absolute error because the values of attributes vary significantly between the variables as well as the variables' hierarchy levels.

\underline{Method}: Let us pair old and new observations for the $i$-th variable and denote paired vector of observations for the $i$-th variable and $j$-th hierarchy level of old vintage as $x_{i,j,o}$ and for the new vintage as $x_{i,j,n}$. Then the mean relative error $E_{i,j}$ is computed as an average of relative errors of each pair of observations in $x_{i,j,o}$ and $x_{i,j,n}$:
\begin{equation}
E_{i,j} = \langle \vert (x_{i,j,o} - x_{i,j,n}) \oslash x_{i,j,o} \vert \rangle,
\end{equation}
for all non-zero elements of $x_{i,j,o}$, where $\oslash$ is the Hadamard division operator (performing element-wise division of vectors) and $\langle \cdot \rangle$ computes the mean.

By construction, all pairs of observation, where an element from $x_{i,j,o}$ is equal to $0$, have to be ignored. If $x_{i,j,o}$ vector has a lot of zero values, then this test may become misleading. In this case one can implement another test of relative change, see~\cite{tornqvist1985should} for review and comparison of such tests.

\underline{Success criteria}: The test considered successful if $E_{i,j} < 0.2$.  A smaller value of the threshold can generate a high number of false alarms based on the feedback of practitioners from EA.

\subsubsection{Correlation test}\label{sec:cor}
\underline{Rationale}: We expect that there should be a strong mutual relation between the observations of a given variable in the old and new vintages. To measure the strength of this relation, we compute correlations between the values of a given variable in the old and new vintages. The relation does not necessarily have to be linear but it should be monotonic. Thus, to assess these properties, we use Pearson product-moment correlation coefficient~\cite{pearson1896mathematical, myers_research_2010} (to assess linearity) and Spearman rank-order correlation~\cite{spearman1904proof, myers_research_2010}  (to assess monotonicity).

\underline{Method}: We compute Pearson and Spearman correlations coefficients (deemed $r_{i,j}$ and $\rho_{i,j}$, respectively) for pairs of $x_{i,j,o}$ and $x_{i,j,n}$ for each variable $i$ and hierarchy level $j$. Correlation values range between $-1$ and $1$, with $1$ being perfect correlation, $-1$ --- perfect anticorrelation, and $0$ --- no correlation.

\underline{Success criteria}: The test is considered successful if $r_{i,j} \geq 0.8$ and $\rho_{i,j} \geq 0.8$, and unsuccessful otherwise. From a practical perspective, a lot of real-world variables exhibit nonlinear relations (plus Pearson correlation assumes data normality which is often not the case). Thus, EA data scientists pay more attention to the case of $\rho_{i,j} < 0.8$ than to the case of $r_{i,j} < 0.8$, because (empirically) they observed that it is a stronger indicator of a defect in the data.

\subsubsection{Distribution test}\label{sec:dist}
\underline{Rationale}: The previous test assesses the correlation between the $i$-th variable of the old and the new vintages. In this test, we generalize this approach by comparing distributions of the old and new vintages of this variable. If the distributions are significantly different, then it may be an indicator that there is a defect in the data.

\underline{Method}: We use the nonparametric two-sample Kolmogorov-Smirnov test~\cite{smirnov1939estimation} to compare the differences between the two distributions. The null hypothesis of the test is that the samples are drawn from the same distribution.

\underline{Success criteria}: The value of the Kolmogorov-Smirnov test $p$-value for the $i$-th variable and $j$-th hierarchy level is denoted by $S_{i,j}$. If $S_{i,j} < 0.05$, we assume that the null hypothesis is rejected and declare test failure. If $S_{i,j} \geq 0.05$ --- the test succeeds (even though it does not imply that the distributions are not different).

\subsection{Higher-order testing}\label{sec:h-test}
The set of higher-order tests (i.e., those that combine the values of the metrics computed in Section~\ref{sec:p-test}) is composed of the following:
\begin{enumerate*}
    \item the comparison of Spearman correlation coefficients for different levels of hierarchy,
    \item hybrid test, and
    \item ranking of the number of test failures.
\end{enumerate*}
The details of the tests are given below.

\subsubsection{Comparison of Spearman correlation for different levels of hierarchy}\label{sec:pair-cor}
\underline{Rationale}: In Section~\ref{sec:cor}, we computed Spearman correlation $\rho_{i,j}$ for $i$-th variable and $j$-th level of hierarchy. EA data scientists observed that a significant difference in the $\rho$ values for two adjacent levels of hierarchy (i.e., $\rho_{i,j}$ and $\rho_{i,j+1}$) may indicate a defect in the data of the $i$-th variable. The root cause of such defect often relates to different aggregation procedures (from the raw data) associated with different levels of hierarchy.

Note that while we compute both Pearson and Spearman correlations in Section~\ref{sec:cor}, the comparison test focuses only on the latter. As we discussed in Section~\ref{sec:cor}, the $\rho_{i,j} < 0.8$ (Spearman correlation) is a stronger indicator of a defect in the data than $r_{i,j} < 0.8$ (Pearson correlation). Analogously, it was found that comparison of differences in $\rho_{i,j}$ is a better indicator of a defect than a comparison of differences in $r_{i,j}$. Thus, to reduce tester's information overload, it was decided not to include the comparison of $r_{i,j}$ in the report.

\underline{Method}: We compute relative difference $C_{i,j}$ between two adjacent levels of hierarchy:
\begin{equation}
	C_{i,j} = (\rho_{i,j} - \rho_{i,j+1}) / \rho_{i,j}, \text {if $\rho_{i,j} \neq 0$. }
\end{equation}
Given $J$ levels of hierarchy, with the $1$-st level being the top one and the $J$-th level being the bottom one, we perform $J-1$ computations of $C_{i,j}$, with $j = 1, \ldots, J-1$.

\underline{Success criteria}: Based on the experience of EA data scientists, $ -0.1 < C_{i,j} < 0.1 $ is considered acceptable. $C_{i,j}$ values outside of this range may indicate a problem with the data of the $i$-th variable and $j$-th or `$j+1$'-th levels of the hierarchy.

\subsubsection{Hybrid testing}\label{sec:hybrid-test}
\underline{Rationale}: We described multiple tests in the sections above. Intuitively, the higher the number of tests that failed for a given variable and hierarchy level $x_{i,j,n}$ --- the higher the chances that there is something wrong with the observations of this variable. EA data scientists observed that a simultaneous failure of four tests --- namely, mean relative error (Section~\ref{sec:mre}), Spearman and Pearson correlations (Section~\ref{sec:cor}), and Kolmogorov-Smirnov test (Section~\ref{sec:dist}) --- is a very strong indicator of a defect in the underlying data. Thus, if $x_{i,j,n}$ fails all those tests, it should attract the attention of the data team.

\underline{Method}: We identify all the variables that failed four above-mentioned tests simultaneously and report them along with the values of the associated metrics. Table~\ref{tbl:mult_failure} displays an example of this report.

\begin{table}[t]
\caption{Example: results obtained from the hybrid test. }
\label{tbl:mult_failure}
\begin{tabular}{l|l|r|r|r|r}
\toprule
Variable & Hierarchy & $E_{i,j}$ & $r_{i,j}$ & $\rho_{i,j}$ & $S_{i,j}$  \\
Name ($i$) & Level ($j$) &  &  &  &  \\ \midrule
$v_5$         & National        & 0.578               & 0.401               & -0.278                & 0.001 \\
$v_5$         & City            & 0.617               & 0.672               & 0.693                & 0.002                           \\
$v_8$         & City            & 0.669               & 0.532               & 0.454                & 0.046                           \\
\bottomrule
\end{tabular}
\end{table}

\underline{Success criteria}: As shown in Table~\ref{tbl:mult_failure}, a variable's name and corresponding hierarchies are listed in the report if and only if all of the following criteria are satisfied:
\begin{enumerate*}
    \item $E_{i,j} \geq 0.2$,
    \item $r_{i,j} < 0.8$,
    \item $\rho_{i,j} < 0.8$, and
    \item $S_{i,j}< 0.05$.
\end{enumerate*}

\subsubsection{Ranking of the number of test failures}\label{sec:ranking-test}
\underline{Rationale}: All of the above metrics are computed for each variable and hierarchy level individually. EA data scientists observed that a test failure at multiple levels of the hierarchy of a given variable acts as a reliable indicator of a defect in the data associated with this variable.

\underline{Method}: Thus, it is useful to count the number of test failure for each variable and test type and then order them in descending order from the highest number of test failures to the lowest. To reduce clutter, we report only the variables that have at least one test failure associated with them. Example of such ranking is given in Table~\ref{tab:tab2}.

\underline{Success criteria}: An ultimate success is when there are no test failures associated with a variable and this variable does not show up in the report. The higher the number of tests and types of tests that failed --- the higher the chances that a variable has a defect in its data.

\begin{table}
\caption{Example: raking test. To preserve space, a subset of tests is shown in this example.}
\label{tab:tab2}
\begin{tabular}{l|r|r|r|r|r}
\toprule
Variable Name ($i$) & $E_{i, \text{all}}$ & $r_{i, \text{all}}$ & $\rho_{i, \text{all}}$ & $S_{i, \text{all}}$ & Total  \\ \midrule
$v_7$    & 4   & 6        & 6       & 6            & 22  \\
$v_3$    & 4   & 5        & 2       & 5            & 16  \\ \bottomrule
\end{tabular}
\end{table}

\subsection{Discussion}\label{sec:discussion}
\subsubsection{Root causes of test cases' failures}
As mentioned at the beginning of Section~\ref{sec:tests}, not every test failure will lead to exposure of a data defect. Instead, a failure suggests that a new vintage is different from the old one in some unexpected way, and that a tester should take a close look at the failure.

For the tests operating at a particular hierarchy level (i.e., those discussed in Sections~\ref{sec:na-test},~\ref{sec:p-test}, and~\ref{sec:hybrid-test}), a good starting point of an investigation is a review of data transformation procedures for a particular variable and level of hierarchy for which the test case failed. In the case of the test discussed in Section~\ref{sec:pair-cor} (examining adjacent levels of hierarchy), the problem typically is associated with data transformation procedures for one of these levels. The root cause of a failure of the test described in Section~\ref{sec:ranking-test} often resides in the general procedure that touches multiple levels of the hierarchy of the variable under investigation.

The tests discussed in Section~\ref{sec:diff-test} operate at an even lower level of granularity (as they deal with potentially missing variables or observations). While removal or addition of variables is not uncommon, sometimes an analyst renames a variable by mistake, which often ends up being the root cause for the variable to appear in the report of this test. If the datasets have significantly different number of observations, it may be caused by datasets truncation or data corruption. A failure of the final metadata-related test, discussed in Section \ref{sec:disc_cnt}, may indicate corruption of the values in the `Key' or `Hierarchy Level' columns.

\subsubsection{Predictive power of tests}
As discussed above, not every failure of a test ``translates'' into an actual defect. However, anecdotally, EA data scientists observed that higher-order tests described in Sections~\ref{sec:hybrid-test} and~\ref{sec:ranking-test} yield the lowest number of false alerts, followed by the correlation-related tests in Sections~\ref{sec:cor} and~\ref{sec:pair-cor}.

On the other side of the spectrum, the distributions comparison test discussed in Section~\ref{sec:dist} yields the highest number of false alarms. This is expected, as the underlying distributions for a large number of variables in periodically refreshed datasets experience legitimate change to their underlying distributions (which the test detects successfully). However, the list of such variables are typically known to the dataset curators and, thus, can be filtered out with relative ease during the analysis of the report (generated by RESTORE).

The rest of the test fall in the middle of the spectrum. For example, the change to a distribution also translates into changes to statistics (such as mean, min, and max), which we analyze in Section~\ref{sec:magn}. However, because we are comparing the magnitudes of these statistics, these tests are less prone to false alarms.

\subsubsection{Data types}\label{sec:data-type}
All of the tests can process variables to which ratio and, arguably, interval scales~\cite{stevens1946theory} can be applied.

We will also be able to compute the test for numeric variables measured on nominal or ordinal scales~\cite{stevens1946theory}, but the results of some of these tests (e.g., magnitude comparison of averages for the ordinal scale) would be questionable from the statistical perspective. Thus, one has to be careful when interpreting the results of the tests.

The test cannot be computed for non-numeric tests, except for the tests discussed in Section~\ref{sec:np-test}.

Fortunately, curators of datasets typically know data types and measuring scales of the variables in the datasets and can recommend which variables should be excluded from the analysis.

\section{Introduction to the RESTORE package}\label{sec:restore}
In this section, we introduce the interface of the RESTORE package in Section~\ref{sec:interface}.
Then, we  assess RESTORE usefulness based on user's feedback in Section~\ref{sec:validation}. Finally, we discuss potential extensions of RESTORE in Section~\ref{sec:potential}.

\subsection{The interface of RESTORE}\label{sec:interface}
We implement the set of tests discussed in Section~\ref{sec:tests} in an open-source R package, available at~\cite{restore_repo}. The installation of the package follows a standard installation process for R package, details are given in the README file of~\cite{restore_repo}.

The tests are controlled by a single function \code{test\_two\_datasets}. The function ingests old and new vintages of the dataset as well as specification of the hierarchy either from CSV files or from R data frames.

We found that for interactive testing, when a tester adjusted the datasets and wanted to quickly assess the results, the CSV files were more convenient. On the contrary, for automated testing, when the datasets were tested as part of the automated regression test harnesses, the data frame option was more suitable.

The final report is written into a user-specified XLSX file or saved as R data structure (so that it can be easily parsed later, if necessary). Users can select the test which should be stored in the final report. Parameters of the \code{test\_two\_datasets} function  are as follows.

\begin{comment}
\begin{lstlisting}
test_two_datasets (legacy_file = NULL,
                  legacy_df = NULL,
                  target_file = NULL,
                  target_df = NULL,
                  hier = NULL,
                  hier_df = NULL,
                  hier_pair = NULL,
                  hier_pair_df = NULL,
                  thresholds = NULL,
                  thresholds_df = NULL,
                  final_report,
                  final_data,
                  key_col,
                  hier_col,
                  report_join = TRUE,
                  report_hybrid = TRUE,
                  report_mre = TRUE,
                  report_spearman = TRUE,
                  report_pearson = TRUE,
                  report_distribution = TRUE,
                  report_appear = TRUE,
                  report_spearman_diff = TRUE,
                  report_na = TRUE)
\end{lstlisting}
\end{comment}

The parameters \code{legacy\_file} and \code{target\_file} set the path to the files that contain the old vintage of the dataset and the new vintage of the dataset, respectively.

The parameters \code{hier\_pair} sets the path to a CSV file containing 2-tuples `Parent Hierarchy Level' and `Child Hierarchy Level', this enables RESTORE to operate on non-linear hierarchies. For example, a tree depicted in Figure~\ref{fig:tree} will be encoded by 2-tuples shown in Figure~\ref{tab:hierarchy_sample}.

\begin{figure}
  \centering
  \subfloat[]{\label{fig:tree}
  \includegraphics[width=0.2\columnwidth]{tree.png}}%
  \enskip
  \subfloat[]{%
    \begin{tabular}[b]{l|l}
    \toprule
    Parent & Child \\ Hierarchy & Hierarchy \\ Level & Level \\
    \midrule
    A & B \\
    A & C \\
    B & D \\
    C & D \\
    \bottomrule
    \end{tabular}
    \label{tab:hierarchy_sample}
  }
  \quad
  \subfloat[]{%
    \begin{tabular}[b]{r|l}
    \toprule
    Rank & Hierarchy \\ & Level \\
    & \\
    \midrule
    1     & A               \\
    2     & B               \\
    3     & D               \\
    4     & C               \\
    \bottomrule
    \end{tabular}
    \label{tab:hierarchy_ranking_sample}
  }
  \caption{An example of hierarchy configuration. (a) Example of a non-linear hierarchy. (b) 2-tuple encoding of the non-linear hierarchy from Figure~\ref{fig:tree}. (c) Sample ranking of the hierarchy levels in Figure~\ref{fig:tree}.}
  \label{fig:figure-and-table}
\end{figure}

The parameter \code{hier} points to a CSV file containing an ordered list of hierarchy levels, which is used for sorting the test results in the reports containing hierarchy column (e.g., the one shown in Table~\ref{tbl:mult_failure}), see Figure~\ref{tab:hierarchy_ranking_sample} for an example of such file. Note that this parameter is not used to define the actual hierarchy.

The parameter \code{thresholds} points to a CSV file containing values for success criteria of tests described in Section~\ref{sec:tests}.

The variables described above have corresponding ``twin'' parameters (namely,   \code{legacy\_df}, \code{target\_df}, \code{hier\_pair\_df}, \code{hier\_df}, and \code{thresholds\_df}) which allow to pass the dataset and configuration files in the R data frame format.

\begin{comment}
\begin{table}[t]
\caption{The pairs of thresholds' names and values.}
\label{tab:tab5}
\begin{tabularx}{\columnwidth}{X|R|R|R|R}
\toprule
Thresholds & Significance & MRE & Correlation & Spearman Diff. \\ \midrule
Values     & 0.05         & 0.2 & 0.8         & 0.1 \\
\bottomrule
\end{tabularx}
\end{table}
\end{comment}

The \code{final\_report} parameter specifies the location of the output report file in XLSX format. The parameter \code{final\_data} specifies an output location for the report stored in the R data structure format. The rest of the parameters are used to determine important feature names and a list of tests to run, as summarized in Table~\ref{tbl:params}.

While RESTORE reads data only from CSV files or data frames, it does not imply that we cannot leverage other data formats. We simply need to convert our data into one of these two formats. For example, if the data resides in a relational database, one can issue SELECT SQL query from R using \code{DBI}~\cite{dbi} package, which will automatically extract ad convert the data into the R data frame format.

As part of the package, we provide a sample file demonstrating the usage of RESTORE program interface (see \code{example.R} in~\cite{restore_repo}).

\begin{table*}[!th]
\caption{Additional parameters of \code{test\_two\_datasets}.}
\resizebox{\linewidth}{!}{
\begin{tabular}{l|l|l|l}
\toprule
Parameter & Type & Details & Default Value
\\ \midrule
\code{key\_col} & String & Name of the column containing `Keys'. & NA \\ \hline
\code{hier\_col} & String & Name of the column containing `Hierarchy Levels'. & NA \\ \hline
\code{report\_char} & Boolean & TRUE --- generate the report of characteristic comparison; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_na} & Boolean & TRUE --- generate the report of missing (NA) variables; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_discard} & Boolean & TRUE --- generate the report of discarded observations; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_magnitude} & Boolean & TRUE --- generate the report of magnitude metrics; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_mre} & Boolean & TRUE --- generate the report of mean relative errors; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_spearman} & Boolean & TRUE --- generate the report of Spearman test; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_pearson} & Boolean & TRUE --- generate the report of Pearson test; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_distribution} & Boolean & TRUE --- generate the report of distribution test; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_spearman\_diff} & Boolean & TRUE --- generate the report of Spearman comparison for hierarchies; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_hybrid} & Boolean & TRUE --- generate the report of hybrid metrics; FALSE --- do not generate the report. & TRUE \\ \hline
\code{report\_ranking} & Boolean & TRUE --- generate the report of ranking; FALSE --- do not generate the report. & TRUE \\ \bottomrule
\end{tabular}}
\label{tbl:params}
\end{table*}

\subsubsection{Special case: flat hierarchy}\label{sec:special}
To deal with the case of a flat hierarchy (i.e., non-hierarchical dataset), we do not need to pass \code{hier\_pair} and \code{hier\_pair\_df} values to \code{test\_two\_datasets}. Under the hood, RESTORE adds a dummy hierarchy column to the dataset and runs all the tests against the dataset except for the test comparing correlation coefficients for different values of hierarchy (discussed in Section~\ref{sec:pair-cor}).

\subsection{Validation}\label{sec:validation}
The RESTORE R package has been institutionalized into EA's product development cycle. The data scientists use the package to detect defects in the new vintage of the datasets. To assess the benefits of the package, we seek an answer to the following two questions:

\begin{enumerate}
    \item Does RESTORE package make the testing procedure more efficient?
    \item Does RESTORE package decrease the resource cost of dataset testing?
\end{enumerate}
The first question is discussed in Section~\ref{sec:h1}, the second one ---  in Section~\ref{sec:h2}.

\subsubsection{Does RESTORE package make the testing procedure more efficient?}\label{sec:h1}

First, we quantified the time needed to run all the tests on two reference geodemographic datasets (named $D_1$ and $D_2$). The summary statistics for these datasets are shown in Table~\ref{tbl:perf}. The table also shows the average and the standard deviation of the execution time of \code{test\_two\_datasets} function based on 10 runs of the function for each dataset. We kept the parameter values of the function to defaults, i.e., all of the reports were generated.

Our testbed is a laptop equipped with 2 GHz Intel Core i5 CPU and 16 GB memory, running R v.3.5.1 on MacOS v.10.14.3. The datasets are read from files (which is slower than reading the datasets from R data frames). Executing a complete set of tests and generating the final report took, on average, $\approx 3.4$ minutes for the $D_1$ and $\approx 2.3$ minutes for the $D_2$.

Based on the feedback from EA data scientists, the same set of tests, when conducted manually by an experience data tester takes $\approx 2$ hours of the tester's time (per dataset). Thus, using RESTORE speeds up\footnote{Note that we do not take into account the analysis of the test results. However, this time would be identical for both manual-based and RESTORE-based workflows.} this testing process by $\approx 97\%$.

To further understand the benefit of RESTORE, an anonymous poll was sent to 15 EA data scientists with the following three questions.

\begin{enumerate}
    \item Is RESTORE helpful? Possible answers were ``Extremely useful'', ``Very useful'', ``Somewhat useful'', ``Not so useful'', and ``Not at all useful''.
    \item Does RESTORE save time? Possible answers were ``Yes'' or ``No''.
    \item Does RESTORE identify errors? Possible answers were ``Yes'' or ``No''.
\end{enumerate}

One respondent ($\approx 7\%$ of respondents) found RESTORE extremely useful, eleven respondents ($\approx 73\%$ of respondents) --- very useful, and three respondents ($20\%$ of respondents) --- somewhat useful. All fifteen respondents unanimously agreed that RESTORE saves time and identifies errors. Thus, we can conclude that RESTORE is ``very useful'' (based on the median value of the answers to Q1), as it saves time (Q2) and effectively identifies errors in datasets (Q3).

In combination, these results provide an \textbf{affirmative} answer to the first question, suggesting increased efficiency in the identification and remediation of errors in refreshed datasets.

\begin{table}[t]
\caption{Performance evaluation of RESTORE.}
\label{tbl:perf}
\begin{tabular}{l|r|r|r|r|r|l}
\toprule
Dataset & \multicolumn{1}{c|}{Hierar.} & \multicolumn{2}{c|}{Variables} & \multicolumn{2}{c|}{Observations} & \multicolumn{1}{c}{Time}         \\
\multicolumn{1}{c|}{Name}                       & \multicolumn{1}{c|}{Levels}  & \multicolumn{2}{c|}{Count}     & \multicolumn{2}{c|}{Count}        & \multicolumn{1}{c}{$\pm$ St. Dev.}   \\
\cmidrule{3-4} \cmidrule{5-6}
                      & \multicolumn{1}{c|}{Count}   & \multicolumn{1}{c|}{Old}           & \multicolumn{1}{c|}{New}           & \multicolumn{1}{c|}{Old}             & \multicolumn{1}{c|}{New}            & \multicolumn{1}{c}{(Seconds)}   \\  \midrule
$D_1$ & 7 & 757 & 762 & 67,370 & 67,370 & $202 \pm 11$ \\
$D_2$ & 7 & 716 & 584 & 67,370 & 67,370 & $137 \pm 8$ \\ \bottomrule
\end{tabular}
\end{table}

\subsubsection{Does RESTORE package decrease the resource cost of dataset testing?}\label{sec:h2}

RESTORE has been institutionalized by EA and integrated into the dataset development process discussed in Section~\ref{sec:background}. Incremental changes made to the new vintage of the dataset are tested by RESTORE to make sure that the new vintage did not regress. If the tests failed, a root cause detection of the regression is easy to detect, as the failure is typically related to data transformations applied between the two increments.

In the long term, this increased automation and a better approach to data testing will significantly reduce the cost of delivering products to market. We estimate that the time of getting data to market can be reduced by about half (leading to cost reduction and improving customers' satisfaction).

\subsection{Potential extensions of RESTORE}\label{sec:potential}
The current version of RESTORE works by focusing on a pair-wise comparison of numerical datasets (measured using ratio and interval scale, as discussed in Section~\ref{sec:data-type}) that can be loaded into memory. This is sufficient for our use-cases. We released RESTORE as an open-source package so that one can extend or alter the tests implemented in RESTORE based on their specific use-cases or requirements. Below, we sketch potential ways to extend the package if one needs to compare large volumes of data, desires to compare other types of variables, or would like to do non-paired comparison of variables.

Currently, RESTORE reads all data into memory. This may be an issue for very large datasets (a.k.a. Big Data). This can be mitigated by altering the process of ingestion datasets into the package: rather than loading the whole dataset into memory, one can process a subset of columns (e.g., loaded using \code{fread} function from R \code{data.table} package~\cite{datatable}) in multiple iterations\footnote{Given that computations for every variable are independent of each other, the computations can be easily parallelized using \code{foreach}~\cite{foreach} and \code{parallel}~\cite{r_core} packages.}. Alternatively, if the number of observations is such that they cannot be loaded into memory, then one can leverage an external framework, such as Spark, and perform the computations outside of the R engine. Note that Spark integrates into R, e.g., using \code{sparklyr} package~\cite{sparklyr}.

If a tester needs to apply RESTORE to other types of data, some of the tests (discussed in Section~\ref{sec:data-type}) are readily applicable. One can extend the package by adding additional tests. For example, to extend comparison of distributions to ordinal data, one can adopt Mann-Whitney U test~\cite{mann1947test}.

Finally, as discussed in Section~\ref{sec:disc_cnt}, we did not perform comparison on non-paired observations (i.e., non-joined ones) of the datasets, as, empirically, they were found less useful for detecting defects in our datasets. However, if one desires to apply the tests to non-paired observations of a given variable, then it can be done with relative ease --- all the tests, with the exception of the ones discussed in Sections~\ref{sec:mre},~\ref{sec:cor}, and~\ref{sec:pair-cor}, are applicable to non-paired data.

\section{Related work}\label{sec:related}
There exists a significant amount of test frameworks for testing database engines and business logic that alters the data in the databases~\cite{kapfhammer2003family,maule2008impact,haraty2002regression,nanda2011regression,haftmann2005efficient}. In addition, some database testing frameworks are available~\cite{DbFit,DbUnit,NDbUnit,DBTest,SQL}. However, none of them are suitable for testing dataset vintages. Below, we provide a short summary of related but complementary papers.

\emph{Regression testing for database applications with code changes}. Haraty et al.~\cite{haraty2002regression} and Haftmann et al.~\cite{haftmann2005efficient,haftmann2007framework} focus on black box testing for database applications and leverage existing testing techniques for traditional software on database applications. In our case, the code of the application remains the same, while the underlying data are changing, hence the complementarity.

\emph{Regression testing of schema change}. Testing for database schema changes is another relevant topic. Maule et al.~\cite{maule2008impact} analyze the impact of database schema changes on database-driven applications. For example, a column can get renamed in a table, breaking existing queries accessing this column. They present an approach for predicting the impact of relational database schema changes upon object-oriented applications. Namely, they propose a technique to extract dependency relationships between applications and database schemas to perform impact analysis. The existing test frameworks (e.g., DbFit~\cite{DbFit}) can detect such an error. However, in our case the schema typically remains constant (rather the data in the tables change); therefore, this work is complementary to ours.

\emph{Regression test selection techniques for data-driven applications}. In regression testing, test suites can be large, and it can be time-consuming to process all the test cases. Thus, test selection techniques are widely used. Engstr{\"o}m et al.~\cite{engstrom2008empirical, engstrom2010systematic} report a review of existing regression test selection techniques based on empirical evaluations. Kapfhammer and Soffa~\cite{kapfhammer2003family} as well as Willmor and Embury~\cite{willmor2005safe} present test criteria, which capture interactions between an application and a database. Nanda et al.~\cite{nanda2011regression} introduce a regression test selection technique to selects a subset of existing test cases. This work assumes the presence of non-code changes, such as configuration files of databases. Rogstad et al.~\cite{rogstad2013test} present a similarity- and partition-based test case selection approach for database application regression testing. The test cases are generated from classification tree models. Haraty et al.~\cite{haraty2004regression} propose a two-phase test selection technique. In phase one, they adopt an impact analysis based on dependencies that exist among the components of database applications. In phase two, they propose two algorithms to reduce the number of test cases. The existing test selection techniques focus on the regression testing for applications rather than the data that these applications ingest. Thus, they are complementary to our work.

\emph{Open source projects for regression testing of databases}. Regression testing tools for databases try to assure that a query (captured in one of the previous releases) executes successfully (in the release under test). This functionality is available in many existing automated database testing frameworks~\cite{DbFit,DbUnit,NDbUnit,DBTest,SQL}. However, this will typically be inadequate for our needs as successful execution of a statement cannot guarantee that the returned results are correct (as was discussed in Section~\ref{sec:intro}). Some database testing frameworks, e.g.,~\cite{DbFit}, can readily check if the recordsets are identical and highlight the difference between them. However, as we discussed before, changes between vintages of a dataset are expected. Thus, these tests are not sufficient for our needs.

\section{Conclusion}\label{sec:conclusion}
In this paper, we present a set of tests that enable automated detection of defects in a new vintage of a dataset. We implement the tests in an open-source R package called RESTORE. We show that RESTORE can be used to quickly and efficiently detect defects in a new vintage.

We believe that this set of tests is of interest to practitioners,  as using the RESTORE package on their datasets gives them the advantages to
\begin{enumerate*}
    \item have more certainty about delivery dates for products,
    \item reduce the occurrence of data defects in products, and
    \item dedicate more time to developing new functionality, rather than testing the existing one.
\end{enumerate*}

This work is also of interest to academics, as it can serve as a building block in the movement of bringing lightweight software engineering practices into the data science realm to improve the quality of data-science-related products~\cite{larson2016review,cruz2017data}.

\begin{acks}
The work reported in this paper is supported and funded by Natural Sciences and Engineering Research Council of Canada, Ontario Centres of Excellence, and Environics Analytics. We thank Environics Analytics data scientists for their valuable feedback.
\end{acks}

\bibliographystyle{ACM-Reference-Format}

redpajama_set_name: RedPajamaArXiv


What does a love of fashion and a desire to empower people with chronic illnesses have in common? In the case of Keisha Greaves, this week's guest all of the above are what inspired her to create her fashion line, Girls Chronically Rock.
On this week's episode Keisha and I covered the day to-day realities of running an online product business, while living with a chronic illness, and juggling a part-time day job.
What is Girls Chronically Rock?
Who Girls Chronically Rock is for…not just people with chronic illnesses!
Why motivation/ encouragement from other people is what Keisha believes is the most important for those who run businesses with chronic illnesses… especially on days when you feel the most tired.
Why she believes that people with chronic illnesses shouldn't let feeling tired stop them. Even just posting a meme can go a long way!
The flexibility of Keisha's part-time day job, and how she juggles both aspects of her professional life.
Keisha's interest in making a line of Girls Chronically Rock clothing for the gym, while doing partnerships with other fashion designers who want to create an adaptive clothing line.
I have Limb-Girdle Muscular Dystrophy (MD); I was diagnosed at the age 24. I was born and raised in Cambridge, MA and went to high school in Cambridge. I received my bachelor's degree in Fashion Design and Merchandising and then got my master's degree in Business.
While in graduate school I began to repeatedly fall, and my leg would just give out on me without notice. I would fall to the ground and not be able to pick myself back up, I would need assistance from someone to lift me up with all their might, because it was something I just wasn't capable of doing on my own.
I also began notice when I tried to reach for certain things in the kitchen cabinets or try to exercise I was unable to lift my right arm up all the way.
I kind of just thought to myself maybe I need to go on a diet and lose weight, but in the back of my mind I knew it something more and after getting all the different testing done, I was officially diagnosed with Limb Girdle Muscular Dystrophy.
I always wanted to be a business owner and create something in fashion, so I started with a t-shirt line. Because I always like to wear different graphic logo t-shirts with different writing on them and wanted something inspiring.
That's when I came up with the name Girls Chronically Rock because I wanted it to have some form of the name chronic in it for chronic illness, so I thought of Girls Chronically Rock and I loved it.
Girls Chronically Rock was inspired by my own experiences, because I was diagnosed with Muscular Dystrophy and wanted to create something to help motivate, encourage, and let people know you rock and you can accomplish anything you put your mind to.
My t-shirt line consists of many different designs/logs with multiple meanings behind it. I add about a few different designs every month or so with a new look to see what customers may like best.
Each month 10% percent of proceeds go towards the Muscular Dystrophy Association.


My daughter is a student and she is pursuing her master's degree. She is currently living in Tiruchirappalli. We come from a middle class, nuclear family background with moderate values.
I have a Bachelor's degree and I am employed in Private Sector as a Hospitality Professional currently based in Tiruchirappalli. I belong to a Middle class, Nuclear family with Moderate values.
MY son completed MBBS and working christian Mission Hospital for last two years and has written entrance to do PG .
My son has a Bachelor's degree and he is employed in private Sector as a marketing professional currently based in Tiruchirappalli. We come from a middle class, nuclear family background with moderate values.
I'm a caring and loving, family attached person. My school and college life was at Alappuzha Kottayam and Tiruvalla. I completed Diploma in Polymer Technology with First Rank. Presently working with MRF Tyres as a Plant technical executive at Trichy plant. DOB: 24.10.1985 Time: 3.50pm.
My brother's name is Karkee R and he is working as a Designer in the Private Sector. He have completed BE. We belong to a Upper Middle Class, Nuclear Family with Moderate Values. He is a simple, down to earth kind of person. He loves his job and is very friendly with his close ones and has good humor sense. Loves mom and dad. Expecting a bride from a good family to be a daughter in our family.
My name is Suresh Babu and I am working as a Not in List in the Private sector. I have completed my Diploma. I grew up in a Middle Class class, Joint family with Traditional values.
I have a Diploma and I am employed in private Sector as a supervisor currently based in Tiruchirappalli. I come from a middle class, nuclear family background with traditional values.
I am a businessperson with a Bachelor's degree currently living in Tiruchirappalli. I belong to a middle class, joint family with liberal values.
My name is Sathya Bj and I am working as a Hotel / Hospitality Professional in the Private sector. I have completed my B.Sc.. I grew up in a Middle Class class, Nuclear family with Traditional values.


text:
Vorma is a river in Norway that brings water from lake Mjøsa into the Glomma river.  The Vorma is  long and flows through the town of Eidsvoll. The Vorma flows from Lake Mjøsa at the village of Minnesund to join with the Glomma at Årnes.

Vorma was so named because it was a "warm" river that never freezes over, while both the Glomma and the Gudbrandsdalslågen routinely freeze. The junction of the Vorma with the Glomma at Årnes and Nes is the site of the Funnefoss on the Glomma, a  fall.

History
The junction of the Minnesund and the Vorma, Eidsvoll, served as an ancient 'ting' place  - or meeting place for the local council or parliament - for Romerike during the period when travel by boat was much easier than by land. The "Eidsivating" was, according to tradition, founded by Halvdan Svarte, and served as the 'ting' place for all of the Opplands and Viken. Saint Olav also held a ting there and built a church at this site in 1017.

In 1795 a landslide totally blocked the Vorma at Disen for 111 days, completely drying the riverbed. A channel was cut around it to prevent a major overflow, with accompanying downstream damage.

On the Vorma, south of Eidsvoll, lies the Svanfoss where the water level in Lake Mjøsa is regulated. A  lock, which bypassed the Svanfoss, lifting boats  to allow the bypassing of the fall, opened in 1906.

Geology
The Vorma discharges from Lake Mjøsa at the village of Minnesund on the southern lakeshore.  The southern shore of Lake Mjøsa is a terminal moraine deposit formed by a pause in the glacial ice retreat (the Minnesund pause) that occurred ~9,400 years before the present. The terminal moraine is characterized by substantial clay, sand and gravel deposits.  The Vorma has eroded through these deposits, leaving steep banks with many high gradient tributaries forming cascades as they join the Vorma. The river cut down until it met the less-easily eroded intrusive sill (part of the Oslo Graben) at the town center of Eidsvoll where it formed the Svanfoss. Falls are also found where the Vorma meets the Glomma - the Funnefoss.

The glacial deposits also dammed up a side valley, forming lake Hurdalssjøen.  The Hurdalssjøen discharges to the Vorma river at Eidsvoll via the tributary Andelva river. Other tributaries include the Julsrudåa, the Jøndalsåa, and the Holtåa
.

References
 East Norway and its Frontier by Frank Noel Stagg, George Allen & Unwin, Ltd. 1956

External links
An overview of Norwegian canals, including one on the Vormas

Canals in Norway
Canals opened in 1906
Rivers of Viken
Water transport in Viken
1906 establishments in Norway

redpajama_set_name: RedPajamaWikipedia


text:
Friends of Fort Williams Park
Preserving and enhancing the natural resources and visitor experience of Fort Williams Park
Staff & Committee Members
Battery Blair
CLIFF WALK LANDSCAPE
LIGHTHOUSE VIEW
Ecological Gardening & Native Plants
Educational Programs and Partnerships
Invasive Pest Resources
Invasive Plant Resources
Resource Library Titles
Tours and Resources
Location of Fort Williams Park
Fort Williams Park is located at 1000 Shore Road in Cape Elizabeth, Maine, and is open from sunrise to sunset every day of the year. Admission is free and parking is available within the park.
Please see the Town of Cape Elizabeth's website for Fort Williams and Portland Head Light for more information.
Facilities and Policies
To reserve the picnic shelter or other areas of the park for group gatherings, please call the Facilities and Transportation Department at (207) 799-9574.
For more policies and guidelines for bringing your school class or other groups to the Park, refer to the Town's Group Use Policy for Fort Williams Park (pdf).
There are no trash cans within the park. Please carry your trash out with you.
Dogs must be on a leash with the exception of the off-leash area at the south end of the park. Click here to view a Map of the Unleashed Dog Area (pdf).
Click here to visit the Town of Cape Elizabeth's website for more information regarding facilities, policies and events at Fort Williams Park.
"It's Just a Walk in the Park."Join lifelong Cape Elizabeth resident Jim Rowe as he leads free guided walking tours of one of Maine's most scenic and historic public places. Tour dates and times are regularly posted on our events calendar
Back to top ⬆
Become A Friend of The Park
Friends of the Park provide the needed funds and hands to continue our work to restore the Park's natural beauty and enhance the Park experience.
299 Ocean House Road· Cape Elizabeth, ME 04107
info@fortwilliams.org
© 2014– Friends of Fort Williams Park. All rights reserved.
Site design: Leslie Evans & SlickFish Studios
Principal Photography by Jennifer Steele
Friends of Fort Williams Park on Facebook Friends of Fort Williams Park on Facebook Friends of Fort Williams Park on Instagram Friends of Fort Williams Park on YouTube Friends of Fort Williams Park on Linkedin

redpajama_set_name: RedPajamaCommonCrawl


medical-pot changes occur" [page one, April 15], and I'm shocked!
government cannot protect its citizens, what then is its role?
the wrongful arrests and/or seizure of property by the feds.
of manure just what it is - manure.
rightly moving toward focusing on the plant's prohibition.
yard and the time has come to end the bully's reign.
Just say "No" to the feds and their lies.


Check out HPL's calendar for exciting Reading Champions Programs for children grade K-6th. Performances are every Thursday from July 7th through August 11th.
A young athlete takes you on an Olympic adventure! Meet Vladimir Cashewninski, champion badminton player, as he prepares to compete at the Olympic Games. The night before his big day, he has an enthralling dream that takes him to Ancient Greece, where he meets the Olympic Herald and teams up with the audience to explore the first Olympic Games.
All Reading Champions Thursday Programs are free & no registration is required. Performances are geared for children grades K-6, but families are welcome.
"Ready for anything" will be the theme of the day with stories, songs, crafts, and other related activities to help build life-long readers!
Part of the "On Your Mark,Get Set...READ!" summer reading programming.
This program is for children ages 3 1/2 to 6.
Ginny at youth@beaconlibrary.org or 831-1134, x103.
Stop by Thursday, July 7th, to learn details of our 9th annual four-week program and information about our special events. This year's national theme is "Exercise Your Mind." We have an exciting line-up of fun and creative activities and even a Magic Show! Kick-off fun includes refreshments and a fantastic giveaway! Get the full list here or by checking our online calendar. You don't have to wait until the kickoff to register. Sign up at the front desk starting Friday, July 1st.
Join Vickie Raabin (aka Miss Vickie) to create a beautiful painted parasol. Parasols are full-sized and made of paper and wood. Let your creativity shine! All supplies are included in this free event for adults.
Registration is filling up fast! Sign up at the front desk.
HPL is pleased to host Xposure: Discovering What is Underneath, an exhibition of work created by adult artists in Anderson Center for Autism's Expressive Outcomes program. A Second Saturday artists' reception will he held on July 9th from 5-7. Xposure will be on display at the Howland Public Library from July 9th - August 6th.
Join crafter, Amina Chaudhri, as she teaches us how to make beautiful coiled bowls out of paper. Instead of throwing away magazines and catalogs - make something useful and lovely! All supplies are included in this free event for adults.
This event is free but registration is required. Register at the front desk.
Adults, join Magician and Ventriloquist Bob Conrad for a truly entertaining show that has it all! Comedy, puppets, and ... a touch of magic.
This fun and entertaining experience is free and no registration is required. This is an "Exercise Your Mind" 2016 Adult Summer Reading Program event.
Each month, in the Upcycle Art Studio, teens will experiment transforming discarded materials into works of art. This month, we'll create beautiful "mosaic" collages out of recycled magazines. All supplies are provided. Just bring yourself and your imagination.
Upcycle Art Studio is for middle school and high school students.
This program is free but space is limited. Please register in advance by contacting Michelle at communityart@beaconlibrary.org or 845-831-1134 x101.
Local origami expert, "Origami Bob" Voelker, will teach teens how to make beautiful origami creations, starting out simple and getting a little more daring if they wish. Participants will learn about this lovely Japanese art of paper-folding in a fun and relaxed way.
This program is free and open to middle school and high school students. Please register for one or both sessions by contacting Michelle at communityart@beaconlibrary.org or 831-1134 x.101.


text:
Lauren Mowery
Lauren writes the weekly wine column for legendary New York City paper Village Voice. A sixteen-year resident and former Manhattan litigator, she earned her Juris Doctor at New York's Fordham Law School where she was an editor on the Fordham Environmental Law Journal. Dissatisfied with her work in the legal field, she switched paths to the wine industry, founding the wine-dedicated colulmn Unscrewed for the Voice three years ago.
Unscrewed was nominated for Best New Wine Blog in 2013 (she won that year, but for her personal blog Chasing the Vine). It sees nearly a million unique visitors per month as part of the overall traffic received by the Village Voice food and drink pages.
In it, she covers New York City's latest wine-related news, trends, and bar and restaurant openings, as well as important figures in the NY industry. For example, she recently wrote about Chelsea resident Christy Canterbury's path to earning her Masters of Wine, and she was the first to interview Yannick Benjamin, the city's (and possibly the world's) first wheelchair sommelier. She combs the streets for interesting wine shops, and knows where to find aged Austrian whites (Chambers Street) or the best-priced, all-purpose selection of both wine and spirits with an adjacent education center (Astor Wines). Lauren routinely introduces readership to wines from around the country and world, first ensuring recommended bottles are available to New Yorkers.
If you need a wine bar recommendation, she knows where to send you. Looking for Midtown's most pedigreed venue? Head to Aldo Sohm Wine Bar. If you're a lover of natural wines and LCD Soundsystem, visit Brooklyn's Four Horsemen, owned by James Murphy. The best downtown wine bar with French flair and a daytime cold brew coffee program? Compagnie des Vins Surnaturels. Lauren knows the newest, oldest, best, worst, and soon-to-open venues across Manhattan and Brooklyn. No other writer focuses as squarely on the local market as she does.
Lauren has peregrinated across two dozen countries in the last few years, many of which are featured on her award-winning blog Chasing the Vine or in articles published in Wine Enthusiast, Saveur, Wine & Spirits, Tasting Panel, Somm Journal, Voyeur (Virgin Australia's inflight), Punch, Alquimie, Fodor's.com, Men's Journal, Palate Press, The Wine Merchant, and Bar and Restaurant Magazine. Immediately following completion of the W.S.E.T. Diploma with honors, she worked the 2015 vintage at Paul Cluver Winery in Elgin, South Africa. She earned her B.A. with Distinction, Cum Laude at the University of Virginia in 1999.
If New York City is your destination… Lauren will guide your way…
Stay in touch with Lauren:
Blog: ChasingtheVine
Facebook: ChasingtheVine
Twitter @ChasingtheVine
Instagram @ChasingtheVine
Tags:#winelover city Lauren Mowery New York City NYC specialists


text:
Bluewolf80 PM
Favorite Stories 113
10k+The Rise of the Drackens » by StarLight Massacre Harry comes into a very unexpected inheritance. He is a creature both rare and very dangerous, a creature that is black listed by the British Ministry. So now he must avoid detection at all costs, whilst choosing his life partners and dealing with impending pregnancy at just sixteen. With danger coming not just from the Ministry but even other creatures, what was he supposed to do?
Harry Potter, M, English, Romance & Family, chapters: 133, words: 1461k+, favs: 8k+, follows: 7k+, updated: 1/18 published: 4/14/2012, [Harry P., Draco M., Blaise Z., OC]
6k+An Unexpected Malfoy » by RiverWriter Once upon a time Hermione Granger literally ran into Draco Malfoy in a bookshop. His mother sees a connection between her son and the muggleborn that she can't ignore and determines to get to know the girl. An imagining of how things could have gone if Hermione had been taken under the wing of the Malfoy family.
Harry Potter, M, English, Romance & Family, chapters: 60, words: 294k+, favs: 8k+, follows: 10k+, updated: 10/18/2022 published: 9/19/2017, [Hermione G., Draco M.] [Narcissa M., Lucius M.]
7k+Embracing His True Self » by DebsTheSlytherinSnapefan Tom Riddle/Harry Potter Slash. What if Harry didn't want to fight Voldemort? What if he was fed up with the light side judging him constantly and wants to be his true self? The Slytherin side that he had squashed in a bid to fit in and be accepted? Realizing he never would...what of the prophecy? Could they overcome it? Will Voldemort even give him a chance?
Harry Potter, M, English, chapters: 114, words: 642k+, favs: 8k+, follows: 9k+, updated: 10/11/2022 published: 8/8/2014, Harry P., Tom R. Jr.
10k+An Aunt's Love » by Emma Lipardi Harry comes home from school and Petunia is upset at the change in her nephew. Petunia decides to do what she should have been doing all along. Dumbledore had better stay away. Petunia Dursley is upset. Warnings on profile.
Harry Potter, M, English, Drama, chapters: 77, words: 385k+, favs: 14k+, follows: 13k+, updated: 6/25/2022 published: 12/30/2004, Harry P., Remus L., Petunia D.
7k+Of Lies and Deceit and Hidden Personas » by Jessiikaa15 Everything changes the summer of 5th year when a mere letter causes Harry's magic to react wildly and he finds out that Dumbledore has been controlling him and his friends a lot more than first thought. With the truth revealed to him, Harry shows everyone he isn't the Gryffindor Golden Boy everyone thinks he is and he isn't the only the one with masks. DarkEvilTrio! SLASH
Harry Potter, M, English, Drama & Suspense, chapters: 59, words: 579k+, favs: 9k+, follows: 9k+, updated: 5/30/2022 published: 3/3/2013, Harry P., Ron W., Hermione G., Voldemort
3k+ Cold blood » by DerLaCroix Sometimes, the tide of events really changes a person.
Harry Potter, M, English, Drama & Romance, chapters: 35, words: 233k+, favs: 9k+, follows: 7k+, updated: 9/12/2021 published: 3/2/2010, Harry P., Hermione G.
7k+ Basilisk-born » by Ebenbild What if the Dementor attack in 5th year had ended with Harry losing? What if someone had stepped in to save him? And what if Harry ended up in the past with a chance to be more than he ever was before? A story about a forcibly time travelled Harry and its consequences... Manipulative Dumbledore, 'Slytherin!Harry', Time Travel!
Harry Potter, T, English, Mystery & Adventure, chapters: 72, words: 630k+, favs: 12k+, follows: 11k+, updated: 3/26/2021 published: 9/22/2014, Harry P., Salazar S.
2k+The Rise of the Drackens: The Scaled Bits » by StarLight Massacre This is a companion story to 'The Rise of the Drackens' with scenes and back stories that wouldn't fit into the main story and more information on the extensive world of the Drackens. You won't know the characters or the story if you haven't read the main fic first. Slash. Mpreg. Happy Fivesome.
Harry Potter, M, English, chapters: 70, words: 340k+, favs: 1k+, follows: 1k+, updated: 12/25/2020 published: 12/24/2011, Harry P., Draco M., Blaise Z., OC
3k+Barefoot » by Zaxaramas Harry has the ability to learn the history of any object he touches, whether he wants to or not.
Harry Potter, M, English, Adventure, chapters: 56, words: 157k+, favs: 12k+, follows: 14k+, updated: 12/22/2020 published: 7/7/2015, Harry P., N. Tonks
5k+Twins: A Different Life » by Jessiikaa15 When Voldemort attacked he didn't kill Lily and James. When he met his downfall they mistook Harry's twin as the GWL, they send their son away to the Dursleys but he doesn't quite make it in to the Dursley's loving care. Harry will take Hogwarts and his 'family' by storm and by the time the 'light' realize their mistake it will be too late for them to fix. OOC!Dark!Powerful!Harry.
Harry Potter, T, English, Drama & Adventure, chapters: 34, words: 279k+, favs: 13k+, follows: 14k+, updated: 8/1/2020 published: 1/4/2013, Harry P., Sirius B., Remus L., Voldemort
2k+ My Dark Protector » by DebsTheSlytherinSnapefan A Dark Protector rescues Harry from the abyss that he has only ever known. When Severus gets to Harry he isn't what they would have previously hoped. Life hasn't been kind to him, will Harry take to the magical world or will he always fear everything around him? Either way, Harry had a long way to go before they could determine this.
Harry Potter, M, English, chapters: 39, words: 153k+, favs: 3k+, follows: 3k+, updated: 7/10/2020 published: 1/16/2008, Harry P., Sirius B., Severus S., Albus D.
156 Virginia is for Lovers » by Lady FoxFire You find yourself in an unwanted marriage contract. Your death is not an option, your future spouse and in-laws' deaths might be an option. That is until Neville shows you how to escape and perhaps a way to deal with a Dark Lord who just doesn't want to die.
Harry Potter, M, English, Drama & Humor, chapters: 4, words: 10k+, favs: 686, follows: 326, updated: 6/14/2020 published: 6/12/2020, Harry P., Neville L.
4k+ Luna Lovegood and the Dark Lord's Diary » by The madness in me Tom Riddle's plans fall through when Ginny Weasley loses his diary shortly after starting her first year and it is found by one Luna Lovegood. A series of bizarre conversations follow. Luna? - Yes Tom? - I've been giving this a lot of thought...and I believe you may be insane. (Not crack. Plot takes a while to appear but it's there) See last chapter for fan art and translations.
Harry Potter, K, English, Humor, chapters: 101, words: 72k+, favs: 4k+, follows: 3k+, updated: 5/27/2020 published: 3/16/2017, Luna L., Tom R. Jr.
1k+Right Side of Hell » by NeoMare Albus Dumbledore was sure he made the right choice when he sacrificed an innocent child. Too late he understood how terrible that mistake was. Harry Potter was not what he wanted him to be, he was not what the magical world expected him to be. He was not a replica of James. Instead, he inherited more of Lily... Far too much to be convenient.
Harry Potter, T, English, Adventure & Suspense, chapters: 30, words: 324k+, favs: 4k+, follows: 5k+, updated: 4/19/2020 published: 4/8/2019, Harry P.
83 That Idea? by Elsbeth Ravensblood The war is over. The forces of the Light have lost. Harry is to be executed in Diagon Alley in front of a crowd by Voldemort. However, Voldemort grants his request to tell the people what he thinks of them before he does. AU Wizarding World bashing.
Harry Potter, T, English, words: 3k+, favs: 623, follows: 241, 4/1/2020, Harry P., Lily Evans P., Voldemort
146 Twelve and One » by Lomonaaeren HPTMR. AU. King James Potter has twelve daughters, all under a devastating curse. He also has one son, who serves as his father's steward. Harry has begun to wonder if his sisters will ever be free from the curse, until Prince Thomas Slytherin comes seeking a consort. (Very) loosely based on the fairytale "The Twelve Dancing Princesses." Seven parts. COMPLETE.
Harry Potter, M, English, Romance & Adventure, chapters: 7, words: 27k+, favs: 946, follows: 445, updated: 12/7/2019 published: 12/1/2019, [Tom R. Jr., Harry P.] James P., Merope G.
6k+ Benefits of old laws » by ulktante Parts of souls do not go on alone. When Voldemort returns to a body he is much more sane than before and realizes that he cannot go on as he started. Finding some old laws he sets out to reach his goals in another way. Harry will find his world turned upside down once more and we will see how people react when the evil is not acting how they think it should.
Harry Potter, T, English, Family, chapters: 109, words: 878k+, favs: 7k+, follows: 6k+, updated: 10/3/2019 published: 10/3/2015, Harry P., Severus S., Voldemort
211Harry Potter and the Final Straw » by courderouge2006 4th Year: Once again Harry is front and center for the Wizarding World's controversy. Bolstered by close friends and a surprising alliance & impugned by greasy bats and students, he reaches a breaking point. The timid, respectful knockaround is going to fight back using lessons ingrained in him by his upbringing with his "loving relatives". Dumbles/Ron bashing, slightly OOC Harry.
Harry Potter, T, English, Adventure & Humor, chapters: 6, words: 45k+, favs: 781, follows: 1k+, updated: 8/23/2019 published: 5/17/2016, Harry P., Hermione G., Fleur D., Neville L.
7k+Eros & Psyche » by RZZMG Draco challenges Harry and friends to play EROS & PSYCHE, a scandalous card game with a dark, mysterious history. It's Slyth vs. Gryff, male vs. female, pride vs. desire in the ultimate game of hearts and amour! Pairings: Draco/Hermione,Blaise/Ginny,Ron/Pansy,Seamus/Lavender,Theo/Daphne,Harry/Tracey. AU 7th yr. Secrets, romance, angst, and sex await the turning of the first card...
Harry Potter, M, English, Romance & Drama, chapters: 67, words: 433k+, favs: 10k+, follows: 11k+, updated: 5/14/2019 published: 8/2/2010, Hermione G., Draco M.
8k+The Reclamation of Black Magic » by ShayaLonnie Harry Potter's family isn't only at Number 4 Privet Drive. Unaware to even Dumbledore, an upheaval is approaching. The Ancient and Noble House of Black is reclaiming their power and changing the future of the magical world. *Updated Sporadically—Not Abandoned*
Harry Potter, M, English, Family & Drama, chapters: 39, words: 199k+, favs: 12k+, follows: 16k+, updated: 3/14/2019 published: 7/19/2016, [Harry P., Hermione G.] Sirius B., Dorea Black/Potter
113 Page 1 2 3 4 .. Last Next »
All CategoryHarry Potter (112)Sherlock (1) Sort: Updated Sort: Pub Date Sort: Title Sort: Words Sort: Reviews


1. Search Google for LDPlayer/Directly enter the official website of LDPlayer (//www.ldplayer.net/) to download LDPlayer.
2. Install LDPlayer: Directly install/Overlay installation (The installation path can be customized), click Install and wait for the installation process till it is successful.
3. After LDPlayer is installed, check whether the mark "VT" is shown in the top right corner of LDPlayer if LDPlayer is stuck at 94% or 50% (If VT is not shown, it means that VT has been enabled).
4. If VT is enabled but the problem remains, you need to check wheter your graphics driver is updated.
5. Open LDStore and type the name of app or game to search for it, and click INSTALL.
6. The solution to the crash: Increase CPU and RAM of LDPlayer; Switch from integrated graphics to discrete graphics; Update the graphics driver; View the CPU and RAM of your PC and then reduce the usage of them.
7. Lag: Increase CPU and RAM of LDPlayer; Switch from integrated graphics to discrete graphics; Update the graphics driver; View the CPU and RAM of your PC and then reduce the usage of them/the used memory of graphics card should be viewed when multiple instances of LDPlayer are running.
8. The APK file you want can't be found: Go to Internet to download the installation package (APK file), and then directly drag the APK file that you downloaded to LDPlayer. Or you can install it through the button Install APK on the right toolbar of LDPlayer; If you have a Google account and the game you want is in the Play Store, you can directly open the built-in Play Store of LDPlayer and then search for the game or app you want and directly click install!
9. Disable root permission when you encounter the prompt such as An emulator is running/Anti-virus software confilicts with the game program/ This device can't be used to play the game and so on.
10. The prompt Installation failed appears during installation: Restart your network.
11. Disconnect from Internet: Exit all your anti-virus software, including the built-in firewall of Windows; Download the tool for resetting the network; Right-click this tool and select Run as administrator; Restart your computer after it finishes resetting.
1. Optimized 리니지M, greatly increasing the frame rate and reducing CPU utilization.
2. Optimized the emulator kernel and reduced the CPU utilization of most games.
3. Solved the problem that some games cannot be operated through keyboard.
4. Solved the problem that Clash of Clans froze (for some Intel integrated graphics cards).
5. Solved the problem that the text cannot be displayed in MU: Awakening.


text:
Conventional oil based plastics vs innovative
Thesis grammar checker
Academic writing a handbook for international students pdf to excel
Home homework help tutors Theme of old story time
Theme of old story time
Sep 17, Zavia Currie rated it it was amazing The story was set in a rural settings. Miss Aggie was also called mama by her son Len who who later became a economist. Miss Aggie was very superstitious every thing that happened she blamed it on ghosts.
He began his theatre career as a teacher after a three year stint at Rose Bruford College, an English drama school. He was part of the renaissance of Jamaican theater in the early 's. Rhone participated in a group called Theatre '77, which established The Barn, a small theater in Kingston, Jamaica to stage local performances.
Theme of old story time vision of the group that came together in was that in 12 years, bythere would be professional theatre in Jamaica. Among his works is the script to The Harder They Come, a crime film, which was instrumental in popularizing reggae in the US. He also wrote the script for the romance One Love.
Concerns that are brought out in Old Story Time are still evident in the Jamaican society today. The issues of class and colour that are presented within the play are still issues in Jamaica today. In the effort to rise to a European standard of beauty, blacks are still slaves to this mentality.
This European standard of beauty aquiline nose, l The tradition of the story telling idea originated from Africa. This book tells about a mother who was enslaved by her past.
Toddler Storytimes | Sunflower Storytime
She grew up in a society, which she was taught, "any thing that was black wasn't good" and also that black signifies failure and hard ship, while white signifies prosperity and advancement.
This belief was passed down as a result of slavery and has followed her through out her life. The story tells of a single mother by the name of Miss Aggy also called mama, who lived alone with her son Len and beside her lived her best friend Pa Ben.
Mama would normally beat her son Len for associating with the black persons in the community; especially girls of his age group. She would tell her son what is it that she wanted for him and what is best for him; also that if he does as she said, and then he wouldn't have a problem.
As the years go by Len had left home to study overseas and his mother was getting really worried about him but after writing her she felt much better. In one of the letters that she received from him he told her that he was married and sent a picture of his wife who was a black woman; and this got mama very agitated.
She also thought that Len's wife Lois was turning her son against her. Pa Ben who was secretly keeping in contact with Len told mama that it was nothing of that sort, but mama thought that she had to do something about it.
One day Len came to the house; and they were very surprised and happy to see him, mama hugged, kissed and welcomed her son home. While talking him she would of course passed her remarks about his wife, which he did not like to hear.
While he was away mama paid down on a house for him at the bank and George also known, as Mr. Mac was her financial advisor and also one of Len's old enemies. When Len came back to Jamaica with his wife to stay, mama gave him the papers about the house since he was an economist and had a knowledge of banking.
While they were living there, mama would pass their remarks about Lois, whenever she went to visit them. George, who wanted to talk about the house with Len, went to his house and because of the rumors that Len heard about George he did his own check on him; and found out that George was a very sly and cunning fellow.
He was known to be a trickster. He would go around and trick persons and he would use the money that those people would invest in the bank. Due to Len's experience in the banking system George tried to convince him to invest in his bank and Len choose not to.
Mama hated that Lois; so she told George that Lois was the one who told Len not to invest. George was upset and because he head some secrets for Lois he thought he could black mail her and Len.
Lois who was a teller at his bank had borrowed some money to help George in acquiring his education, and so George wanted her to sleep with him and also to convince Len to help his business, she refused to and left the room and ran right into Lens arms who had over heard every thing.
Len was very upset but that was the least of his problem for his was concentrating on protecting his wife from the obeah that his mother placed on her. Len too went to an obeah woman to protect her but that woman was a fraud and this caused him to panic.
He sent Pa Ben for his mother while he talked to George telling him that he cant possibly tell the police about Lois, for then he Len would tell the court about George's actions and he can be charged for conspiracy to defraud.Six hours may seem like a long time for a slow-moving period piece, but there's a reason this story keeps getting adapted: it's thoroughly engaging, and this particular cast is especially so.
I Am Special Theme and Activities. It is important for young children to feel good about themselves. It is never too early to teach the power of a positive attitude and the importance of having self confidence.
At that time, there was great tension between Jews and Gentiles in Judah and that is a major theme of the book. What's It About? Jonah is about a prophet who . Nov 02, · Wow Thank you so much. Great blog. Your blog has awesome themes, and interesting book list, I wanted to start story time in my area, as we dont have libraries closeby.
I have a question, the themes, is it applicable to children of ages The Time story deals with synergy, Disney's upcoming streaming strategy and Iger's management style and successes. Iger, now 67 years old, is now scheduled to retire in Secondary are.
A book's theme is the main idea that flows through the narrative and connects the components of the story together.
A work of fiction may have one theme or many, and they aren't always easy to pinpoint right away; it's not always obvious and direct. In many stories, the theme develops over time, and it isn't until you're well into reading the novel or play that you fully understand the.
Essay outline on education in pakistan
King lear essays on nature
Deadlock prevention
How to write a story outline example of an outline
A paper on presentation skills
Cover letters for teacher positions
Written sales presentation omni wheel walker
Summary of neil postman
How to write a business letter outline
Abortion and the view of richard selzer
Microbiology case studies questions and answers
Orlando Sentinel - We are currently unavailable in your region


DRU is a modern company that continues to specialise in producing gas heating appliances.
It is the largest producer of balanced flue gas fires and wall heaters in Europe and also produces wood and multi-fuel stoves and fires.
The Largo Tunnel gas fire features the DRU Slim Line II technology. This is a design innovation, displaying only the glass window and the flame picture visible. The frames have been concealed giving the impression of a real wood fire.
The DRU Trio freestanding gas stove combines the nostalgia of a rustic stove with the innovative technology that is associated with DRU. It is a freestanding, balanced flue model in archetypal anthracite finish with a striking, 3-sided log fire display that will produce a comforting radiance all around your living room. For city or country living.
Paco Eco Wave is a flexible corner gas fire that can be installed to either side of a chimneybreast or turned into a prominent stand-alone room feature as our picture shows. It is available in either 'inset' or 'outset' versions, with frame colours of alu-metallic or anthracite. It has a heat output of 5.3 kW and an efficiency rating of 76%.