text:
\section{\uppercase{Introduction}}
\label{sec:Introduction}
The tremendous growth of social networks and the Internet of Things (IoT) brings various organisms exploit more and more data. Such amounts of so-called big data are mainly characterized by high volume, velocity and variety, as well as a lack of veracity, which together exceed the capacity of traditional processing systems \cite{Miloslavskaya2016}.
To tackle these issues, Dixon introduced the concept of data lake, a large repository of raw and heterogeneous data, fed by external sources and allowing users to explore, sample and analyze the data \cite{Dixon2010}.

In a data lake, original data are stored in a raw state, without any explicit schema, until they are queried. This is known as schema-on-read or late binding
\cite{Fang2015,Miloslavskaya2016}. However, with big data volume and velocity coming into play, the absence of an explicit schema can quickly turn a data lake into an inoperable data swamp \cite{Suriarachchi2016}. Therefore, metadata management is a crucial component in data lakes \cite{Quix2016}. An efficient metadata system is indeed essential to ensure that data can be explored, queried and analyzed.

Many research works address metadata management in data lakes. Yet, most of them focus on structured and semi-structured data only \cite{Farid2016,Farrugia2016,Madera2016,Quix2016,Klettke2017}. Very few target unstructured data, while the majority of big data \textit{is} unstructured and mostly composed of textual documents \cite{Miloslavskaya2016}. Thus, we propose a metadata management system for textual data in data lakes.

Our approach exploits a subdivision of the data lake into so-called data ponds \cite{Inmon2016}. Each data pond is dedicated to a specific type of data (i.e., structured data, semi-structured data, images, textual data, etc.) and involves some specific data preprocessing. Thus, we propose in this paper a textual data pond architecture with processes adapted to textual metadata management. We notably exploit text mining and information retrieval techniques to extract, store and reuse metadata.

Our system allows two main types of analyses.
First, it allows OLAP-like analyses, i.e., documents can be filtered and aggregated with respect to one or more keywords, or by document categories such as document MIME type, language or business category. Filter keys are comparable to a datamart's dimensions, and measures can be represented by statistics or graphs.
Second,  similarity measures between documents can be used to automatically find clusters of documents, i.e., documents using approximately the same lexical field, or to calculate a document's centrality.
We demonstrate these features in the context of  the COREL research project.

Our contribution is threefold.

1) We propose the first thorough methodological approach for managing unstructured, and more specifically textual, data in a data lake.

2) We introduce a new type of metadata, global metadata, which had not been identified as such in the literature up to now.

3) Although we articulate existing techniques (notably standards) to build up our metadata management system, adaptations are required. We especially combine a graph model and a data vault \cite{Linstedt2011} for metadata representation, and extend an XML representation format for metadata storage.

The remainder of this paper is organized as follows. Section~\ref{sec:RelatedWorks} surveys the research related to metadata management in data lakes, and especially textual metadata issues. Section~\ref{sec:MetadataManagement} presents our metadata management system. In Section~\ref{sec:POC}, we apply our approach on the COREL
data lake as a proof of concept. Finally,  Section~\ref{sec:Conclusion} concludes the paper and gives an outlook on our research perspectives.

\section{\uppercase{Related Works}}
\label{sec:RelatedWorks}

\subsection{Metadata Management in Data Lakes}

\subsubsection{Metadata Systems}
\label{sec:mdsys}
There are two main data lake architectures, each adopting a particular approach to organize the metadata system.
We call the first architecture storage-metadata-analysis \cite{Stein2014,Quix2016,Hai2016}, where the metadata system is seen as a global component for the whole dataset. Every analysis or query is then performed through this component.

The second architecture structures the data lake into data ponds. A data pond is a subdivision of a data lake, dealing with a specific type of data \cite{Inmon2016}. In this approach, storage, metadata management and querying are specific to each data type (voluminous, structured data from applications; velocious, semi-structured data from the IoT; various, unstructured textual documents). This organization helps conform to data specificity.

\subsubsection{Metadata Generation}
\label{sec:mdgen}

There are many techniques in the literature to extract metadata from a data lake. For instance, generating data schemas or column names make formulating queries and analyses easier \cite{Quix2016,Hai2016}. In the same line, integrity constraints can be deduced from data \cite{Farid2016,Klettke2017}. However, such operations are not applicable to textual data.

There are three main ideas in the literature for building and managing metadata that are appropriate for textual data.
The first proposal consists in indexing the documents.
This is notably applied in the CoreDB data lake to support a keyword querying service \cite{Beheshti2017}.

The second idea, called semantic annotation \cite{Quix2016}, semantic enrichment \cite{Hai2016} or semantic profiling \cite{Ansari2018}, adds a context layer to the data, which defines the meaning of data. This is done using World Wide Web Consortium standards such as OWL \cite{Laskowski2016}. It is used for enriching a couple of data lakes \cite{Terrizzano2015,Quix2016}.

Finally, the third proposal applies a textual disambiguation process before document ingestion in the data lake \cite{Inmon2016}. Textual disambiguation consists in, on one hand, providing context to the text, e.g., using taxonomies; and on the other hand, transforming the text into a structured document.

\subsection{Discussion}

Current metadata management techniques for textual data (Section~\ref{sec:mdgen}) are all relevant. Each brings in a crucial feature: indexing permits filtering data with one or more keywords; semantic enrichment complements data with domain-specific information; and textual disambiguation makes the automatic processing of textual data easier.

To achieve an efficient metadata system, an idea can be to take advantage of these three techniques, combining them into a global metadata management system. However, it is not so simple.

A first problem is that textual disambiguation implies that the original data are transformed before being fed to the data lake \cite{Inmon2016}. Thus, raw data are lost during this process, which contradicts the definition of data lakes \cite{Dixon2010}.

A second problem also concerns textual disambiguation. We need to define concrete examples of structured formats in which textual data can be converted to be easily analyzed. This remains an open issue as far as we know.

A third problem is that seeking for semantic enrichment via all possible semantic technologies seems illusory. There are indeed many semantic enrichment methods, which limits user flexibility.

Eventually, current metadata management techniques do not consider an important type of metadata, i.e., relational metadata.
Relational metadata, also called inter-dataset metadata (Section~\ref{sec:InterMetadata}), express tangible or intangible links between datasets or data ponds \cite{Maccioni2017}.
Obtaining relational metadata indeed allow advanced analyses such as centrality
and community detection proposed in a semi-structured data context \cite{Farrugia2016}, and that could be extended to unstructured documents. In the context of textual documents, a community typically means sharing a  lexical field, and can serve to automatically classify documents by topics.

\section{\uppercase{Textual Metadata Management System}}
\label{sec:MetadataManagement}

\subsection{Metadata Identification}
\label{sec:mdid}

A categorization of metadata in data lakes \cite{Maccioni2017} distinguishes intra-dataset (Section~\ref{sec:IntraMetadata}) and inter-dataset metadata (Section~\ref{sec:InterMetadata}). In addition, we propose a third category:  global metadata (Section~\ref{sec:GlobalMetadata}).

\subsubsection{Intra-dataset Metadata}
\label{sec:IntraMetadata}

This category is composed of metadata concerning only one dataset (a textual document in our context) in a data pond. Three subcategories are relevant to textual documents.
The first category is made of \textit{properties} that take the form of key-value pairs and provide information about  data  \cite{Quix2016}, e.g., document creation date, document creator, document length, etc.

The second category of intra-dataset metadata is called \textit{previsualization metadata}. They consist of a summary of the document through a visualization or a set of descriptive tags \cite{Halevy2016}. Their role is to provide users with an idea of the document's contents.

Finally, the last category concerns \textit{version metadata}. These metadata are inspired from
the idea of  keeping different versions of each document in the data pond \cite{Halevy2016}, e.g., a lemmatized version, a version without stopwords, etc.

To enhance version metadata, we introduce the notion of \textit{presentation metadata}.
Presentation metadata are obtained by applying two operations to the original data. The first is a transformation operation to get a cleaned or enriched version of a document, e.g., stopwords removal, lemmatization, etc.

The second operation consists in giving a structured format to the transformed document to actually achieve the presentation as, e.g., a bag of words, a term-frequency vector or a TF-IDF vector.

The generation of presentation metadata is similar to text disambiguation \cite{Inmon2016}, except that our proposal does \textit{not} involve any data loss. Original data are retained.

\subsubsection{Inter-dataset Metadata}
\label{sec:InterMetadata}

Inter-dataset or relational metadata specify relationships between different documents \cite{Maccioni2017}. There are two subcategories of inter-dataset metadata: physical and logical links~\cite{Farrugia2016}.

Physical links represent a clear (tangible) connection between documents. Such connections are typically induced by the belonging to some natural or business clusters, e.g., creation by the same person or belonging to the same business category, etc.
In contrast, logical (intangible) links highlight similarities between documents from their intrinsic characteristics, such as common word rate or inherent topics.

\subsubsection{Global Metadata}
\label{sec:GlobalMetadata}
This category of metadata covers the whole data lake. Global metadata may be used and reused to enrich documents or to perform more advanced analyses. They generally include semantic data such as thesauri, dictionaries, ontologies, etc.

Such metadata can be exploited to create enriched or contextualized documents through semantic enrichment \cite{Hai2016}, annotation~\cite{Quix2016} or profiling \cite{Ansari2018}. This process is then the first part of presentation metadata generation (Section~\ref{sec:IntraMetadata}) .

Global metadata can also serve to enrich data querying, e.g., a thesaurus can be used to expand a keyword-based query with all the synonyms of the initial keywords.

Eventually, semantic resources such as ontologies and taxonomies can also help classify documents into clusters \cite{Inmon2016,Quix2016}. For example, Inmon uses two taxonomies of positive and negative terms  to detect whether a text brings out a positive or negative sentiment, respectively.

\subsection{Metadata Representation}
A data lake or data pond can be viewed as a graph, where nodes represent documents and edges express connections or similarities between documents \cite{Farrugia2016,Halevy2016}. Such a representation allows to discover communities or to calculate the centrality of nodes and, thus, to distinguish  documents sharing the same lexical field from those with a more specific vocabulary
 \cite{Farrugia2016}. We adopt this approach because it is relevant to inter-dataset metadata (Section~\ref{sec:InterMetadata}).

Moreover, to handle the changing number and form of intra-dataset metadata, we combine the graph view of metadata with data vault modeling.
Data vaults are alternative logical models to data warehouse star schemas that, unlike star schemas, allow easy schema evolution \cite{Linstedt2011}.  They have already been adopted to represent a data lake's metadata \cite{Nogueira2018}, but retaining a relational database structure that does not exhibit an explicit graph representation.

In contrast, we associate data vault satellites representing intra-dataset metadata with graph nodes.
In our context, a satellite stores descriptive information, i.e., a set of attributes associated with one specific document.
Yet, a document may be described by several satellites \cite{Hultgren2016}.
Then, with the help of this one-to-many relationship, we can easily associate any new intra-dataset metadata with any document by creating new satellites attached to the document's node.

Figure~\ref{fig:MetadataGraphPres} shows an example of metadata representation for three documents. Visualization metadata, presentation metadata and properties are associated with each document. The association arrow from a document to metadata indicates that it is possible to associate several instances of every type of metadata with a document.

\begin{figure*}[hbt]
 \centering
 \includegraphics[width=9cm]{GraphPresentation.png}
 \caption{Sample metadata representation}
 \label{fig:MetadataGraphPres}
\end{figure*}

Inter-dataset metadata are subdivided into logical links represented by dotted edges and physical links represented by solid edges. It is also possible to define several instances of inter-dataset metadata. The only constraint that we introduce is that each logical link, when defined, should be generated between every couple of documents, which is not the case of physical links.
That is, we assume that there is always some similarity
between two documents. The question is then to know the strength of this similarity.
For physical links, the question is simply whether the link exists.

Finally, global metadata are not as such part of the graph, because they are not directly connected to any document. This is why they are isolated.

\subsection{Metadata Storage}

To efficiently store the metadata identified in Section~\ref{sec:mdid}, we adopt the idea to associate an XML metadata document with each document. This XML document serves as the textual document's ``identity card" and permits to store and retrieve all its related metadata.
This approach has notably been used to build a data preservation system for the French National Library \cite{Fauduet2010}.
Each digital document is ingested in their system as a set of information pieces represented by an XML manifest (Section~\ref{sec:manifest}) that can be viewed as a metadata package \cite{Fauduet2010}.

Depending on the type of metadata, we propose three storage modes: integrally within, partially within and independent
from a manifest (Sections~\ref{sec:atomic}, \ref{sec:nonatomic} and \ref{sec:relational}, \textit{irrespectively}).

\subsubsection{XML Manifest Structure}
\label{sec:manifest}

The manifest XML document associated with each textual document is composed of three sections, each dedicated to a specific metadata type. The first two sections are defined by the Metadata Encoding \& Transmission Standard (METS) \cite{TLoC2017}.

The first section, named \textit{dmdSec}, stores atomic metadata in a \textit{mdWrap} subsection (Section~\ref{sec:atomic}).
The second section is another  \textit{dmdSec} section where non-atomic metadata are referenced by a pointer in \textit{mdRef} XML elements (Section~\ref{sec:nonatomic}).

The third section is our proposal, which we name \textit{prmSec} for physical relational metadata section. It stores physical links in \textit{prm} elements.

Figure~\ref{fig:DatasetManifestDTD} shows the document manifest schema as an XML DTD, which we choose because it is more humanly understandable than an XML Schema.
Eventually, to make exploring and querying metadata easier, we strongly advocate for storing the set of XML manifest documents into an XML DBMS.

\begin{figure*}[hbt]
 \centering
 \includegraphics[width=12cm]{DatasetManifestDTD.png}
 \caption{Document manifest DTD}
 \label{fig:DatasetManifestDTD}
\end{figure*}

\subsubsection{Atomic Metadata Storage}
\label{sec:atomic}

Metadata in an atomic or near-atomic form are
directly included in the XML manifest. It is generally the case of properties, e.g., the document's identifier, its creation or last modification timestamp, its creator's name, etc.
More precisely, such metadata are stored in a \textit{dmdSec} section/\textit{mdWrap} subsection. In this subsection, atomic metadata are represented by XML elements from the (standard) Dublin core namespace \cite{DublinCore}.
However, the proposed namespace can easily be replaced by a customized namespace.

\subsubsection{Non-atomic Metadata Storage}
\label{sec:nonatomic}

Several types of metadata bear a format that requires a specific storage technology. Therefore, such metadata cannot be stored internally in the manifest like atomic and near-atomic metadata. They are thus stored in a specific format in the filesystem.
Yet, to make the retrieval of this kind of metadata easier, they are referenced in the second \textit{mdRef} section of the manifest, through URIs.

Presentation and previsualization metadata fall in this category. The original data must also be referenced this way, to allow users easily retrieve the raw data whenever necessary.

Global metadata are also stored using specific technologies and externally referred to. However, as they do not concern a specific document, they cannot be included in a document manifest. Thus, we propose the use of a special XML manifest document for all global metadata. This manifest contains a set of global metadata for which the name, location and type (e.g., thesaurus, dictionary, stopwords, etc.)
are specified. The global manifest's schema is provided in Figure~\ref{fig:GlobalManifestSchema} as a DTD.

\begin{figure}[hbt]
 \centering
 \includegraphics[width=6.75cm]{GlobalManifestDTD.png}
 \caption{Global manifest DTD}
 \label{fig:GlobalManifestSchema}
\end{figure}

\subsubsection{Relational Metadata}
\label{sec:relational}

Inter-dataset metadata have some specific requirements because they concern two documents. It is thus difficult to include them in a document manifest.
Hence, we propose to include physical links in the \textit{prmSec} section of all concerned document manifests.
The physical link's name must be specified. Its value represents a cluster name, e.g., for a physical link expressing the document's language, the link's name can be "language" and possible values "fr", "en", "de", etc.
Thus, clusters can be reconstituted by grouping all documents of equal value with respect to a specific physical link.

Unlike physical links, logical links cannot be defined internally within a manifest, since they materialize between each couple of documents with a given strength.
Thus, to store logical links, we propose to use a graph DBMS such as Neo4j \cite{Neo4j}, which allows to easily store link strengths and to conform to the graph view from Figure~\ref{fig:MetadataGraphPres}.
In this approach, each node represents a document by the same identifier as the one in the manifest. Each type of logical link is then defined between every couple of documents as an edge labeled with its name and strength.

\subsection{Metadata Extraction}

Several techniques can be applied to generate metadata. We categorize them with respect to the type of metadata in the following sections.
Moreover, here, we redefine intra and inter-dataset metadata as intra and inter-document metadata, respectively, to fit our textual document data pond context.

\subsubsection{Intra-document Metadata Extraction}
Properties can be obtained from the filesystem, e.g., document name, size, length, location or date of last modification \cite{Quix2016}. Once metadata are extracted, they are inserted into the manifest document, into the corresponding element from the Dublin Core's namespace.
Moreover, to conform to our metadata storage system, an ID must be generated for each document.  ID generation can be based either on the document's URI, if any \cite{Suriarachchi2016}, or on the document manifest's generation timestamp.
We choose to use the manifest's generation timestamp to avoid possible conflicts resulting from moving or changing the name of the document.

Then, there are two ways to generate previsualization metadata. The first is manually adding a set of tags to a document, in the form of atomic metadata. The second is automatically generating metadata by applying, e.g., topic modeling techniques.

Presentation metadata generation requires two advanced operations. The first operation consists in either data cleaning (e.g., stopwords removal, lemmatization, filtering on a dictionary, etc.) or data enrichment (e.g., adding context using taxonomies, translations, etc.).
At this stage, a transformed version of the document is obtained. The second operation, e.g., presentation as a bag of words or a term-frequency vector, is then applied on the transformed document to generate presentation metadata.

Generated metadata can then be stored either in the filesystem, possibly with an indexing technology such as Elasticsearch \cite{Elasticsearch} on top, or within a DBMS.
We opt for a hybrid storage solution exploiting the filesystem and Elasticsearch for metadata in a raw format (e.g., original documents),
and a relational DBMS for metadata in structured format, with respect to metadata type. This allows to take advantage of each storage mode's specific features.

Metadata are also referred to through an \textit{mdRef} element in the manifest document.
The \textit{XPTR} attribute of the \textit{mdRef} element is set with the metadata piece's URI, while its \textit{LABEL} attribute is set to the concatenation of the names of the two operations used for generating presentation metadata.

Table~\ref{tab:PresentationMetadata}  presents a short list of sample operations for generating presentation metadata.
To the transformation operations, we add a special operation with a neutral effect named \textit{original version}. It is equivalent to a ``no transformation" operation on the original document.
We also define a presentation operation with a neutral effect, \textit{classic presentation}, which leaves the transformed document in its raw format.

When these two special operations are applied to a document, presentation metadata are exactly identical to the original data.
These two special, neutral operations actually allow to retain original documents in the data lake as presentation metadata.

\begin{table*}[hbt]
\caption{Transformation and presentation operations}
\label{tab:PresentationMetadata}
\begin{tabular} {|l|l|l|}
\hline
 \textbf{Transformation operation (to)} &  \textbf{Presentation operation (to)} & \textbf{Resulting metadata} \\
\hline
 Original version & Classic presentation & Original document \\
\hline
 Original version &  Term-frequency vector & Term-frequency vector  \\
\hline
 Original version & TF-IDF vector & TF-IDF vector \\
\hline
 Lemmatized version & Classic presentation & Lemmatized document \\
 \hline
  Lemmatized version &  Term-frequency vector & Term-frequency vector   \\
  & & of lemmatized document \\
\hline
Lemmatized version & TF-IDF vector & TF-IDF vector of lemmatized document  \\
\hline
\end{tabular}
\end{table*}

Eventually, we assume that previsualization metadata represent a special case of presentation metadata. An example of presentation metadata can be obtained by filtering the original document on most frequent terms and presenting the result in a tag (term) cloud.

\subsubsection{Inter-document Metadata Extraction}

Some physical links can be automatically generated, e.g., the Apache Tika framework \cite{ApacheTika} permits to automatically detect document MIME type and language \cite{Quix2016}.
Once the MIME type or language is detected, documents of same type or language can be considered physically linked, respectively.

However, other physical links must be defined through human intervention. It is the case of links expressing documents belonging to the same business category. For example, in a corpus composed of a company's annual reports, the department to which each document belongs must be defined either by the document's folder name or by a tag.

Finally, we propose to generate logical links by computing document similarity measures representing the similarity strength between each couple of documents. Some examples of textual data similarity measures include the cosine similarity \cite{Allan2000}, the chi-square similarity \cite{Ibrahimov2002} and Spearman's Rank Correlation Coefficient \cite{Kilgarriff2001}. Such similarity measures have a high score for documents sharing the same terms and a low score for documents using different terms.
Moreover, many different logical links can be obtained by applying the same measure on different presentations of the documents.

\subsubsection{Global Metadata}
Global metadata generation generally needs human intervention, because such metadata are domain-specific and must be designed by a domain expert  \cite{Quix2016}.
Global metadata may also be derived from pre-existing metadata, e.g., a list of stopwords can easily be found on the Web and then complemented or reduced.

\section{\uppercase{Proof of Concept}}
\label{sec:POC}

Since there is no such systems as ours currently available, we put the metadata management system from Section~\ref{sec:MetadataManagement} in practice within a research project related to management sciences, and more specifically strategic marketing, as a proof of concept. This project is named COREL (``at the heart of customer relationship") and
 aims to study, analyze and compare  customer policies between and within companies.

We first present COREL's
corpus in Section~\ref{sec:presCODAL}.
Section~\ref{sec:architecture} is dedicated to the corresponding data lake's architecture.
Then, we explain in Section~\ref{sec:metadataExtraction} how we build the metadata system. Finally, Section~\ref{sec:queries} shows how our metadata system is exploited to perform analyses.

\subsection{COREL Corpus}
\label{sec:presCODAL}

The COREL
corpus is composed of 101 textual documents
from 12 different companies from various business domains. Documents are categorized in 3 business categories: interviews, annual reports and press articles. The documents bear 2 different MIME types (application/pdf and application/vnd.openxmlformats-officedocument.wordprocessingml.document) and 2 different languages (French and English).

Although the COREL
corpus does not fall in the big data category in terms of volume (the  size of  raw documents  is 0.15~GB), it does in terms of variety, with significant variations in annual reports with respect to companies, various sponsors in interviews (CEOs, marketers...) and a variety of press releases.

\subsection{Data Lake Architecture}
\label{sec:architecture}
To allow various analyses on the COREL
corpus, we build a data lake called CODAL (COrel DAta Lake)
that is composed of one single data pond, since the whole corpus is purely textual.
To fit CODAL within our metadata management system, we adopt the architecture and technologies shown in Figure~\ref{fig:architecture}. This architecture conforms to the storage-metadata-analysis system presented in Section~\ref{sec:mdsys}.

\begin{figure*}[hbt]
 \centering
 \includegraphics[width=11cm]{ArchitectureCODAL.png}
 \caption{Architecture of CODAL}
 \label{fig:architecture}
\end{figure*}

The lowest-level component in this architecture is dedicated to document and metadata storage. It is hybrid and exploits the following technologies: the filesystem to store presentation metadata, Elasticsearch to index presentation metadata and Neo4j to store logical links.
To speed up queries, we plan to replace the filesystem by a relational DBMS, but we currently keep it as is for simplicity's sake.

The second component stores XML manifest documents that contain atomic metadata, physical links and pointers to non-atomic metadata. The set of XML manifest documents is stored in BaseX, an XML-native DBMS \cite{BaseX}.

Finally, the top-level component in the CODAL architecture is a layer that allows OLAP-like analyses through a Web platform.  In addition, CODAL users can access the metadata system through  BaseX to execute ad-hoc queries or to extract data.

\subsection{Metadata Extraction}
\label{sec:metadataExtraction}
The metadata extraction process is composed of two phases. The first consists in generating the set of XML manifest documents (Section~\ref{sec:intraExt}) and all relevant metadata, while the second is logical link generation (Section~\ref{sec:physExt}).

\subsubsection{Intra-document Metadata and Physical Links}
\label{sec:intraExt}
For each document,
we generate an XML manifest document bearing the format defined in Section~\ref{sec:manifest}.
Properties are an identifier,  the document's title, its creator, the creation and the last modification date. These metadata are then stored as elements in the manifest first \textit{dmdSec} section/\textit{mdWrap} subsection.

Presentation metadata are each generated by applying a transformation followed by a presentation operation. The different transformation operations applied are stopwords removal, lemmatization, filtering on a dictionary and preservation of the original version. The operations retained for presentation are term-frequency vector format, TF-IDF vector format and classic presentation (raw format).

Once presentation metadata are generated, they are stored in the filesystem as either simple text files or in CSV format (for TF-IDF or term-frequency vectors).
They are then referred to in the corresponding XML manifest document.
Presentation metadata in classic presentation are also indexed in Elasticsearch.

Presentation metadata extraction is achieved using global metadata such as a list of stopwords and a dictionary. These global metadata are manually created and then referred to in the global XML manifest.

In the context of the COREL
project, we retain four physical links: belonging to the same company, document type (business category), MIME type and language. The first two links are obtained via the documents' directory structure in the filesystem. The last two are extracted with Apache Tika.

Once all the XML manifest documents are created, they are inserted in a BaseX database.
Figure~\ref{fig:manifestExample} shows an example of XML manifest document.

\begin{figure*}[hbt]
 \centering
 \includegraphics[width=12cm]{SampleManifest.png}
 \caption{Sample XML manifest}
 \label{fig:manifestExample}
\end{figure*}

\subsubsection{Logical links}
\label{sec:physExt}
To generate logical links, we calculate  the cosine similarity of each couple of documents using as presentation metadata TF-IDF or term-frequency vectors, since the cosine similarity can only be calculated with such vectors.

To store the generated measures, a node is created in Neo4j for each document. The different measures are then integrated into the database as edges carrying the similarity's strength. Each edge is named by concatenating the presentation metadata piece's name with the similarity measure's name.

\subsection{Possible Queries and Analyses}
\label{sec:queries}
Authorized users can freely access the CODAL metadata management system to perform either ad-hoc queries or analyses through the BaseX and the Neo4j DBMSs. All the generated metadata can be filtered, aggregated and extracted through these interfaces. However, some advanced data management skills are required, which are not possessed by researchers in management sciences.

Thence, we developed an intuitive platform to allow some recurrent analyses, i.e., OLAP-like analyses (Section~\ref{sec:olapAnalyses}), document proximity analyses (Section~\ref{sec:linkAnalyses})  and document highlights (Section~\ref{sec:highlights}).

\subsubsection{OLAP-like Analyses}
\label{sec:olapAnalyses}
These analyses are done through the left-hand side of our Web interface (Figure~\ref{fig:interface}). A set of multiple choice boxes allows documents filtering and aggregation with respect to physical links.

\begin{figure*}[hbt]
 \centering
 \includegraphics[width=16cm]{InterfaceCODAL3.png}
 \caption{CODAL analysis interface}
 \label{fig:interface}
\end{figure*}

Documents can also be filtered and aggregated using keyword-based queries. Here, a transformation operation must be specified and filtering operates using the classic document presentation  resulting from this transformation. Another option allows to extend keyword filtering with all synonyms of the given terms through a thesaurus.
The filtering options serve as analysis axes that are much similar to OLAP dimensions. Thus, filtering the corpus on a set of physical links is similar to OLAP slice and dice operations.

The characteristics of the aggregated documents are provided through statistics and visualizations. These visualizations can be compared to OLAP measures, because they also provide some aggregated information about documents. We propose four types of visualizations:

    1) distribution of documents through clusters induced by physical links, e.g., displaying what companies provide documents in English (filter documents on the English language and then observe the results' distribution with respect to companies);

    2)~timeline created from the document's creation or last modification date, which provides a temporal distribution of documents;

    3) most common terms (plotted as a bar graph);

    4) average term frequencies (depicted as a tag cloud).

The left-bottom part of Figure~\ref{fig:interface} displays a visualization of the most common terms in the selected documents with a tag cloud where the size of terms represents their weight.

As the analysis platform is a Web application, different filtering can be performed simultaneously in different windows, to compare a given visualization with respect to various filter values.
Similarly, the same filter can be set to observe simultaneously two or more visualizations.

\subsubsection{Proximity Analyses}
\label{sec:linkAnalyses}
The right-bottom side of the analysis platform  (Figure~\ref{fig:interface}) is dedicated to  proximity analyses on, and  proximity visualizations of, selected documents.
To show what documents are similar or different, we cluster them automatically by applying the Walktrap community detection algorithm \cite{Pons2006} on the sub-graph induced by the selected documents, given a similarity measure and a user-defined threshold. Walktrap's time complexity is $O(n^{2}~log~n)$ in most cases, where $n$ is the number of nodes.
The result of clustering allows identifying the documents that form groups with strong internal links, while links with documents outside of the group are weak.

Then, we represent the documents in a graph with a different color for each cluster. In Figure~\ref{fig:interface}, we observe in the yellow-colored cluster documents from two distinct companies. This can be interpreted as a similarity in the vocabulary used by these companies in some of their documents and, thence,  a potential similarity in their marketing strategy.

Another relevant technique for proximity analysis is documents centrality calculation \cite{Farrugia2016}, which can be applied in the context of textual documents to identify the documents bearing a specific or common vocabulary. Documents with a specific vocabulary are weakly linked to others, which implies a low centrality. In contrast, documents with a common vocabulary are highly connected together and have a high centrality.

Centrality can also be interpreted as the document's importance in the graph. Thus, documents with a high centrality can be considered essential because they are involved in a large number of links. Such documents should then be handled carefully, so as not to be destroyed.
However, this technique is not implemented yet in our analysis platform.

\subsubsection{Highlights}
\label{sec:highlights}

These visualizations are provided in the right-top part of the CODAL analysis interface (Figure~\ref{fig:interface}). After keyword-based filtering, the corresponding documents are listed.
In addition, we show, for each resulting document, a set of highlights where the keywords appear. This constitutes a summary of the document "around" one or more keywords.

Moreover, advanced options in the left-top side of the interface allow to customize highlight display. The \textit{highlights size} option can be used to increase or decrease the highlight's length. The \textit{thesaurus} option allows to expand the given keywords with all their synonyms, with the help of a previously selected thesaurus.
For example, we observe in Figure~\ref{fig:interface} that a query on the term ``client" also returns a highlight on the term ``consommateur", i.e, consumer in French.

\section{\uppercase{Conclusion}}
\label{sec:Conclusion}

We propose in this article the first, to the best of our knowledge, complete methodological approach for building a metadata management system for data lakes or data ponds storing textual documents.
To avoid the data swamp syndrome, we
identify relevant metadata extraction, structuring, storage  and processing techniques and tools.
We notably distinguish three types of metadata, each of which having its own extraction and storage techniques: intra-document metadata, inter-document metadata and global, semantic metadata (which we introduce). Eventually, we extend the XML manifest metadata representation to suit textual document-related metadata storage.

We apply and validate the feasibility of our metadata management system on a real-life textual corpus to build the CODAL data lake.  As a result, non-specialist users (i.e., with no data management knowledge) can perform OLAP-like analyses. Such analyses consist in filtering and aggregating the corpus with respect one or more terms, and in navigating through visualizations that summarize the filtered corpus. Users can also cluster documents to identify  groups of similar documents.

In future works, we first plan to replace the filesystem by a relational DBMS to store structured presentation metadata, and thus allow easier and faster queries and analyses.
We shall also improve our platform by integrating centrality analyses (Section~\ref{sec:linkAnalyses}).
Finally, since our current test corpus is small-sized, we plan to apply our method on a bigger one from a new project in management sciences and test its scalability.

Moreover, our objective is to turn the specific CODAL platform into a generic (i.e., not tied to the COREL project) analysis platform that implements the metadata management system we propose, and make it available to the community.  This would allow non-computer scientists to easily exploit any textual data pond.

Eventually, in the long run, we aim at designing a metadata management system that would help querying data ponds storing different types of data (structured, semi-structured, unstructured -- textual and possibly multimedia) altogether.

\section*{\uppercase{Acknowledgements}}
The research accounted for in this paper has been funded by the Universit\'e Lumi\`ere Lyon~2 through the COREL project. The authors would like to thank Angsoumailyne Te and Isabelle Prim-Allaz, from the COACTIS management science research center, for their constant input and feedback.

\bibliographystyle{apalike}
{\small


If your really want to acquire these three things, you should study and practice everything that this course teaches you. We will show you the path of success.
It is necessary for you to know that is an immensely superior power greater than electricity and dynamite: that is, the power of mind. When you think of a friend or a member of the family, etc., mental waves escape from your brain. These waves are similar to those of a radio transmitter. These waves travel through space and reach the mind of the person you are thinking about. Scientists have already begun to experiment with the power of the mind. Soon, the telementometer will be invented, an instrument with which the mental forces of each person will be measured. In the future, optics will advance some more, and then the photomentometer will be invented, an instrument that will allow us to see and measure the mental force radiated by the human brain.
You must know that just as man has a mind, the entire universe also has a mind. The human mind and the cosmic mind exist. The earth is condensed mind. The entire universe is condensed mind. Universal mind waves saturate the infinite space.
The first thing that an architect who will build a house does is to create the project mentally. He constructs it in his mind, projects it on the plan, and lastly crystallizes it materially. Thus, every thing, every building, first existed in the mind. Nothing can exist in the physical or material world in which we live before having existed in the world of the mind.
It is necessary to learn how to concentrate and project the mind with precision and great force. It is necessary for you to know that to concentrate the mind is to fix your attention on a single object. When you fix your attention on a distant friend, when you concentrate on that friend, you can be certain that your brain emits powerful mental waves that will inevitably reach your friend's brain. What is important is for you to truly concentrate. It is necessary that no other thought be capable of distracting you. You should learn how to concentrate your mind.
You are studying this course and we believe that you want to succeed in life and have health, money, and love. Reflect a little: learn how to use mental power. Whoever learns how to use the power of thought proceeds to success with absolute certainty, as the arrow guided by the hand of the expert archer reaches its target. Remember that the world is a product of the mind. You are what you are because of the mind. You can totally change yourself by using mental power. The poor and miserable are like that because they want to be so; they keep themselves miserable and poor with their mind. The rich and powerful person is so because he made himself that way with his mind. Each person is what he wants to be with the power of the mind. Each person projects into the world of cosmic mind what he is and what he wants to be. The projections of the mind physically crystallize and we then have in the physical world our rich or miserable, happy or disgraceful life.
Everything depends on the type of mental projections that have crystallized. As the cloud condenses into water and the water freezes into ice, so does mental energy: first, clouds (projects), then water (circumstances, development of the project), and lastly, hard ice (the project converted into concrete facts or things). If the project has been well done and with enough strength, if the actions or the development of the facts and circumstances have been marvelous, victory will be the result. The perfect condensation of a project is victory.
Without mental force it is impossible to achieve the crystallization of a project (commercial, etc.). It is necessary for our students to learn how to use mental force. But it is necessary for the student to first learn how to relax his physical body. It is indispensable to know how to relax the body to achieve the perfect concentration of the mind. We can relax the body seated in a comfortable chair or lying down in the corpse posture (with our heels touching each other, arms close to our sides, etc.). The second of the two positions (the corpse posture) is the better.
Imagine that your feet are subtle, that a group of dwarves escape from them. Imagine that your calves are full of small playful dwarves that leave one by one and that as they leave, the muscles become flexible and elastic. Continue with your knees, performing the same exercise. Continue with the thighs, sexual organs, abdomen, heart, throat, face, and head muscles successively, imagining that those small dwarves flee from each of those parts of the body, leaving the muscles completely relaxed.
When the body is perfectly relaxed, mental concentration becomes easy and simple. Concentrate on the business you have in mind. Vividly imagine the entire business, the people that are related to the work. Identify yourself with those people. Speak as if you were them; mentally say what you would like those people to say. Forget about yourself and change your human personality to that of those persons, acting the way you would like them to act. Thus you will define powerful thought waves that will go through space and reach the brains of the persons related with the business. If the concentration is perfect, success will then be inevitable.
This law is well known in the Orient. Millions of people in the entire world know of it because this law is universal. This law operates in the whole universe. If you misuse the power of the mind, the Law of Karma will fall upon you and you will be horribly punished.
Mental energy is a gift of God and should only be utilized for good purposes and with good intentions. It is just for the poor to better their economic situation, but it is not just to utilize mental power to harm people. Before doing a mental exercise to carry out the crystallization of a project, reflect and meditate: if you will use mental power to harm others, it is better not to do it because the terrible Ray of Cosmic Justice will inevitably fall upon you like a ray of vengeance.
Thought and action should always march totally united. Materialization is only possible when the circumstances are favorable. Learn how to determine favorable circumstances for your business. Freud, the great psychologist, said that everything that man does in life has two fundamental causes: (1) sexual impulse, (2) the desire to be great. Every human being is motivated by the sexual impulse. Everybody wants to be appreciated. Nobody likes to be despised. If you want to be surrounded by good circumstances for the crystallization of your business, then recognize the good qualities of your fellowmen; do not humiliate anyone. Do not despise anyone. It is necessary to encourage each individual in his job, office, or profession. By means of appreciation and encouragement, we can awaken enthusiasm in all those people who relate to us. Learn how to wisely praise your fellowmen without falling into flattery. People feel reassured with the food of esteem. Be gentlemanly; do not criticize anyone, thus you will form a favorable atmosphere for the crystallization of your business. The sincere appreciation of the merits of one's fellowman is one of the great secrets of success.
It is necessary to abandon the bad habit of talking about ourselves at every moment. It is urgent to employ the word to strengthen and encourage the good qualities of our fellowman. The Gnostic student should abandon the extremely bad habit of naming himself and telling the story of his life at every moment. The man or woman who only talks of himself or herself becomes unbearable. Persons like this fall into misery because people become tired of them.
Every morning before getting out of bed, say with force and energy: "We are strong. We are rich. We are filled with luck and harmony; Om, Om, Om." Recite this simple prayer and you will see that you are prosperous in everything. Have great devotion in this prayer. Have faith.
Hang a silk thread from the ceiling of your room. A needle should be at the end of the thread.
Concentrate on that needle and try to move it with the power of your mind.
When developed, mental waves can move this needle. Work ten minutes daily on this exercise. In the beginning, the needle at the end of the silk thread will not move. With time, you will see that the needle oscillates and moves vigorously. This exercise is to develop mental power. Remember that mental waves travel through space and pass from one brain to the next.


text:
Q: Mongoose query is taking too long in mqtt on_message callback I have a Mqtt client subscribing to a topic. When the subscriber receives message, the on_message callback starts working. I have a mongoose query based on the message inside the on_message callback. That query is taking too long to execute.
However, keeping the query outside the on_message callback executes immediately. But I want it inside the callback.
Here is my code sample -
var mqtt = require('mqtt');

mqttClient  = mqtt.connect("mqtt://broker.emqx.io:1883",{clientId:"mqttClient", keepalive: 1000});

let options = {
    retain:false,
    qos:1};

mqttClient.subscribe('topic' ,{qos:1});
mqttClient.on('message', async function (topic, message){
    let msg = JSON.parse(message);
    msg.forEach(async letter => {

    let dev = await Device.find({
        letters: letter}).lean();
  })
});

mqttClient.publish('topic', JSON.stringify(['a', 'b', 'c', 'd', 'e', 'f']), options);

here, dev query is the one taking too long.

A: As @hardillb suggested, I removed the query from the callback and queried somewhere else. Using query callback also does not help, it is still taking too long. I strongly believe it is because of mongoose. Because I tried the same with mongodb and it is more faster as the usual query would behave.

redpajama_set_name: RedPajamaStackExchange


text:
Q: firebase-collection won't refresh data on firebase-auth login, if location doesn't include user value If location (firebaseURL) in code below includes the user value (like auth.uid), then firebase-collection data will refresh on-login from firebase-auth. I am using Polymer 1.6.0
<firebase-auth id="auth"
      location="[[firebaseURL]]"
      provider="{{provider}}"
      params="{{params}}"
      user="{{user}}">
    </firebase-auth>

    <firebase-collection
      location="[[firebaseURL]]"
      data="{{messages}}">
    </firebase-collection>

This is because on-login from firebase-auth the user value changes and that changes the location value, refreshing the data coming from firebase-collection.
However, if the location path doesn't include the user value, the data will not show up on-login, and you will have to refresh the page for that to happen.
The only way I have found so far to work around this limitation is a bit hackish.
I have to modify the html code for firebase-collection as follow:
<firebase-collection
      location="[[_location(user)]]"
      data="{{messages}}">
    </firebase-collection>

Adding the _location() function like this, to force the firebase-collection data to refresh and show up on-login
  _location: function(u){
    // hacking - to trigger the update firebase-collection/document on-login / on-user-change
    return this.user ? this.firebaseURL : "";
  }

It works but it doesn't look right to me.
Is there a better way, a more polymeric way?
Thanks
AskPolymer

A: You could use a dom-if to stamp firebase-collection conditionally on user:
<template is="dom-if" if="[[user]]" restamp>
  <firebase-collection
    location="[[firebaseURL]]"
    data="{{messages}}">
  </firebase-collection>
<template>


text:
« Schiff T. Coyote meets the TrumpRunner | An impeachment peel-off: 'Rats flee Pelosi's sinking ship »
What is going on at Drudge Report?
By J.B. Shurk
The NOQ Report noted over the weekend that the Drudge Report has lost almost a third of its regular visitors since the summer. This observation came a week after Rasmussen Reports called attention to a rumor that Matt Drudge had sold his eponymous media behemoth some time ago. Does it make more sense that one of the savviest communicators of information in the history of the world (no exaggeration when you think of the sheer number of ideas he has used to change minds; affect populations; and influence leaders, movements, and governments for twenty-five years) decided to burn his creation down in nihilistic rage and madness, or that some other person or entity is now directing the show in his name?
I suppose time will tell. It does remind me an awful lot, though, of the Left's long tradition of capturing institutions that threaten its hold on power by seducing, intimidating, or converting the people who run them. It's plausible that a reclusive genius might take inexplicable pleasure in destroying what he created. It's even possible that Drudge is so committed to his personal beliefs that he is willing to sacrifice the lifeblood of his power by advancing media narratives that turn off visitors to his site. If the latter is true, though, his legacy as a master scrimshawer of stories with the faculty to subtly shift social consciousness disappears. He becomes just another town crier standing on the street corner, hiding behind a white sign with big red letters proclaiming, "The End is Nigh."
On the other hand, if I were a ruthless billionaire bent on world domination and concluded that the Drudge Report had stolen an imminently winnable presidential election from my favored candidate, then I might ask myself just what combination of sticks and carrots may be required to make me the new super-silent proprietor of the most successful engine for driving public opinion in the world. Would Matt Drudge promise to quietly sell me his beloved website while promising never to reveal that "the switch" was in? What would it take? One billion dollars in cash? Ten billion dollars in cash? There is no price too high for the privilege of owning yourself, you say? Well, what if I told you that I had personally helped elect district attorneys all across our great land with the jurisdictional strength to turn over any rock or look into any nook or cranny you might have long forgotten? Are you sure every tax return was complete? Every red light camera obeyed? How about any document ever signed by your father, grandmother, or childhood friend?
That sounds like extortion, you say. Time to bring in the FBI to make sure the guilty are punished and the innocent feel safe. Except the last FBI director committed fraud upon the FISA Court while participating in a conspiracy to bring down the sitting American president. And the FBI director before him took down a three-star general who had served his country in the U.S. Army for thirty-three years and as the director of the Defense Intelligence Agency by threatening the freedom of his son. And the current director of the FBI finds it more troubling and offensive that normal Americans have lost trust in the FBI than the actual, documented corruption on display from his bureau. Just spitballing here, but under that set of conditions, a billion-dollar payday with a promise to stay quiet doesn't sound all that bad for a guy whose privacy has meant as much as his life's work. We'll see, I know.
But then there are all those Republican congressmen who retired before the midterms and who are bailing from government right now. There's no way something like this might be playing out in their lives as well, right? I mean, how many conservatives can possibly be intimidated or bought off or provided sinecures at The Dispatch? Would that take more than one billionaire or more local district attorneys? It's too bad there's no federal agency that specializes in justice that might be willing to help these poor people out. Maybe if we started a 1-800 number hotline where such coercion could be reported or ran a PBS fundraiser for the victims, more conservatives would come out of the shadows and tell us their stories. Maybe.
Image: Chris Davis via Flickr.

redpajama_set_name: RedPajamaCommonCrawl


Sugar Shock  reports that in Mauritania, women are aggressively encouraged to be morbidly obese  – to the point of daily force-feeding of gallons of camel milk to female children in some cases. But before you go pointing fingers at this unhealthy and upsetting cultural oddity, remember that we are living in the land of the Heart Attack Grill  and the 2,700 calorie onion appetizer . The difference is that we seem to be willing to become obese.
We just can't get over this burger. For the millionth time.
We recently linked to blogger Moldybluecheesecurds' (yes) snippet on school nutrition reform. Moldy was nice enough to review our site , so we want to give a quick shout out! Be sure to visit this thought-provoking political 'n social commentary blog if you're into such topics as Iraq, oil, Uncle Sam, and global warming. We're not really political (except when it comes to health!). But if you are, you just might like this frequently updated blog.
Think getting healthy is an uphill battle that makes summiting Everest look like a walk in the park parking lot? Think again! These people did it , and you can, too!
We Americans seem to be having trouble with babies these days (for the love of intact ceilings, don't tell the Fuji). First, New Scientist reports  that the weight standards for babies are ridiculously high, paving the way for widespread obesity. And rotten teeth in the tiny tots is a growing problem  thanks to things like soda and "sports drinks". Little chicks deserve better!


Reverse copy of LD395. It may have been used for the Caricaturana collection. According to Provost there are doubts that this print is really by Daumier. We can assume that the first stone DR 395 was damaged and had to be redone as a mirror image (by Daumier?) to publish DR 396. The illustrated print in its only state originally belonged to the Longstreet Collection and is today part of the collection of the Metropolitan Museum of Art in New York. Robert Macaire may seem to be a realistic figure, however one should remember that in reality he is an artificial personality, created in 1823 by Benjamin Antier for his play " L'Auberge des Adrets". The breakthrough came in 1834, when Frédéric Lemaitre used the names of Macaire and his somewhat simple partner Serres-Bertrand for his various comedies in Paris. The figure of Macaire goes back as far as 1371, when - as the legend goes - Aubri de Montdidier, a nobleman from the court of Charles V was assassinated by Macaire. The murder was vindicated by his dog, the only witness of the assassination. A duel between the dog and Macaire, who was armed with a cudgel, should settle the case. The dog killed Macaire, who confessed the murder before he died. Philipon was the author of all the texts of this series. It should be noted that Daumier did not monogram or sign the prints, presumable because Philipon wanted to document his position as author of the texts and as "inventor" of the series.
Published in: Le Charivari, April 9, 1937.


Vaido (Võrumaa, Estonia) on 5 February 2019 in Landscape & Rural.
Sel aastal on jälle korralik talv / This year we have a decent winter again.
Beautiful, I'm shivering just looking at this image.
A beautiful composition and winter mood !
It looks like winter is there in full force. We have some snow and cold weather here in Olympia. Fairly unusual.
Fantastic winter scenery. Wonderful light and fine detail all around.


I found this cute store, Oliver Bonas, over the weekend and HAD to share some of my favorites from it! I first fell in love with this beautiful packaging…it's seriously perfect and I love everything about it. Once I got to searching around more, I found all this other CUTE STUFF! Take a look!
Uhhhh….this is AMAZING! I love these patterns!