Building power consumption datasets: Survey, taxonomy and future directions

In the last decade, extended efforts have been poured into energy efficiency. Several energy consumption datasets were henceforth published, with each dataset varying in properties, uses and limitations. For instance, building energy consumption patterns are sourced from several sources, including ambient conditions, user occupancy, weather conditions and consumer preferences. Thus, a proper understanding of the available datasets will result in a strong basis for improving energy efficiency. Starting from the necessity of a comprehensive review of existing databases, this work is proposed to survey, study and visualize the numerical and methodological nature of building energy consumption datasets. A total of thirty-one databases are examined and compared in terms of several features, such as the geographical location, period of collection, number of monitored households, sampling rate of collected data, number of sub-metered appliances, extracted features and release date. Furthermore, data collection platforms and related modules for data transmission, data storage and privacy concerns used in different datasets are also analyzed and compared. Based on the analytical study, a novel dataset has been presented, namely Qatar university dataset, which is an annotated power consumption anomaly detection dataset. The latter will be very useful for testing and training anomaly detection algorithms, and hence reducing wasted energy. Moving forward, a set of recommendations is derived to improve datasets collection, such as the adoption of multi-modal data collection, smart Internet of things data collection, low-cost hardware platforms and privacy and security mechanisms. In addition, future directions to improve datasets exploitation and utilization are identified, including the use of novel machine learning solutions, innovative visualization tools and explainable recommender systems.

Introduction
Recent studies have shown that buildings are in charge of more than 40% of power consumption demand and greenhouse gas emissions around the world. Indeed, this steady growth in energy consumption has been closely tied to the increasing number of population and the rising levels of prosperity [1,2]. Moreover, even climate conditions in certain regions of the world obliged households to call for more energy for heating, cooling, cooking and refrigeration needs. Business buildings, principally offices and university structures, are also viewed as structures exhibiting high power consumption [3,4]. Consequently, the expansion of power usage and carbon emission as well as expensive energy prices in the above mentioned environments has made energy preserving a vital goal for various public authorities of all governments in order to accomplish efficient energy reduction [5,6,7,8,9].
The building energy sector has recently attract the attention of various public energy efficiency initiatives to accomplish greater energy sustainability. Additionally, a clear correlation is found between household energy consumption and user behavior [10,11,12,13,14]. Every day, new appliances are installed and used in households resulting in an incredible rise of power demand. Monitoring the power usage of these appliances is dependably the initial step towards energy preserving. From this point of view, understanding and controlling user consumption behavior is also a key parameter to help householders reduce energy costs. According to the ongoing evolution of the Internet of Things (IoT), the use of smart meters for monitoring electricity consumption is expanding exponentially [15,16,17]. The up-and-coming generation of energy saving systems should be more effective, easy to follow and more challenging in order to improve end-user behavior.
As of now, as well as other research areas, energy, environment and sustainable development research topics are encountering the urgency of developing openly accessible datasets. In fact, power consumption datasets progressively come to be more consistent when estimating the precision of power monitoring techniques and perceiving how good they may behave under realistic circumstances. Consequently, checking the precision of outputs in real scenarios is critical in this research area [18,19,20]. Moreover, simulated database does not reasonably fit realistic datasets as "an experimental database or repository would ordinarily have unpredictable and unexplained complication nature that is laboriously anticipated and most of the time can be laboriously hard to manage [21,22]". On that account, Energy scientists have proved that it is important to have open access databases that provide aggregated power consumption as well as appliance based consumption for the various devices that constitute the overall consumption [23].
Moreover, end-users behavior is responsible of wasting more than 20% of the total energy consumed in buildings, and hence it is a key element in energy consumption [24,25,26]. Therefore, it is of paramount importance to: (i) design real consumption datasets; (ii) deploy novel platforms and smart-meters to collect granular and appliance-specific data, which have the means or incentive for sharing consumption data with end-users; and (iii) develop tools that help end-users in understanding their energy consumption footprints, such as innovative visualizations, and further implement novel strategies that help them in improving their behavior and reduce wasted energy [27], e.g. via deploying recommender systems. In this context, seeking to study how to save energy and understand power consumption behaviors in buildings, various datasets have been collected globally. They provide a big amount of information and create an immense quantity of readings about daily power usage and user behavior. Therefore, the use of machine learning (ML) algorithms becomes essential for handling large-scale datasets and extracting meaningful features from collected data. This can assist in many applications including forecasting power demand, energy efficiency and electricity preserving, appliance recognition, cost prediction, and other things in relation to energy usage [28,29,30]. In this respect, any ML algorithm for building power consumption deals with information drawn from smart meters and solar panels during the different periods of the day. This huge quantity of data including multivariate time series is of utmost importance to ML algorithms because future usage can be effectively anticipated [31,32,33]. In other words, ML algorithms can forecast such data and help in developing energy efficiency frameworks proficiently.
Extremely inspired by the rising relevance of public datasets, we opted to give a general review of up to 31 existing datasets in the field of building power consumption. Presently, householders, firms and public authorities are confronting difficulties to guarantee the energy efficiency and reduce usage costs. The use of a large number of appliances increases the energy demand, cost and carbon emissions as well. In this context, we present in this paper a deep overview of various power consumption datasets, their taxonomy and classification. The taxonomy is adopted to examine existing datasets, what kind of information they provide, their applications, their benefits and their limitations. To the best of our knowledge, this is the first framework that provides a comprehensive and universal survey of building energy consumption datasets, their applications and future trends. Following, discussions and important findings are presented via analyzing and comparing the features of existing datasets, their data collection platforms and related modules. Moreover, a novel dataset, namely Qatar university dataset (QUD) is proposed to answer the challenges and issues raised from the analytical study. Afterward, valuable future orientations to improve the quality of power consumption datasets and enrich their content are discussed, in which a set of recommendations to use novel hardware devices and Figure 1: General framework of an energy efficiency system. platforms are described. Moreover, future direction to improve datasets exploitation and hence improve energy saving are also also identified. To summarize, this paper presents a set of novel contributions, which can be listed as follows: • Reviewing up to 31 building power consumption datasets, describing their properties and highlighting their pros and cons via adopting a multi-perspective comparison based on various parameters. • Proposing a taxonomy of building power consumption datasets to assess the existing repositories based on their applications an characteristics. • Analyzing data collection platforms used to record power consumption datasets and related modules used for data transmission, data storage and privacy concerns. • Presenting a novel dataset called QUD that responds to various issues raised in the analysis of state-of-the-art datasets. QUD can be used for different applications, among them detecting of anomalous power consumption. • Providing a list of valuable future orientations for (i) improving datasets collection mainly through the use of novel hardware platforms, and (ii) improving datasets exploitation via adopting innovative tools such as visualization strategies and explainable recommender systems.
The rest of this paper has been organized into four sections. Section 2 reviews up to 31 existing building power consumption datasets and describes their usage contexts, properties, advantages and limitations. Section 3 presents a comprehensive discussion about the different characteristics of existing power consumption datasets. In addition, a novel dataset called QUD is presented which presents new functionalities. In Section 4, challenging orientations and future directions that should be followed in order to improve datasets collection and enhance datasets exploitation are described. Section 5 concludes the paper with a set of proposals for improving the quality of power consumption datasets and highlights future works. Finally, a list of abbreviations and nomenclatures used in this paper is presented in the Appendix.

Overview of building power consumption datasets
Several datasets can be found in literature and each one has its specific characteristics, making it difficult to select a database for treating energy efficiency issues. To this end, this work makes a deep comparison between all datasets based on various specifications, such as the period and region of collection, sampling rate, number of monitored houses, number of deployed sub-meters, collected features and release date. As a matter of fact, existing realistic datasets are divided into two major groups; appliance-level datasets versus aggregated-level based databases. The first group class provides sub-meter readings of appliance-by-appliance consumption. This kind of data is used for various applications, including energy saving [34], appliance recognition [35], occupancy detection [36,37] and preference behavior [38,39]. The second class group focuses on collecting overall consumption profiles of different buildings. It can be employed for energy disaggregation, energy efficiency, and further predicting energy consumption. Figure 1 illustrates a flowchart of a dataset collection process along with its associated modules, required to pre-process, analyze and interpret power consumption patterns. This is a general representation that can be used for different applications.

State of the art of existing datasets
To fit realistic scenarios of daily power usage and test energy efficiency solutions, scientists and specialists of smart energy monitoring systems need power consumption databases, in which developed algorithms can be evaluated in advance. Different databases have been collected and shared publicly. Under this section we review up to 31 power consumption datasets that are proposed in literature in addition to our novel dataset named QUD. We specify briefly the characteristics of each dataset and registered features in terms of current (I), voltage (V), active power (P), reactive power (Q), apparent power (S), normalized power (Np), energy (E), frequency (f), phase angle (φ), power factor (pf), energy cost (EC), weather (Wt), Temperature (T), humidity (H), occupancy (O) and light level (L).
In [40,41,42,43,44,45], large-scale datasets are formed, namely, HES, IHEPCDS, UMSM, SustData, REFIT and Dataport, respectively. While HES, IHEPCDS, UMSM and Dataport assembled energy consumptions patterns at a minutely level, SustSata and REFIT reported power usage profiles over intervals in seconds. All these databases provide consumption records at the appliance-level for long periods of monitoring. For example; in HES and UMSM data are raised for a period of one year, in REFIT and SustData energy patterns are accumulated for 213 days and 1114 days, respectively. Further, different features are gathered during the experimental campaign, such as I, V, P, Q, S f and T. REFIT has also the particularity of providing EC in $. Dataport repository [45] is also quite similar to UMSM database, since it captures energy usage at the same sampling intervals of 239 households but for a short collection period of two months. Dataport repository is also quite similar to UMSM database, since it captures energy usage at the same sampling intervals of more than 1200 households for a long collection period, which is more than 4 years.
In [46], OCTES is proposed, which is similar to REFIT. It records P, φ and EC ($). In addition, data are collected in a shorter investigation period. A bigger examination size and information are recorded at a comparable rate to REFIT. It lists the power consumption of each house; nonetheless, other pieces of information about the houses are not provided except their geological position. The case study specfied in this work depicts the utilization of a sauna in one home; as though, this data isn't shared publicly. Consequently, a presumption should be put with regards to the energy usage. In addition to power consumption, REFIT provides also readings about temperature, light, and motion patterns expanded with dwelling reviews specifying; size, age, warming sort, isolation,fabrication type and details about the tenants or occupants, job description and age.
In [47], Tracebase database includes power consumption patterns of various devices, which enables to examine disaggregation. The readings are collected at a sampling rate of 1 second. This dataset can be utilized for energy efficiency applications. However, it can not be employed for appliance recognition, preference detection or energy disaggregation since no data are provided about the devices being investigated and their properties. It gathers data of 43 distinct appliances, in which every one has various recordings from several days and several households. Furthermore, date and time records, P and Np are provided at a sampling frequency of 8 seconds.
In [48,49,50,51], AMPds1, AMPds2, ECB and PSD are proposed, respectively, which are minutely power datasets. Overall, AMPds1 and AMPds2 repositories are deemed as largely used databases, which compiled information of one and two years, accordingly, with a sampling rate of 1 min. In fact, energy consumption of 11 appliances is observed using 21 sub-meters. On the other side, ECB that provides electricity consumption benchmarks of 25 domestic residents located in Victoria State in south-eastern Australia is released. Consumption patterns were extracted from the aggregated circuit and for individual appliances over a duration of two years and at a sampling rate of 30 min. Further, consumption footprints of device-event labels from 10 homes in Austin, USA, were assembled.
In [52] and [53] authors released MEULPv.1 and MEULPv.2 datasets, respectively. MEULPv.1 gives energy consumption readings of 12 Canadian households. Data were recorded at 1-min sampling rates at both the aggregated and appliance levels. A total of 8 appliances are monitored during the data collection process. Meanwhile, MEULPv.2 provides one year monitoring of 23 households using a sampling rate of 1 min that designates aggregated and appliancebased consumptions as well.
In [21,54], RAE and GREEND databases are proposed, in which data are collected at a frequency of 1 Hz. The RAE is the initial version of an energy consumption repository that includes 1 Hz recordings for aggregated and sub-metered levels of two households. Besides power information, T and H records from a house's indoor regulator are incorporated. On the other side, GREEND is proposed to describe detailed energy consumption patterns collected through an experimental campaign via assessing electricity usage of various individual appliances in Austria and Italy. During the collection campaign, eight households are monitored, where each one contains up to nine different individual devices. The power usage patterns at a device-level are gleaned at a resolution of 1 Hz through a period of six months.
In [55,56,57], ECO, IWAE and DRED that capture energy information at 1 Hz sampling intervals are nominated, accordingly. ECO is an entire measurement campaign managed in order to collect comprehensive information of consumption patterns in six Swiss homes through an eight months duration. During the collection campaign, I, V, and π are collected from aggregated circuits and a set selected appliances at a sampling frequency of 1 Hz. Through the IAWE campaign, measurements were performed in a pilot household with three floors in Delhi in order to measure power, water and environmental profiles. Data are collected for a duration of 73 days from May to August 2013. In addition, 33 sub-meters are deployed through the whole house. DRED is publicly launched to capture energy, occupancy patterns and environmental data of one pilot house in the Netherlands. Sensor units are installed to measure aggregated energy consumption and appliance level electricity usage. In fact, 12 different domestic appliances are sub-metered at sampling intervals of 1 min while 1 Hz sampling rates are used to gather aggregated consumption.
In [58], DISEC is launched, in which various data are collected for 19 apartments at an Indian faculty housing complex during 284 days. Different features, such as P and Wt, are collected in a 30 seconds sampling intervals and then aggregated to 15 min, 30 min and 60 min intervals. As well, Wt variations are updated through measuring atmospheric conditions from nearly station measurements.
In [59] and [60], two hourly electricity consumption datasets are proposed. The first one called CRHLP includes energy patterns of 16 residential and commercial buildings monitored at every hour for a period of one year. Additionally, solar radiation and meteorological records are also collected. The second one, namely HUE, captures long-term energy usage profiles from five households with a sampling frequency of 1 hour. Furthermore, while device-level consumptions from house 1 are collected for a period of two years with sampling intervals of one minute, data from house 2 are extracted for a one year period with a resolution rate of 1 Hz. In [61], UK-DALE is proposed, which summarizes the current and voltage profiles of three houses at sampling intervals of 16 KHz and two houses at sampling frequencies of 1 Hz. Moreover, patterns of individual devices of five other households are collected at a sampling rate of 6 seconds for various periods varying from 39 to 655 days.
In [62,63,64], REDD, BLUED and BLOND datasets are proposed. Energy consumption records are captured at a sampling frequency of more than 10 kHz. The monitoring process, by contrast, is conducted for only a few weeks. For example, in REDD, six households are monitored, where the aggregated electricity consumption is measured at a sampling rate (15 KHz). Also, electricity consumption reviews of up to 24 devices are monitored at sampling intervals of 0.5 Hz. Furthermore, load patterns of other 20 appliances are observed at a frequency of 1 Hz while BLUED resumes the current and voltage readings of an individual household in Pittsburgh, Pennsylvania, USA. Data are listed at a sampling frequency of 12 kHz over a period of one week. For BLOND, it aims to capture continuous power consumption data. It delivers voltage and current records at the aggregated and device levels. This database includes data from 53 devices that represent 16 appliance groups. It englobes two main repositories; (i) BLOND-50 that in turn has consumption data obtained at sampling intervals of 50 kSps for grouped circuits and 64 kSps for individual devices; and (ii) BLOND-250 that entails usage patterns for a period of 50 days gathered using sampling rates of 250 kSps at the aggregated-level and 50 kSps at the appliance-level.
In [65], PLAID expresses power consumption profiles for more than 56 specific domestic equipments that represent about 11 appliance categories. Data are captured at a sampling frequency of 30 kHz that is judged among the highest resolution frequency used in existing building power consumption datasets when collecting load profiles. In addition, energy consumption information is captured for a period of three months during the summer of 2013 and the measurement campaign has been carried on in Pittsburgh, Pennsylvania, USA.
ACS-F1 [66] and BERDS [67] datasets that monitor load patterns at a comparable sampling rate are proposed. ACS-F1 records the amount of energy used in a set of households at an appliance-level. In this context, electricity sub-meters were employed to measure the energy consumption of 100 house devices that represent 10 appliance classes. Power sub-metering is managed at sampling intervals of 10 seconds for a period of only one hour. This database is especially suitable for appliance recognition applications. On the other side, BERDS collects energy consumption outlines at 20 seconds sampling rates for a period of one year.
In addition, QUD is presented in this framework, which is based on an appliance-based collection campaign. It can be used for different for different purposes, such as the energy saving, anomaly detection and energy demand prediction. QUD is collected using a system that incorporates sub-metering modules registering power consumption footprints in terms of P and other indoor climate conditions, including O, T, H and L. The data are recorded with sampling intervals ranging from 3 seconds to 30 min. The collection process will be spread over a one year period, while three months of data recording have been already completed.

Taxonomy
Power consumption datasets are split into two main groups: Appliance-level versus aggregated-level. The first one traces power consumption arrangements of individual devices. The second one provides the whole power consumption of households. Datasets can also be classified based on different aspects including application purposes or the nature of buildings, where data are acquired among which households, commercial buildings, academic buildings, industrial, etc. Figure 2 details the global taxonomy of various building power consumption datasets found in the literature.

Applications (A)
Using detailed power consumption readings and based on the nature of data collection procedures at appliance or aggregated levels, existing datasets could be exploited for various applications including, but not restricted to, energy saving, appliance recognition, occupancy detection, user preference detection, abnormal detection, energy disaggregation and energy demand prediction.

A1. Energy saving:
Investigating the building sector in terms of energy saving which is a principal element of its environmental and financial effects is of utmost importance. Consequently, energy saving is the most popular application of building power consumption datasets [68,69,70]. It can effectively reduce energy bills and decrease carbon dioxide emissions. It is made out of the following four stages: (1) the dataset collection stage, in which information is reaped from various sources, including energy sub-meters, ambient condition sensors and climate sources. The information gathered from these heterogeneous sources is saved in a specific dataset; (2) the pre-processing step, in which the information stored in the first step is pre-processed before utilizing various ML strategies. the pre-processing includes data cleaning, data resampling, features and events extraction and normalization; (3) The learning stage, in which ML algorithms are utilized to learn functions and models; and (4) The adoption of visualizations and recommendations phase, in which visualization tools are first adopted to provide end-users with interpretation of their consumption patterns. Following, specific recommendations or directives are derived in order to promote energy efficiency behaviors. Since the energy saving application is very relevant, we focus in this paper on studying how to improve systems developed in this direction along with related applications.
A2. Appliance recognition: Appliance recognition systems can help detecting operating conditions of devices using collected power usage patterns, and thoroughly recognizing the nature of each appliance [71]. In [72], a model was designed to detect the device activity and then to associate activities with devices using collected data. Analyzing power signals and checking relations among activities can assist detecting unattended devices, which use energy power without taking part the domestic's activities. In [73], in order to fit realistic conditions, experiments are usually conducted on a set of building power consumption databases, such as ACS-F1, PLAID, BLUED and UK-DALE.
A3. Occupancy detection: Solutions presented in this area detect individuals' occupancy in each specific part of a building based on power consumption profiles, as well as other environmental specifications, such as the temperature, humidity, luminosity and carbon dioxide emissions [74]. Dataset patterns are inspected before using ML approaches to derive the occupancy of the monitored part. Generally, occupancy is detected in two stages; (i) the presence or absence of individuals is investigated; and (ii) the number of individuals in the monitored building/room is then calculated [75,76,77]. In [78], a set of ML models as well as their boosting forms are developed and tested to detect occupancy using collected data from the AMPds2 measurement campaign.
A4. Preference detection: Methods described in this class deal with evaluating individual preferences through analyzing energy usage profiles. Most approaches treat the thermal comfort, although there are other arrangements that address visual comfort. Works released in this area investigate information-driven methodologies from an ML point of view and yielded arrangements that determine the preferences (e.g. the habits related to appliance usage) even through getting reports from individuals, i.e. information labeling or via observing the historic behavior of end-users to construe (in a straightforward manner) their consumption priorities or contexts that satisfy their well-being [38,79].

A5. Energy disaggregation:
Energy disaggregation is the issue of segregating the overall power consumption record into particular signals, in which each one represents an individual consumption of each electrical device [80,81,82,83]. This is valuable since getting separated power consumption of each appliance helps individuals to save energy and provide consumers with indexes on how to make appropriate actions [84]. Most of existing energy disaggregation frameworks resolving the problem of non-intrusive load monitoring (NILM) attempt to segregate the overall energy consumption without utilizing separate meters for each appliance [85,86,87,88,89]. For this specific application, REDD, BERDS, REFIT, AMPds1 and AMPds2 datasets are reputed among the famous repositories used for energy disaggregation.
A6. Demand prediction: ML algorithms generate precise power demand forecasts and they can be selected by public authorities and project managers instrumenting energy-efficiency procedures [90,91,92,93,94]. For domestic households, academic and industrial buildings, if the power demand could be predicted using ML strategies, directives and mechanisms that should be followed in advance can be established with a view of reducing load consumption of equipments and appliances inside these infrastructures [95,96,97,98]. Moreover, even if most the above presented databases (Section 2.1) are used for energy forecasting, we can find in the literature other datasets that are only designed for the specific problem of load and energy price forecasting, such as GEFCom2012 [99] and GEFCom2014 [100].
A7. Anomaly detection: With the progressive widespread use of smart-meters and smart sensors to monitor load usage in households, the utilization of power consumption observations as a solution to detect abnormal usage of energy is absolutely fascinating. Specifically, early detection approaches can be deployed to identify a large set of failures. In addition, recent works illustrate that for example, anomalous in lighting appliances can be responsible of 2-11 % of the whole power consumption of households and and commercial structures [101]. Furthermore, detecting faults or anomalies can permit analysts to comprehend energy consumption behavior of end-users and to be conscious of unpredictable energy usage values [102,103]. Various data mining approaches have been explored and deployed to detect anomalous events during energy usage process [104,105,106,107,108,109]. In addition, it is worthy to mention that there is an absence of annotated datasets dedicated to power consumption anomaly detection.
However, in order that a dataset could be correctly and efficiently used for a specific application, it should respect some specific requirements. For energy disaggregation, datasets should include both aggregated and appliance-level consumption fingerprints to compare the results obtained from disaggregation solutions with individual patterns. To conduct a user preference detection or even an occupancy detection, datasets should encompass appliance-level power consumption because it is difficult even impossible to infer user preferences from aggregated data. In addition, for occupancy detection, it is also required that consumption and ambient condition should be gleaned from individual appliances and from various parts of the building. For anomaly detection, it is of utmost importance that it includes labels annotating normal and anomaly consumption footprints to train developed algorithms. Lastly, for energy demand prediction, collecting power consumption at appliance-level or aggregated level will be appreciated, however, the collection period should be long to be useful.

Characteristics comparison of existing datasets
Aiming to extract representative outputs and relevant interpretations, a deep comparison study of existing building power consumption datasets is conducted in this section. Various dataset properties are investigated, which have a great importance when collecting data for developing energy efficiency solutions. Table 1 presents a comparative investigation of existing power consumption datasets. The analysis is built based on various characteristics that were collected in each dataset, including the region and period of collection, number of monitoring houses, number of monitoring appliances per house, collected features, sampling rate and release year. Additionally, we check and compare collected features for each database.

Data collection Platforms
Data collection platforms used to glean big energy consumption fingerprints are significantly impacting the energy efficiency systems. Specifically, sensing devices and attached platforms have a big role in gathering and safely storing data in appropriate databases. In this line, in this subsection, we focus on inspecting different architecture platforms used in the literature to collect energy consumption datasets and their properties, including wireless capability, data logging process and and data storage. In addition, because of the nature of collected data and their public access capability, privacy concerns are of utmost importance when producing datasets. Specifically, transmitting and sharing individuals' real-time power usage footprints and further their identities are probably quite harmful. To that end, it is important to investigate if the connections to the servers are secure or not in the presented dataset platforms. It is worthy to mention that in this section we focus on analyzing hardware architectures and related modules for only the datasets from Table 1, which present a description of their implemented platforms.
In [43], a power consumption monitoring and feedback platform is deployed, which is based on the use of sensors and a notebook for recordings data, storing them on MongoDB database, performing calculations and providing feedback to end-users. In [44], readings from several smart appliances are collected and transmitted using a commercial communication gateway called Vera3 smart home controller. The latter uses an encryption protocol to transmit data before their storage in a MySQL database. In [49], electricity footprints are gleaned using industrial meters and transmitted using a commercial platform named Obvius AcquiSuite EMB A8810, which includes many feature, among them the security provision. After that, they are stored offsite on a MySQL database server. In [54], platforms based on Raspberry Pi or BeagleBone along with a Plugwise Basic kit4 are used, in which collected data from sensing outlets are transmitted via a Zigbee network. Collected data are then stored on via a remote storage on a MySQL server without considering privacy concerns. In [62], a wireless plug monitoring device with an off-the-shelf system are used to collect power consumption data before transmitting them to central server. To keep the privacy of end-users, REDD dataset has focused only on hiding the identity of end-users and without deploying any secure protocol for data transmission.
In [21], power consumption readings of several appliances are wirelessly gathered using a data acquisition platform based on a Raspberry Pi 2B. Then, data are locally stored on an USB drive. In [61], a Nanode platform is used to wirelessly collect consumption data from individual appliance monitors and current transformers. Following, gleaned data are stored in a Nanode base station. It is worthy to mention privacy issues have not been considered. In [111], consumption records are acquired using a commercial plug, namely ENERTALK PLUG, which includes a microcontroller unit to process and save them in a device storage unit. After that, they are wirelessly transmitted to a data collector server. Finally, data are saved on a NoSQL Hadoop database server.
In this framework power consumption is measured using submeters components such as NodeMCU and SEN-11005 current transformer. Furthermore, occupancy patterns, luminosity, temperature and humidity data are also recorded using smart-sensors and then transmitted wirelessly using Raspberry Pi 4 Model B platform. The latter includes a No SQL CouchDB server that is used to store the gathered data using the JavaScript Object Notation (JSON). JSON represents a vastly used text format for data exchange, which keeps data structure without adding notation overhead. Table 2 summarizes the properties of hardware platforms used to collect different datasets, including wireless capability, data logging process, data storage and privacy consideration.

Discussion
Under this framework, a large number of building power consumption databases have been described, reviewed and evaluated according to different parameters as indicated in Table 1. In what follows, we derive pros and cons of each dataset, based on what has been discussed in the previous lines. This can adequately guides us to map recommendations for enriching and improving energy consumption databases.
• The biggest databases in terms of length and period of study are UMSM, HES, SustData and REFIT. Otherwise, for the case of HES, the observing period is too short and the sampling frequency of 2 minutes is a bit big. The same for UMSM, where data are gathered at a sampling rate of 1 min. Therefore, these datasets are inadequate for energy disaggregation as it will be difficult to differentiate between individual devices and occurrences. In contrast, these two repositories provide properties data about monitored homes, among others, the nature of building, size and rooms number and occupants number. Moreover, even if SustData and REFIT use a sampling rate of 8 sec and 10 sec, this is still not enough when conducting a real-time monitoring. • In some databases, e.g. PLAID, REDD and BLUED, high frequency monitoring is proceeded for only a few number of houses. This draws upon the prerequisites of energy disaggregation, where comprehensive characteristics catching transitory behavior can be extracted when high frequency collection is explored. • The majority of databases were gathered in the USA and Canada, under a 120V voltage and European nations under 230V. It can be deduced from Table 1 that the existing databases are collected in 13 different countries which are located in four continents; inter alia, America, Europe, Asia and Australia. In this context, these real databases have been produced in distinct climate zones, which cover humid regions (UMSM, REDD, BLUED), humid semitropical (IAWE), marine west coast atmosphere (UK-DALE,HES, AMPds1, REFIT, Tracebase, BLOND, IHEPCDS and OCTES), Mediterranean weather (BERDS) and arid zone (ECB ad QUD). However, no databases from Africa countries have been gathered under this investigation since there is no work in the literature who treat this topic in such countries. Moreover, to the best of the authors knowledge, QUD is the first dataset in the Middle East, where ordinarily 240 V voltage is used. Also, some collected particularities; for instance, the climate and environmental data depend on the location of the monitoring campaign. • The number and nature of monitored appliances, just like the number of observed houses essentially restrain the final usage of databases. In particular, a high number of houses and appliances is required for statistical inspections. In this case, UMSM, Dataport, OCTES, TraceBase, REFIT and HES are the most suitable databases. Plus, some datasets supervise various houses through multiple time intervals leading to difficulty and even impracticality while comparing between different homes. More than that, the setting under which domestic equipments are employed throughout the day is a basic operator for analyzing the complexity of the usage. This way, experimental campaign should be conducted in real conditions such as households, laboratories, or offices as opposed to simulated environments. • Some databases collect short-term energy consumption and only deliver records of real power, this is the case of COOLL, PSD, ACS-F1 and BLUED. Eventually, seasonal energy usage attitude can not be captured for short-term periods. In this aspect, making use of these databases to track power consumption behavior of end-users is not suitable. • A number of databases, among them UMSM, IAWE , ACS-F1, AMPds1, RAE and DRED have furnished a set of electric parameters, including I, V, P, Q, E, f and φ. Additionally to these records, other conditions such as T, O and L are also reported in QUD and REFIT datasets. The latter provides also analytic information notably related to the monitored electrical appliances and integrates statistics about daily activities in dwelling and residential environments, as well. This endows better a interpretive depth comparable to identical repositories (REDD, BLUED, GREEND). • Most of the studied datasets did not capture the exogenous conditions, such as the weather temperature, humidity, which can affect effectively the energy consumption. However, while the REFIT dataset has identical properties to OCTES and ACS-F1 datasets, it is also different because it adds other environmental data including the temperature, light, and motion patterns. In addition, household reviews are also reported, which include the surface, age, heating system, insulation, nature of buildings along with other data specifying the number of individuals, job quality and age. In this context, quantitative statistics gleaned from the reviews with occupants' statistics offer more possibilities to researchers to study the influence of other parameters on energy consumption. • There is a lack of available publicly annotated power consumption datasets to train/learn anomaly detection algorithms, in which power consumption variables are clearly labeled as normal or anomalous. Specifically, all the investigated datatsets in this framework except QUD do not encompass labels that identify normal or abnormal consumption, and thereby they can only be used to train unsupervised anomaly detection algorithms because they do not require annotated datasets. • Privacy and security concerns have not seriously been considered in most of the existing datasets. This is due to the fact that conventional meters required to be physically accessed and they registered power consumption for longer time periods (i.e. the real-time monitoring was not considered).

Qatar university dataset (QUD)
Using the pros and cons of the state-of-the-art datasets presented in the previous section, a measurement campaign has been conducted in the Qatar university energy lab to glean QUD repository. Specifically, in order to compensate the undersupply of appliance-level datasets dedicated for energy efficiency and anomaly detection in power consumption, a real-time micro-moment laboratory has been developed to gather accurate power usage footprints. Put simply, QUD is a set of consumption records from different installed electrical devices (e.g. air conditioner, heating system, desktop and light lamps) in addition to contextual data; including humidity, temperature, room occupancy and ambient light intensity. To the best of our knowledge, QUD is the first dataset in the Middle East, in which consumption data are collected at an ordinarily 240V voltage. This dataset have multiple usage scenarios such as detecting consumption abnormalities, testing recommender systems and assessing innovative visualization tools. Moreover, it is worth noting that QUD is among the first annotated repositories dedicated for anomaly detection in power consumption.
Therefore, the time-series data representing power consumption footprints for two appliances are registered along with corresponding cubicle occupancy, indoor temperature, indoor humidity, and luminosity. In order to label QUD consumption observations, the micro-moment paradigm is used which helps in identifying the moments of good or anomalous usage. Specifically, the micro-moments are deployed to come up with accurate statistics about consumers [10,34]. Using this dataset, the power consumption observations are labeled via the use of five micro-moment classes according to a set of standards out of the yielded appliance. These five micro moments are defined as; "good usage", "turn on", "turn off", "excessive power consumption", and "consumption when outside". The last two micro-moments  represent anomalous consumption behaviors that are leading to much wasted energy. Table 3 describes the micromoment classes and labels used in QUD (QUD can be accessed through http://em3.i-know.org/datasets/). In addition, it is worthy to mention that the micro-moment "consumption when outside" is limited to a set of appliances, such as air conditioners, televisions, light lamps, desktops/laptops, and fans, in which the end-user should be present during their operation to not be considered as an anomalous consumption [112].

Future directions
After analyzing, comparing and capturing pros and cons of existing datasets, a set of important orientations that can improve data collection and enrich datasets' content are identified. In addition, other directions to improve datasets exploitation are described as well. Figure 3 summarizes the future directions that are identified to improve both datasets collection and exploitation.

Improving the dataset collection
In order to develop powerful energy efficiency systems, it is of paramount importance to improve dataset collection procedures and hence enhance the content of collected data. In this respect, the following recommendations and directions can be establish:

Multi-modal data collection
Multi-modal data collection means merely collecting more than one type of data to accomplish an efficient energy saving task or other related applications. Specifiably, power consumption in buildings depends on multiple factors, which should be gleaned together power consumption footprints in order to design comprehensive datasets [113,114]. Figure 4 summarizes the principal parameters impacting the power consumption in buildings and contributing in the multi-modal data collection.
D1. Occupancy patterns: Domestic residents utilize more energy when they're occupied. Even this may appear glaringly evident, collecting occupancy data is a serious matter that must be inspected when searching for wasteful energy aspects in households. Specifically, we ensure that these structures consume less power when unoccupied. Individuals in households influence power consumption for the most part via lighting, cooling, heating and other plug loads. Analyzing power consumption for the duration of the day demonstrates an immediate relationship amongst

D1. Occupancy patterns (presence/absence of endusers)
Multi-modal data collection D2. User behavior (preferences and habits of end-users)

D3. Weather data (outdoor and indoor ambient conditions)
D4. Energy cost (direct information on energy price) Figure 4: Principal factors impacting the power consumption in buildings and contributing in the multi-modal data collection.
occupancy and power usage. For the moments when individuals are in a household, different rooms are conditioned or heated to an agreeable temperature. Of course, normal day-by-day activities need also power usage. The effect of individuals utilizing energy in a household is the reason we underscore the relevance of individuals turning off unused appliances or other devices in unoccupied rooms [115,116]. In this regard, the use of occupancy sensors is highly recommended in households or other buildings such as offices, laboratories or campus buildings to sense when someone is present or not and then turn off the appliances accordingly. By this way, a lot of energy can be preserved when the absence of individuals is confirmed.
D2. User behavior: Comprehending and improving individual power consumption behavior is among the successful approaches to reduce energy demand and encourage energy preserving. In fact, user behavior can be responsible of about 20-50% of the consumption level [117,118]. Therefore, collecting and inserting data about end-users' behaviors in the energy efficiency model can significantly decrease wasted energy. This can be done through gathering information related to their preferences and habits [119,120,121].
D3. Weather data: Relation between weather circumstances and power consumption has been proved in several works [122,123,124]. As a matter of exemplification, peak energy demand during heat waves is widely seen in so many hot countries. For that reason, gathering weather data is regarded as crucial while investigating user behavior. Over and above, existing and newly built households will certainly undergo the impact of climate change. Accordingly, collection and measurement of new energy consumption databases of these houses ought to consider weather patterns that integrate certain repercussions of climate change, rather than only considering historical climate information [125,126,127].
D4. Energy cost: Estimating the cost of household power consumption and providing this data to end-users can motivate them improving their behavior [128]. Forecasting the user's electricity bill and integrating energy price signals in energy efficiency applications can effectively increase power saving [129,130,131] since it helps the consumer to cognitively bridge the gap between consumption and cost. Moreover, collecting energy price profiles at the appliance level makes it unambiguous for the user which appliance raises more the cost. As a result, the consumer can relatively behave in order to reduce wasted energy.

Smart IoT data collection
Conventional meters are not able to gather the type of granular and device-level data, however, this becomes possible today with smart meters. In this line, in order to achieve target requirements in relation to data accuracy and further supporting real-time data collection and analysis, deploying smart meters and Internet of things (IoT) sensors to ensure a smart IoT data collection strategy is of paramount importance [132]. This helps in optimizing the communication, storage and computing resources.

Low-cost hardware platforms
In order to reduce the cost of datasets collection, the use of hardware platforms, enabling more cost-effective and powerful alternatives to process and transmit collected data is a high priority, such as the Raspberry PI 4 (RPI4) model B [133], ODROID-XU4 [134] and Jetson TX1 [135]. Those platforms can monitor energy consumption data along with collecting other essential contextual information, which ultimately results in a larger pool of data.

Privacy and security consideration
To preserve the end-users' privacy, power consumption footprints and end-users personal information should be protected. Personal data related to end-user specific power consumption patterns can be exploited to identify and supervise behavior patterns inside buildings (households or public structures). This is possible since electrical devices e.g. the microwave, air condition, washing machine, dishwasher, etc. can be detected and recognized from their power consumption fingerprints [2]. Therefore, personal data related to consumption signatures may be deployed to carry out real-time surveillance of end-users. In this regard, the data collection process must encourage producing challenging datasets and make power consumption statistics available to end-users and energy providers while respecting end-users' personal privacy and security. To that end, adopting robust techniques to remove personal information is a must, including encryption, steganography and aggregation.

Improving the dataset exploitation
Almost energy efficiency systems are built and validated using energy consumption datasets, which make them very important. Further, with the increasing amount of data collected in each database, the need for challenging solutions that can extract comprehensive information is becoming inevitable [136]. In this section we present three main directions, which can be investigated to ameliorate energy saving initiatives. It is worthy to mention that although the following directions are from a consumer's perspective, however, they are valuable for both consumers and energy providers. Specifically, they are generally developed by the latter and deployed to the benefit of end-users to help them in optimizing their energy usage. In addition, as discussed in Section 1, consumers are responsible for wasting more than 20% of the total energy consumed in buildings [24,25,26].

Mobile recommender systems
Lastly, it is noticed that mobile smart devices are becoming an indispensable part of our daily life. Unlike earlier mobile phones that provide limited functionality, smart phones can do a variety of very useful jobs. With the widespread usage of smartphones and the fast growing of the internet and network facilities, a massive amount of data is produced. Consequently, modern societies have started the age of Big Data through successfully discovering users' possible demand and preferences. This has raised the necessity for data scientists and energy management stakeholders to conduct studies on mobile recommender systems for controlling users' energy efficiency [137].
Recommendation systems are commonly deployed to polish the use of smartphones and to assist in dealing with the large amount of data through establishing appropriate advices using recommendation schemes and contextual information.
In this regards, the role of recommender systems will be essential to promote energy efficiency and help end-users understanding and improving their consumption footprints [138]. More specifically, every particular recommender application is generally elaborated with an explicit context in mind with the aim of solving in some sense the data overloading issue due to the large-scale datasets of power consumption. The effectiveness of a mobile recommender system has been demonstrated through real-use applications in academic buildings [137], in which a context-aware based recommender app is developed to help in supporting end-users to transform their energy consumption habits.
Furthermore, the architecture of recommendation systems that is usually based on interactional models, graphic user interfaces and recommendation engines makes them productive and useful to deal with energy efficiency applications [139]. To summarize, the use of mobile recommender systems is recommended to improve power consumption datasets exploitation via: • Developing explainable recommender systems can be very supportive to improve data exploitation with a view to replacing inefficient energy habits with efficient ones. An explainable recommender system aims at providing end-users with tailored recommendations, followed by explanations about them [140]. Explanations refer to the motivations behind the recommendation or to the benefits from providing the recommended action or advice. They can enhance the persuasiveness of the system, end-users' understanding and satisfaction and provide an immediate reward to them. • Developing intelligent mobile home monitoring systems using collected data to provide information and monitoring options to the end-user to help him control its load usage, visualize consumption statistics and compare them to those of other users, and further predict the overall charge of monthly bills [141].

Visualization for understanding user behavior
Visualization is seen to be the most effective way to assimilate increasingly large datasets with the aim of interactively and perfectly conveying insights to end-users, consumers, and stakeholders in general. Recent tools, methods, and softwares leveraged for visualization of energy consumption require further improvements to remain more important in a planet with larger low-carbon emissions. Moreover, they are required to sensitize energy-consuming behavior in an approachable and stimulating way. In this context, we present in this section an example of a novel visualization approach based on micro-moments analysis. Figure 5 displays a time-series energy consumption of a television and its micro-moments scatter plot at sampling intervals of 3 min, recorded in DRED dataset. This novel visualization strategy is presented as an example, in which energy usage micro-moments of 2 days are captured and plotted. It is used to display good usage (class:0), excessive usage (class: 3) and consumption while outside (class: 4). Users can seamlessly get the plots at different sampling rate starting from the milliseconds.
As it can be deduced, tracing micro-moments through time patterns facilitates identifying moments of abnormal consumption and then makes it easy to establish precise guidelines helping to reduce energy waste. Moreover, this helps end-users understanding their consumption footprints, increasing their awareness, and hence triggering them to improve their behavior through the use of tailored recommendations. In addition anomalous consumption behaviors can be identified when an adequate visualization tool is adopted, e.g. the micro-moments visualization, and hence end-users can improve their behaviors based on the detected anomaly. Moreover, it is worth noting that the use of the micro-moments paradigm to detect anomalous consumption can be enlarged to identify other kinds of anomalies, e.g. detecting abnormal consumption of an air conditioner while doors/windows are open via considering other information sources. Therefore, end-users will be provided with the appropriate notifications and advices, i.e. close doors/windows to reduce wasted energy.
In addition, a set of valuable recommendations and future directions towards designing effective visualizations aiming to increase end-users energy awareness is summarized as follows: • Visualizations need to catch the attention of their users via using bright colors, contrasts and varied views, where somethings are changing constantly. More importantly, they should implement colors that are legible for people with color vision deficiencies [142]. • Visualizations also need to employ comparisons between different time period consumptions to stimulate energy users' reflection and learning. Individuals are highly motivated in preserving power usage through making comparison of their current consumption to their own previous consumptions. • Fragile long time-series aggregated systems for visualizing energy consumption are better to be avoided. By nature, the human being's mind can not always memorize all what they do in each second or which appliance do they use more in their daily routine. Hence, it was concluded by many studies that: a. Collecting appliance-level consumption data is highly needed for consumers to cognitively decode their consumption data and to finally make a decision towards changing their behavior [143]. b. Instantaneous short-time intervals are the best way to aid the consumer to easily figure out at what instance their energy cost increased [144]. This makes it possible for the users to ameliorate their behavior and to notice the direct effect from it. • 3D visualizations of households with real-time energy consumption statistics can impart better contextual understanding to homeowners and help them control practices shaping their consumption. In relation, it is also crucial to support that by exploring the deployment of other emerging technologies, including virtual and augmented reality, interactive visualizations and wall sized presentations.

ML algorithms of large-scale datasets
Another important direction that can improve the quality and exploitation of power consumption datasets relies on deploying ML algorithms, which can help significantly in reducing energy consumption and thereby decreasing energy costs and carbon dioxide emissions [145,146]. Therefore, boosting novel progresses and challenges with regard to ML models is of paramount importance. In this context, several directions could be identified in which ML play a major role when shifting to more sustainable end energy efficiency environment, among them: • The use of generative models, such as generative adversarial networks (GANs), which can tremendously improve the quality of collected data via completing incomplete power consumption signals (due to data loss occurred during the collection step) and hence leading to a better exploitation of the produced datasets in different applications [147]. • The use of deep learning models to identify consumption anomalies via classifying the micro-moment classes of end-users' power consumption [34]. These algorithms assist in analyzing consumption footprints to detect abnormal consumption related to "excessive consumption" and "consumption when outside". After defining the micro-moments described in section 4.2.2, DNN and other ML algorithms with different configuration parameters could be deployed, including logistic regression, linear discriminant analysis (LDA), support vector machine (SVM), K-nearest neighbors (KNN), decision Tree (DT) and ensemble bagged tree (EBT. The selection of the appropriate ML model is mainly based on ensuring the best compromise between the identification performance and computational complexity. Therefore, this results in a better exploitation of the collected datasets for identifying abnormal power consumption.

Conclusion
Existing building power consumption datasets have been reviewed together with their merits and drawbacks within the lines of this work. Comprehensive comparisons between these datasets have been conducted in terms of different factors and data collection platforms that categorize each of these. Based on the fruitful analysis gleaned from these comparisons, a novel annotated dataset has been presented, namely QUD, which can be very useful for power consumption anomaly detection since it includes labels of good and anomalous usage. Moreover, we came with recommendations and future directives for improving the quality and the content of future power consumption datasets. Thus, a set of relevant orientations have been identified as follows: • Adopting a multi-modal data collection, which is based on gathering various data sources because the energy consumption is affected by different factors. Therefore, data should be gathered from several geographical positions with regard to ambient conditions, atmospheric and environmental resources, occupation patterns, user preferences and energy cost. • Adopting smart IoT data collection strategies to collect data from various IoT sensors at a low sampling rate, which leads therefore to developing real-time and scalable energy saving systems.
• Collecting more annotated anomaly detection datasets in order to encourage testing and developing anomaly detection algorithms. Overall, detecting anomalous power consumption plays a major role in reducing wasted energy. • Producing comprehensive databases by reference to the level of consumption (especially at the appliance-level) and the duration of the collection campaign (i.e. collecting data for the entire seasonal periods of the year) while respecting end-users personal privacy and security • Formulating protocols and standards to characterize building power consumptions datasets that can help make unified metadata strategies and terminologies, facilitating the comparisons, and rigorously understanding the state-of-the-art.
In addition, genuine initiatives to improve datasets exploitation and utilization have been identified in this framework. Consequently, a novel visualization strategy has been presented based on the micro-moments analysis, which enables people to comprehend their own power usage footprints, and accordingly interpret their electricity consuming behavior. Moreover, it helps them easily getting statistics on their actual power consumption. Moreover, another example of using ML algorithms has been introduced to classify power consumption micro-moments and detect anomalous usage, such as "excessive consumption" or "consumption while outside". These two behaviors are responsible for wasting a large amount of energy. Consequently, it will be part of our future work to improve datasets exploitation and help promoting energy efficiency through: • The use of novel ML algorithms, including deep learning and generative adversial networks (GAN), which can effectively deal with imbalanced and large-scale datasets. • The development of innovative visualization tools that can cognitively improve end-users comprehension of their consumption behavior. Therefore, interactive visualization tools will be integrated into smart power consumption dashboards that permit individuals to engage with, apprehend their energy usage and translate them into positive actions throughout their everyday life. • The deployment of explainable recommender systems in order to trigger action recommendations at the correct moment, especially if appropriate hardware materials are used to collect and analyze data in a real-time manner. Generative adversarial networks (EM) 3 Endorsing energy efficiency with micro-moments


Detecting Overlapping Data in System Logs Based on Ensemble Learning Method

Machine learning techniques are essential for system log anomaly detection. It is prone to the phenomenon of class overlap because of too many similar system log data. The occurrence of this phenomenon will have a serious impact on the anomaly detection of the system logs. To solve the problem of class overlap in system logs, this paper proposes an anomaly detection model for class overlap problem on system logs. We first calculate the relationship between the sample data and the membership of different classes, normal or anomaly, and use the fuzziness to separate the sample data of the overlapping parts of the classes from the data of the other parts. AdaBoost, an ensemble learning approach, is used to detect overlapping data. Compared with machine learning algorithms, ensemble learning can better classify the data of the overlapping parts, so as to achieve the purpose of detecting the anomalies of the system logs. We also discussed the possible impact of different voting methods on ensemble learning results. Experimental results show that our model can be effectively applied in a variety of basic algorithms, and the results of each measure have been improved.

Introduction
With the increase of devices such as servers and sensors, the amount of data has explosively increased. The system administrators monitor the status of the log data to ensure the normal operation of the system. Due to the large amount of system log data, using machine learning to detect system log data is the mainstream trend today. However, it is likely that the sample data of different categories have the similar attribute values in the process of processing and analyzing data. It is difficult for classifiers or classification algorithms to divide decision boundary in these data distributions. Because the sample data is too complex to clearly define the class boundary, the problem caused by this situation is often referred to as class overlapping.
Due to the continuous evolution of classification models, the high performance of the models masks many problems. The phenomenon of class overlapping in the data is one of the problems that are easily overlooked. In the past few years, there have been some related studies on the processing of class overlapping data. It can be divided into the following situations.
(i) Identification in the data preprocessing: Liu [1] proposed partial discriminative training (PDT) program. In order to reduce the impact of the data in the overlapping parts of the class on the performance of the classifier, only part of the data is labelled in the preprocessing part. Not only does this change the original data, it is also very time-consuming (ii) Manual identification: Sit et al. [2] used soft decision to solve the class overlapping problem. Assign multiple labels to the data that fall into the class overlapping area. The system administrator analyzes and makes judgments based on these options. Since each data that falls into the class overlapping area is manually judged, labour and time costs are relatively high. Tang et al. [3] combined soft decision-making with optimized overlapping area detection algorithms to balance accuracy and soft decision costs. However, this approach is still inefficient in the era of big data (iii) Fuzzy theory: Szmidt and Kukier [4] adopted fuzzy classifier to identify the class overlapping data and then expressed it with the intuitionistic fuzzy sets. Dabare et al. [5] used deep learning and fuzzy membership together (iv) Machine learning: Fu et al. [6] and Debashree et al. [7] took Support Vector Machines (SVM) to process class overlapping data. Xiong et al. [8] used the naive Bayesian detection method to distinguish the class overlapping areas and nonclass overlapping areas. Zhang et al. [9] improved the KNN algorithm. The modified method can not only find the k-nearest neighbors (even the test object itself) of each sample in the training data set but also find the neighbors of the unknown test object. Lee and Kim [10] divided the data space into soft overlapping areas and hard overlapping areas and used SVM decision boundaries and KNN to classify the separated spaces. To reduce the complexity of the data, Sáez et al. [11] decomposed the problem into several binary classification problems, and each classification only judged the current subproblem. Bogucharskiy and Mashtalir [12] and Gong et al. [13] adopted clustering to solve the problem of class overlapping. The most common algorithm is C-means. Dabare et al. [5] integrated deep learning and fuzzy membership into the C-means At the same time, the problem of class overlap has hot research in the fields of speech recognition [14,15], biomedicine [16], credit card fraud detection [17], and software defect prediction [13]. However, there is no relevant paper on the class overlap problem in HDFS data anomaly detection.
In the previous studies, identifying overlapping data in data preprocessing or manual identification is too timeconsuming to achieve an effective balance between accuracy and performance. The traditional machine learning method will make the data detection classification prefer to the classification with large amount of data. The main components of our model consist of two parts: separate data from overlapping areas and use ensemble learning to detect anomaly system log data. System log data in nonclass overlapping areas is easier to be successfully identified and classified. When there is a large amount of nonclass overlapping data in the system log data set, even if all the class overlapping data are misclassified, the anomaly detection model can still achieve an acceptable accuracy. In order to reduce the impact of the nonclass overlapping part of the system log data on the anomaly detection model, we adopt the combination of fuzzy sets and KNN to separate the phenomenon of class overlapping and nonclass overlapping. First, we calculated the rela-tionship between the test sample data and the membership of different classes. Then, we used the fuzziness to separate the data of the class overlapping areas from the data of the nonclass overlapping areas. The data in the class overlapping areas is regarded as the key part.
In 1985, Keller et al. [18] proposed to use fuzzy set theory in combination with KNN. They assigned membership to each classification output test sample data, with a membership interval of 0 to 1. The closer the test sample data to 1, the greater the probability that the test sample data was classified correctly. However, as the amount of data increases, it costs a lot to calculate membership for each test sample data. Taneja et al. [19] improved the fuzzy KNN algorithm to reduce complexity and calculation time. Maillo et al. [20] also used large data sets to run fuzzy KNN.
Boosting is the most representative tandem classification algorithm of ensemble learning. The original Boosting was proposed by Schapire [21] in 1990 and described a method for transforming a weak learning algorithm into a highprecision model. This method is aimed at using this method as a general tool in practice to transform any weak learning classification algorithm into a high-performance classifier. However, AdaBoost [22] no longer needs to give prior information of weak classifiers such as performance parameters, and the algorithm can dynamically adapt the accuracy of each basic algorithm in an adaptive way and apply multiplicative weight update technology to derive new enhancement algorithms. Freund and Schapire [23] designed the AdaBoost above to study the effect of pseudoloss on the actual learning problem in multiclassification problems and set up two sets of AdaBoost and Bagging experiments for performance comparison using multiple weak classifiers. The experiment confirmed that the adjustment of the sample distribution has a positive effect on the enhancement algorithm. Today, AdaBoost has become the most widely used and most representative ensemble algorithm in Boosting.
AdaBoost, an ensemble learning approach, is used to detect overlapping data in detecting anomaly data by using ensemble learning. We use three different types of traditional machine learning methods, logistic regression, decision tree, and naive Bayes, as anomaly detection algorithms. Then, we use AdaBoost to compare with these three machine learning methods. Compared with machine learning algorithms, ensemble learning can better classify the data of the overlapping parts, so as to achieve the purpose of detecting the anomalies of the system logs.
Our contributions are as follows.
(1) For the first time in the HDFS data set, the problem of overlapping of system logs in anomaly detection is proposed. (2) In order to reduce the impact of class overlapping on system log anomaly detection, this paper proposes a class overlap model for system log anomaly detection based on ensemble learning. The model uses fuzzy KNN to separate the data in the class overlapping areas and uses AdaBoost to detect system log data. (3) Compared with other methods, our model can reduce the impact of nonclass overlapping data on the experiment. (4) We use the HDFS system log data set for experiments and compare the experimental effects of AdaBoost and traditional machine learning algorithm to detect anomalies. The result shows that 2 Wireless Communications and Mobile Computing the fuzzy k-nearest neighbor confirms the existence of class overlapping in this data set, and the effect of anomaly detection on system logs using ensemble learning has been significantly improved. Figure 1 shows the algorithm flow of experiments. The following is the introduction of each method.

Fuzzy KNN.
KNN is one of the most common algorithms in the field of machine learning. It was first proposed by Fix and Hodges in 1951. By searching for the k-nearest neighbors to the test sample data, the classification of the test sample data is determined according to the classification of most neighbors among the k neighbors. Different from linear classification, logistic regression, and other algorithms, the KNN does not have a clear formula that can represent the decision boundary. Whenever the data distribution cannot be identified or accessed in many physical applications, a nonparametric method such as KNN is required. Fuzzy KNN is one of the extensions of KNN, which overcomes uncertainty in classification. Fuzzy KNN no longer outputs its classification when predicting the classification of the test sample data, but outputs the degree of membership of the test sample data for each classification, as the following formula: where ðμ i ðxÞÞ i=1,2,⋯,c ∈ ½0, 1 represents the membership value of the test sample x belonging to the i-th classification. ðu ij Þ j=1,2,⋯,K ∈ ½0, 1 represents the i-th data of the j-th vector of the training sample set. The assignment membership of x is influenced by the reciprocal of the distance from the nearest neighbor and its membership. Variable parameter m weight can be adjusted.

Fuzziness.
In 1968, Zadeh [24] first proposed the word fuzziness, that is, objects cannot be described by a clearly defined set of points. De Luca and Termini [25] proposed that fuzziness is an uncertainty related to the situation described by fuzzy sets, and a quantitative measure of fuzziness is defined by nonprobabilistic entropy that does not use any probability concept. For the first time, they explicitly proposed three attributes that the fuzziness measure should satisfy. These attributes indicate that when all members are equal to 0 or 1, the fuzziness should reach its maximum and minimum. According to the above research, Wang et al. [26] made the following formula definition for the fuzziness: The test sample data is calculated by formula (1) μ i , which is the membership value of the test sample data belonging to the i-th classification. The fuzzy set B = fμ i , μ 2 , ⋯, μ n g is formed, and after derivation, the formula is obtained: Therefore, the fuzziness reaches the maximum when μ i = 0:5.
2.3. AdaBoost. Boosting, also known as reinforcement learning, is an ensemble learning method used to improve the accuracy of weak classification algorithms or classifier. Ada-Boost is the most representative and widely used algorithm in the Boosting series. Freund and Schapire [23] selected the weak classifier with the smallest weight coefficient from the trained weak classifiers to form a final strong classifier by adjusting the sample weights and weak classifier weights under the framework of the Boosting problem.
A train set X = fx 1 , x 2 , ⋯, x n g is given. Each sample data in the training set will correspond to a label l i , L = fl 1 , l 2 , ⋯, l n g. Initialize the weight distribution for each sample D m = fw m1 , w m2 , ⋯, w mn g. As shown in formula (4), the weak classifier f m trained after T times finally obtains the strong classifier F T . Calculate the error function ε m of this iteration based on the output set: 2.4. Basis Algorithm. Logistic regression, decision tree, and naive Bayes are used to detect the system log data with class overlapping. Logistic regression (LR) separates data with two labels as much as possible by fitting a line. During the test, the feature vector of the unknown tag data is input to obtain the tag of the data. If the test data are farther from the fitted line, the probability of belonging to a certain type of tag is greater.
Decision tree (DT) is a kind of tree structure algorithm, which uses the value of the test sample data as a branch. Each internal node of the DT can represent the judgment of an attribute, while each branch represents the judgment result, and each leaf node represents a classification result.
Naive Bayes (NB) is a classification method based on Bayes' theorem and the independent assumption of feature conditions. Unlike other classification algorithms, the NB mathematical theory is very mature. By assuming that the sample condition attributes are independent, the posterior probability results are obtained according to the prior probability and test sample data. 3 Wireless Communications and Mobile Computing 2.5. Voting. Voting is commonly used for data classification, which requires a combination model of at least two algorithms. Each algorithm has its own learning strategy and prediction method, so different algorithms may have different prediction results for data. Hard voting obeys the majority voting method according to the result of minority classification. For the binary classification problem, the number of algorithm combination models must be odd number. Soft voting uses a weighted average of algorithm classification probabilities to predict results. Anomaly detection of system logs is a binary category problem, where N is normal log data and A is abnormal log data. Algorithms 1 and 2 give the calculation process of hard voting and soft voting, respectively. Compared with soft voting, the disadvantage of hard voting is that if the test sample data evades the detection of most machine learning algorithms, although there are a few algorithm classifiers that successfully detect and classify, the results will still tend to vote for most algorithms. Soft voting uses this data to assign the predicted classification probability in machine learning algorithm detection and weights it to average. Different algorithms have different learning and classification strategies. This advantage is that when it is faced with the data of the class overlap area, it can not only make the classifier with a larger decision-making grasp play a better effect but also avoid the judgment error when the classifier decision boundary is blurred.

Experimental Data and Evaluation
Measures. This experiment uses 3.1 GHz Intel Core i5 processor, 8 GB RAM, and macOS operating system. The experimental data is a 1.58 G HDFS_1 data set provided by the Chinese University of Hong Kong, which is extracted on Amazon EC2 platform. HDFS is used to store data and manage data in distributed computing. HDFS uses Block ID to record file storage,   It is easy to cause the occurrence of class overlap phenomenon. In data preprocessing, we distinguish the fixed text and variable parts in the raw log. We take the uncensored fixed text as structured log and use Event to correspond to structured log. As shown in Table 1, there is a message raw log "Verification succeeded for blk_490" converted to structured log: "Verification succeeded for < * > ," and Event3 is used to correspond to it [27]. Count each Event according to the Block ID (such as blk_490) in HDFS. The data set is recorded in chronological order, divided by the ratio of 80% of the training set and 20% of the test set. The experimental evaluation measures use secondary evaluation measures based on confusion matrix: accuracy, precision, recall, and F1 value. These methods are used to evaluate the effectiveness of the algorithm for detecting class overlapping.

Accuracy
The MCC returns a real number between -1 and 1. -1 means that the prediction is completely wrong, 1 means perfect prediction, and 0 means no better than random prediction.

Class
Overlapping. Because a part of system log data which are different types have very similar attributes, they cannot fit decision boundary in parameter-based algorithms. Therefore, fuzzy set theory and nonparametric KNN are combined to calculate test sample data and membership in different classifications. Set the variable parameters in KNN to K = 11, m = 2. It was found that with the increase of fuzziness, the probability of error also increased, and the test sample data fuzziness of all classification errors are above 0.65.
As shown in Table 2, the fuzziness of the test sample data is divided into two groups with 0.65 as the boundary for anomaly detection. The accuracy of the test sample data less than or equal to 0.65 is significantly higher than that of the data with fuzziness greater than 0.65. What is more, this phenomenon is more obvious in the fuzziness KNN. The anomaly detection accuracy of the test sample data with fuzziness less than or equal to 0.65 can reach 0.971, while that of the data with fuzziness greater than 0.65 is only 0.333.
As fuzziness increases, the lower the accuracy of anomaly detection, the higher the probability of class overlapping of data. We use TSNE to reduce the dimension and visualize the test set data in order to more intuitively observe the data distribution. Figure 2(a) shows the distribution of all system log data in the test set, and the data is deduplicated. The data was deduplicated to reduce the impact of large amounts of duplicate data. Figures 2(b) and 2(c) show the distribution of test sample data with fuzziness greater than 0.65 and less than 0.65. The distribution of test sample data shown in Figure 2(c) is relatively regular, which can divide the decision boundary easily. However, the phenomenon of data class overlapping appears in Figure 2(b). Therefore, we separated the test sample data with fuzziness greater than 0.65 for key research. All the test sample data below are class overlapping area data with fuzziness greater than 0.65. Table 3 shows the results of three traditional machine learning methods on class overlapping phenomenon data and filtered data with fuzziness less than 0.65. We can find that the accuracy scores of LR, DT, and NB on filtered data are all higher than the scores on class overlapping phenomenon data. After removing the overlap log data, the accuracy of log anomaly detection of all the above algorithms increases. The increase of DT accuracy score is much significant from 0.857 to 0.971. Like accuracy results, the results of precision, recall, F1, and MCC are generally higher in three algorithms on the data without class overlapping phenomenon. This shows that filtering out class overlapping data is very necessary for anomaly detection. Table 4 shows the results of anomaly detection using AdaBoost on class overlapping phenomenon data and filtered data with fuzziness less than 0.65. AB-LR, AB-DT, and AB-NB are upgraded LR, DT, and NB methods with AdaBoost. As a result, it was found that performance of all machine learning algorithms is generally improved after ensemble. The anomaly detection accuracy of AD-DT reached to 0.952, and the lowest accuracy score is 0.857 using AB-NB. After ensemble, the recall scores of LR and NB have some decline, but the F1 scores have a significant increase. Like Table 2 results, the performances of the three methods are generally higher on the data without class overlapping phenomenon. The results of anomaly detection based on voting in the area of the class overlap phenomenon are shown in Table 5. The prediction result of hard voting is the same as the classification result score of anomaly detection using logistic regression algorithm in AdaBoost. As stated in algorithm voting, hard voting will tend to vote for the best algorithms. Where the probability difference of the algorithm classification result is very small is not considered by hard voting.

Comparison of Results.
Hard voting only considers the voting results of each algorithm. The accuracy of soft voting is 0.952, which is like the accuracy of the decision tree. Compared with the highest accuracy in a single classifier, there is no improvement, while comparing the accuracy and recall rate, the three basic algorithms of logistic regression, decision tree, and naive Bayes have different anomaly detection strategies, and the results are also different. Soft voting's probability-weighted voting changed the final prediction results, but unfortunately in this model, some test sample data became correct after weighted voting classification, and some became wrong.
There are some other research works [28,29] introducing ensemble learning algorithms on the HDFS data set. We reproduced the DLME [28] according to the description in its paper. Table 6 shows the results of DLME on the original HDFS data set and the data set with fuzziness less than 0.65. The test results prove that without phenomenon of class overlapping in the HDFS data set could significantly improve the performance of DLME. The F1 score of DLME has a dramatic increase from 0.711 to 0.982. DLME also has a distinct improvement in MCC from 0.609 to 0.919.
We also compared the performance between DLME and AB-DT on the original HDFS data set which contains class overlapping phenomenon. As shown in Figure 3, the AB-DT method has higher scores than DLME on the accuracy, precision, recall, F1, and MCC.

Case Study.
In order to find the difference of detection between the two algorithms, we analyzed the data set and select two of the data as case study. In the experiment, the feature points of data A are [4, 1, 3, 3, 6, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]. And the feature points of data B are [4, 1, 3, 3, 5, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,      Wireless Communications and Mobile Computing 0, 0, 0, 0, 0, 0, 0, 0]. The difference between the two data is only in the fifth feature point, and the other feature points are the same. The real label of data A is anomaly, and the real label of data B is normal. Data A and data B have a class overlap phenomenon. In the experiment, the anomaly detection result of data B was normal. Data A is detected as normal data when using traditional algorithms, but the true label is anomaly. It was successfully detected as anomaly data when using AdaBoost.

Conclusion
This paper uses ensemble learning to detect system log data anomalies in which class overlapping occurs. Firstly, the combination of fuzzy set theory and KNN confirms the possibility of the formation of class overlapping phenomenon in data set and extracts the data in this area for key processing. Compared to machine learning algorithms that fit decision boundary, nonparametric KNN can calculate the mutual distance relationship between data. The combination of fuzzy set theory and KNN can calculate the membership relationship of each classification of test sample data. In order to reduce the impact of nonclass overlapping area data on the detection results, the fuzziness is calculated according to the data classification membership and the data is divided. Ada-Boost, an ensemble learning algorithm is used for anomaly detection of class overlapping data. Experimental results prove that the higher the fuzziness in the log data, the greater the probability of error. Using TSNE to visualize the dimensionality reduction of the system log data, it was found that the class overlapping phenomenon does exist. The experimental effect of AdaBoost is better than traditional machine learning in each evaluation measure. The class overlapping anomaly detection model based on ensemble learning is successfully applied in the HDFS data set, which can accurately detect the class overlapping area data. However, our work has just begun. Our future work is to solve the problem of class overlapping by studying the rela-tionship between each feature point of high-dimensional data in data preprocessing.

Data Availability
The research data supporting the results of this study can be available from https://zenodo.org/record/3227177# .XvVGE20zbIU.

Disclosure
An earlier version of this manuscript has been presented as conference presentation in the 2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC).