Catalog of Solar Failed Eruptions and Other Dynamic Features Registered by SDO/AIA

In this paper we present our attempt to constrain the first catalog of solar failed eruptions. We used our automatic algorithm that is able to search for dynamic features in the Solar Dynamics Observatory (SDO)/Atmospheric Imaging Assembly (AIA) database. We ran the algorithm on the entire SDO/AIA 171 Å data set. For the time interval from 2010 May 20 to 2019 May 20 we found 12,192 dynamic events. The dynamic events were classified in three groups. Apart from failed eruptions (1214) we obtained a large group of successful eruptions (2064) and other dynamic events (8914). The automatic algorithm enabled us to collect several observational characteristics, which are provided in files that may be downloaded from the catalog web page. In this paper we present the methodology of catalog preparation and preliminary results of the statistical analysis of observational characteristics obtained by the automatic algorithm.

Introduction
The solar atmosphere is a dynamic environment with different types of moving structures. These structures occur on various time and spatial scales. From terrestrial perspective we are mostly interested in the largest events, which may significantly influence the Earth's magnetosphere causing various types of threats (Pulkkinen 2007). For this reason coronal mass ejections (CMEs) are intensively investigated (Webb & Howard 2012;Harrison et al. 2018), automatically searched (Robbrecht & Berghmans 2004;Morgan et al. 2012;Hess & Colaninno 2017;Patel et al. 2018), cataloged (see links and references in Richardson et al. 2015), and reasonably predicted (Song et al. 2006;Qahwaji et al. 2008;Hess & Zhang 2015;Kay et al. 2017;Kay & Gopalswamy 2018). However, the relationship between strong solar flares and CME occurrence is weak and there are many exceptions that do not follow a strong flare-strong CME pattern. Even the strongest, X-class flares may not be accompanied by CMEs (Wang & Zhang 2007). Apart from individual events the Sun can surprise us with huge active regions producing dozens of M-and X-class flares, which are completely CME-less (Chen et al. 2015;Sun et al. 2015). This suggests that we do not know all or at least key physical conditions in whole active regions as well as in individual flares, which determine whether a certain magnetic field reconfiguration will eventually result in a CME occurrence or not. Therefore, we need to investigate in details events that do not develop into CMEs, even if their properties (i.e., magnetic field complexity, power of a flare, etc.) favor the appearance of such.
Two types of events should be investigated from this point of view. The first are confined flares (Schmahl et al. 1990), which do not produce a CME and are not accompanied by any dynamic structures. The second are failed eruptions (Ji et al. 2003), which are a type of dynamic structure that are abruptly stopped in the solar corona after initial acceleration and upward movement. Gilbert et al. (2007) defined the failed eruption as an eruption for which neither magnetic field nor matter escapes the Sun thus no CME is observed. Sometimes, these two types of events are mixed because a flare without a CME but accompanied with a failed eruption can be still classified as a confined flare. However, generally speaking, not every confined flare is connected with a failed eruption and vice versa.
We decided to concentrate on failed eruptions because they start as typical eruptions that should eventually develop into a CME but do not for unclear reasons. The statistical analysis of a large group of failed eruptions should refine boundary conditions for CME occurrence. There are several scenarios that may explain the confinement of the eruption: too strong tethers (Vrsnak 1990), dynamic magnetic tension force in an erupting structure (Myers et al. 2015), flux emergence complexity (Archontis et al. 2007), kink instability leading to a stable configuration of an erupting flux tube (Török et al. 2004), reconnection with a magnetic field overlying the eruption site (Amari & Luciani 1999), and others.
From the observational point of view, before the Solar Dynamics Observatory (SDO)/Atmospheric Imaging Assembly (AIA; Lemen et al. 2012), failed eruptions were observed and analyzed only occasionally. One of the most often analyzed events is a failed eruption that occurred on 2002 May 27 (Ji et al. 2003). This is an example of an eruption stopped by a kink of a magnetic flux rope. During the eruption braking of the reconnection inside the eruption took place (Alexander et al. 2006). Additionally the overlying field decreased slowly with height, which was found to be an important factor leading to the eruption confinement (Török et al. 2004). In recent years an overlying magnetic field decay index became the most investigated observational feature connected with failed eruptions and/or confined flares (Moore et al. 2001;Liu 2008;Guo et al. 2010;Baumgartner et al. 2018).
Failed eruptions may also interact with an overlying magnetic field. This interaction leads to an occurrence of rare type oscillations (Mrozek 2011) or the reconnection with the overlying field (Amari & Luciani 1999). The signs of electrons accelerated due to such an interaction were analyzed in three cases of failed eruptions by Netzel et al. (2012). The authors found one example of a hard X-ray (HXR) source produced by electrons accelerated during the failed eruption interaction with overlying fields. We expect that such weak sources should be more frequently observed by more sensitive future instruments like the Spectrometer/Telescope for Imaging X-rays (Krucker et al. 2013), which will observe the Sun from a short distance. The overlying magnetic field role in flare confinement was also discussed by Wang & Zhang (2007). In their work the other parameter was investigated, i.e., the location of the flare in an active region. The authors found that flares are more pronounced to be eruptive if located closer to the edge of an active region, while confined events were located closer to the center.
The existing analyzed samples of failed eruptions are usually small. In most of published papers the results are presented for one carefully selected event (e.g., Ji et al. 2003;Kumar et al. 2011;Mrozek 2011;Chen et al. 2013;Song et al. 2014;Kushwaha et al. 2015;Hassanin & Kliem 2016;Li et al. 2018;Song et al. 2018). From this point of view it is impossible to investigate statistically processes that lead to eruption confinement. The main reason is that the database of SDO/AIA observations is huge, and automatic search for events of a given type is slow and uncertain. Until now only one algorithm for an automatic search for solar eruptions in SDO/AIA database has been constructed (Hurlburt 2015). In this method the authors search for optical flows based on the Fourier analysis restricted to the first few elements. Therefore, the algorithm is optimized to search for large structures. The results were compared with the Heliophysics Events Knowledgebase (HEK; Hurlburt et al. (2012)). The comparison showed that the algorithm is far more effective than a human. The HEK database contains 75% fewer entries than the automatic algorithm. The authors explain this result as a product of moments of inattention or subjective selection of events-not all are considered as interesting by a human. The authors showed that a human has less sensitivity for fast, short-lived eruptions while the automatic algorithm performs worse for non-slow, long-evolving eruptions. It has to be mentioned that the algorithm searched for all types of eruptions and was unable to automatically detect failed ones.
In this paper we present our attempt to constrain the first catalog of failed eruptions. We used the automatic algorithm that is able to search for dynamic features in the SDO/AIA database. Next the found dynamic events were classified in a few groups. One of them contains failed eruptions for which we determined several observational characteristics. All the events are collected in the online catalog.

Observational Data
The SDO (Pesnell et al. 2012) was launched on 2010 February 11. It carries three experiments. One of them is the AIA (Lemen et al. 2012) dedicated to image the entire solar disk with the use of several wide-band filters. AIA consists of a set of four 20 cm, normal incidence telescopes. Their 41′ field of view is covered with 4096×4096 charge-coupled devices (CCDs), giving the angular resolution of 1 5. Images are taken in 10 bands (including 7 extreme ultraviolet (EUV) bands) with 12s time resolution. Therefore the amount of data produced by the instrument is enormous: 1.5 TB per day. It is impossible to run an automatic algorithm that downloads and analyzes original full-resolution data in a reasonable time. For this reason we decided to use images reformatted to the synoptic 1.5 level that are the full-disk images compressed to 1024×1024. The time cadence of synoptic 1.5 level data is 2minutes. Such data provide time and spatial resolutions that are very suitable for searching different types of moving structures.
We restricted our search to the 171Å band. However, the developed algorithm may be easily adopted for other bands and other instruments. The characteristic of AIA 171 filter is suitable for searching hot (above 10 6 MK) as well as cold (around 3×10 5 K) plasma structures. It was important to have both hot and cold components mixed because the erupting plasma may be heated or cooled and become invisible in filters responding for cold or hot plasma only.
Downloading and processing of the data was performed automatically. We decided to perform our search on data packets covering 3 hr of observations. The data packets had 15 minutes overlap to reduce edge effects. This overlap worked well for short events, but longer ones were often divided into two when occurred on the border between two data packets. Every not a number (NaN) pixel was replaced with the use of IDL's CONVOL function. The last step of data cube preparation was blurring of each image with a Gaussian filter (FWHM=2.8).

Automated Search for Dynamic Events
Each 3 hr long data packet searched for eruptive pixels according to the algorithm described in Mrozek et al. (2016). We used Bayes approach to classify pixels into two groups: eruptive and quiet pixels. The search was performed on various areas with different sizes and shapes. This strategy slowed the algorithm but allowed us to find large as well as small eruptions. Complexes of eruptive pixels found were classified as a dynamic structure if several conditions were fulfilled.
1. Each pixel had a probability of being eruptive greater than 0.35. 2. The structure was visible on eight or more consecutive frames. 3. The area covered by the eruptive pixels complex was greater than 600 arcsec 2 on at least one frame. 4. The structure mean brightness was above 30 digital number (DN) on at least one frame 5. The structure revealed the change of centroid position greater than 25″.
The algorithm is moderately effective in searching for dynamic events in large data sets (Mrozek et al. 2016). However, it performs searches in a reasonable time. The search performance depended on the solar activity level. The analysis of one month of SDO/AIA observations lasted from 3 to 10 days. We run the algorithm on up to three personal computers, which means that the search of the whole 171Å image database finished in 10 months.
Apart from eruptions, the algorithm is very sensitive to motions of faint plasma structures in the solar atmosphere like expansion, untwisting, oscillations, and globally propagating waves in the solar corona (commonly called "EIT waves"). In the case of large eruptions, when even the entire solar disk is disturbed, the algorithm could find an eruption and a variety of small disturbances that were related to the eruption. Therefore, the number of small events may be overestimated and should be interpreted with care.
When a dynamic event was detected the algorithm produced two types of outputs. The first is a three-panel plot. An example is presented in Figure 1. It contains a 171Å image, the same image with overlay contour presenting the eruptive pixels complex, and the differential image. The second output is an External Data Representation format file containing a structure with various parameters obtained by an automated algorithm for a given dynamic event. Table A1 (in the Appendix) contains the description of each field in the structure.
We ran the algorithm on the entire SDO/AIA data set. For the time interval from 2010 May 20 to 2019 May 20 we found 12,192 dynamic events. The distribution of events on the solar disk is presented (see Figure 2, left panel). They are concentrated along active latitudes, but there is a significant fraction of events located closer to the poles. The right panel of Figure 2 presents the dynamic events' frequency in 120 day long time bins. Two maxima are visible around the years 2012 and 2015. These are time correlated with the distribution of the Wolf numbers during the 24th cycle presented with the dotted line.

Classification of the Events
The solar atmosphere is full of various types of static and dynamic structures. A large number of various classification schemes exists depending on the wavelength, morphology, kinematics, spectroscopy, spatial scale, etc. (e.g., Tomczak & Chmielewska 2012;Zhang et al. 2012;Freed & Russell 2014;Nicewicz & Michalek 2016;Barnes et al. 2017;Paraschiv & Donea 2019). We concentrate on moving structures therefore we refer to Gilbert et al. (2007) who classified eruptions depending on their ability to escape from the Sun. Namely they divided eruptions into three types: full, partial, and failed. The first is a type of an eruption in which the entire (more than 90%) magnetic structure and/or plasma escapes from the Sun. The partial eruptions are further divided into: the "type a" when the entire magnetic structure erupts with a small amount or without plasma, and the "type b" for a partial eruption of the magnetic structure with a small amount or without plasma. The third type is a failed eruption whose magnetic structure filled  with plasma, after initial acceleration, stops in the low corona. These eruptions may be related to a CME occurrence. The full and partial eruptions are related to CMEs, while the failed eruptions do not produce CMEs. This kind of an eruption-CME relation is sometimes confused with another classification which raised from the flare-CME relation (Pallavicini et al. 1977) who divided flares into two groups: flares accompanied by CMEs (eruptive) and not accompanied (confined).
We merge the existing classification of flares and eruptions by collecting them into one diagram presented in the the Figure 3. We extended the eruption classification of Gilbert et al. (2007) with one additional type of eruption. These are confined jets (Wyper et al. 2016) which may be treated as a type of a failed eruption in which plasma motion is observed without visible dynamics of the magnetic structure. Namely we observe only plasma moving along the magnetic structure. This is a type of an event that is similar to a surge in Hα observations (Engvold 2015). The internal tiles in Figure 3 refer to eruptive and confined flares. An eruptive flare may be accompanied by a full or a partial eruption, but there is also possible that the full or the partial eruption occurs without a flare. That is the reason why the tiles representing full and partial eruptions are slightly larger than the flare tiles. A similar situation is for confined flares that may be accompanied by a failed eruption or a confined jet. However, there are failed eruptions and confined jets which occur without visible flares.
The confined jets have been excluded from the failed eruptions group as they may represent physically a different group. Vršnak (2019) analyzed the analytical flux tube model to investigate the pre-eruptive phase of solar eruptions. The author distinguished three processes leading to the eruption: twisting motions of the flux-rope foot points, emergence of a new magnetic flux beneath the flux rope, and a mass leakage down the flux-rope legs. In our morphological classification the first refers to confined jets, and the second to failed eruptions. We do not take into consideration the mass draining since it may occur in any type of an eruption. However, apart from a naming convention, we treat failed eruptions and confined jets as eruptions that do not escape from the Sun. Therefore, in the catalog, they both are collected in one group named "failed eruptions," but for confined jets we include information in a "comments" file.

Catalog Content
The classification of the dynamic events has been done by a human. We decided to divide the dynamic events into three groups.
Each event from this group can be related to a CME from the SOlar and Heliospheric Observatory (SOHO)/Large Angle Spectrometer COronagraph (LASCO) CME catalog. 4 3. Failed eruptions: both failed eruptions and confined jets are included in this class. We restricted the number of confined jets to events associated with flares.
The present version of the catalog 5 contains data for the time period from 2010 May 20 to 2019 May 20; this version is archived in the RepOD repository at [doi:10.18150/F0BC6C] under a CC0 Creative Commons Zero 1.0 license. There are two types of catalog entries. The first is dedicated to small/ unclassified and successful eruptions. We provide the information for them obtained from automatic algorithm results only. It means that there is a plot containing location of an event, a time history of centroid location, intensity, and an area of the eruptive pixels complex. Additionally we include a movie showing animations of triple panels (an example of a single frame is presented in Figure 1) and a file with all parameters estimated by the automatic algorithm. A description of the parameters is presented in Table A1 (in the Appendix).
The more comprehensive catalog entry is dedicated to failed eruptions only. Here we provide more detailed information concerning kinematics. Using automatic algorithm results we calculate a position of an eruption front. The obtained time evolution of the altitude of the eruption front is spline smoothed and its first (velocity), and second (acceleration) derivatives are calculated. Additionally, for each failed eruption, we estimate the eruption front position manually, and derive kinematic parameters as well. The results obtained are presented on estimated height and velocity plots. Moreover, we include the RHESSI quick look plot for HXRs in the following energy channels: 6-12, 12-25, 25-50, 50-100, and 100-300 keV. The maximum values of speed, deceleration, and altitude are also presented in the catalog entry. For more details, please refer to the help page of the catalog. 6

Observational Characteristics of the Events
Among the dynamic events we have classified 1214 failed eruptions, 2064 successful eruptions, and 8914 small/unclassified events. In Figure 4 we present histograms of some observational characteristics estimated with our automatic algorithm.
1. Direction-an angle between the eruption front direction and the radial direction. 2. Duration-a time span of a dynamic feature defined by parameters described in Section 3. 3. Latitude-the heliocentric Y coordinate of a dynamic feature. It has to be mentioned that the small/unclassified group is not uniform. For example, when a large eruption occurred, disturbing almost the entire visible corona, then the automatic algorithm found dozens small dynamic features like side eruptions, oscillations, waves, etc., connected with this large eruption. Moreover, this group contains false positives, which are brightenings without any dynamic behavior. We counted such events for the period of 2012 April 1-2013 April 1. Thus, we may expect that around 4% of small/unclassified events are false positives, which is less than 3% of all events found by the algorithm. Nevertheless, we do present the results for small/ unclassified events for completeness. The first row in Figure 4 presents directions of eruptions with regard to the radial direction. It can be seen that the successful eruptions are mainly ejected in the radial direction. They are concentrated around direction 0°. Directions of the failed eruptions and the confined jets are spread between 0°and 90°. We expect that this is mainly connected with more numerous weak eruptions, and inclusion of the confined jets which move along magnetic field lines. Directions of the small/ unclassified events are even more spread with the maximum visible around 70°. We expect that such an effect is caused mainly by geometrical effects since we have relatively many events close to the solar limbs.
The second row in Figure 4 presents durations of the detected events. About 30% of the small/unclassified events are concentrated in the interval 20-30 minutes. We may expect power-law distribution of the durations of small events. Thus, majority of them will occupy the shortest times of the histogram that can be detected by our algorithm. The algorithm was defined to search for events that last at least 16 minutes (eight images taken with a 2 minute cadence), which is quite long for small events. The successful eruptions have the second maximum around a value of 160 minutes. It means that the 3 hr long data packets that we analyzed are too short for some events. We expect that they could be detected in two separate data packets as two separate events. This may produce some bias in data interpretation, but they are less than 7% of all successful eruptions. Distribution of the duration of the failed eruptions and the confined jets is wider than for the small/ unclassified events and slightly narrower than for the successful eruptions. It can be seen that above 100 minutes the population of events is stable in each time bin, and there is no maximum around 160 minutes. This suggests that the assumed 3 hr long time window for the analyzed data packets was chosen well and allows us to detect the majority of the failed eruptions as single events without splitting them in two.
The last row in Figure 4 presents the heliocentric Y coordinate of detected dynamic events. All three types are concentrated mainly in active latitudes. Each histogram has a slight asymmetry when comparing both solar hemispheres. Namely we detected more events in the Northern Hemisphere (positive values of Y) than in the Southern one. Moreover, we noticed that the failed eruptions and the confined jets are slightly more concentrated toward the equator than the successful eruptions. The interpretation of this effect is outside the scope of this paper and will be analyzed in further work.

Summary
In this paper we present the first catalog of failed eruptions. We used our automatic algorithm that is able to search for dynamic features in the SDO/AIA database. Additionally, as a side effect we obtained a large group of successful eruptions and other dynamic events that we decided to include in the catalog. The dynamic events were classified in three groups: the small/unclassified, the successful eruptions, and the failed eruptions (merged into one group with the confined jets). The automatic algorithm enabled us to collect several observational characteristic,s which are provided in files that may be downloaded from the catalog website. Apart from the parameters provided by the automatic algorithm we determined more kinematic characteristics for the failed eruptions and the confined jets. All events are collected in the online catalog.
We analyzed several observational characteristics obtained by the automatic algorithm. We found that the successful eruptions are mostly radial eruptions. The failed eruptions and the confined jets show more diverse directions. The histograms containing durations of the small/unclassified and the successful eruptions show some data selection effects. Namely we assumed that events lasting less than 16 minutes or more than 180 minutes are not taken into consideration. The correctness of such an assumption is confirmed by the histogram of durations of the failed eruptions and the confined jets that show almost no data selection effects. The last of the presented parameters is a heliocentric Y coordinate. We detected here some asymmetry between the hemispheres for each of the defined groups. Moreover, we observed that the failed eruptions are more concentrated around equator than successful eruptions.
In this paper we present only the catalog and the general results from the automatic algorithm. Each group of events in our catalog is not homogeneous. Thus, a more sophisticated analysis based on our crude classification of events (small/ unclassified/successful/failed) is not possible. Even the subgroup of failed eruptions (after exclusion of confined jets), on which we focus our attention, cannot be considered homogeneous. As mentioned in Section 1, there are several possible mechanisms, which may be involved in confinement of an eruption. Therefore, a deep analysis of failed eruptions, including their kinematics, energetics, and relation to overlying magnetic fields, needs a proper selection of events first and then their careful examination. Such an analysis will be presented in our subsequent paper.
All events are presented in the online catalog, and may be freely downloaded and analyzed. This is the first such a collection of failed eruptions and confined jets. Apart from them we provide data for successful and small/unclassified events. We hope that this kind of collection may be useful for training the artificial intelligence algorithms to improve their ability in finding events of a given type. We will perform such a test of a new algorithm in the near future.


extfieldsofstudy:
Computer Science
Engineering

text:
Submodular Rank Aggregation on Score-based Permutations for Distributed Automatic Speech Recognition

Distributed automatic speech recognition (ASR) requires to aggregate outputs of distributed deep neural network (DNN)-based models. This work studies the use of submodular functions to design a rank aggregation on score-based permutations, which can be used for distributed ASR systems in both supervised and unsupervised modes. Specifically, we compose an aggregation rank function based on the Lovasz Bregman divergence for setting up linear structured convex and nested structured concave functions. The algorithm is based on stochastic gradient descent (SGD) and can obtain well-trained aggregation models. Our experiments on the distributed ASR system show that the submodular rank aggregation can obtain higher speech recognition accuracy than traditional aggregation methods like Adaboost. Code is available online~\footnote{https://github.com/uwjunqi/Subrank}.

INTRODUCTION
The recent development of computer architectures allows automatic speech recognition (ASR) to be set up in a distributed way [1,2], where the outputs of different deep neural network (DNN)-based acoustic models are somehow combined to obtain higher ASR accuracy. Our previous work on distributed ASR [3] focused on how to set up a distributed ASR system based on several non-overlapping data subsets, which are produced by applying the submodular data partitioning approach. However, the traditional Adaboost [4] was used in that work to combine the outputs of different subsystems from distributed acoustic models for a better ASR performance. This work proposes a novel method on submodular rank aggregation on score-based permutations for combining scores of acoustic models into a final consistent list of scores. Rank aggregation is the task of combining a couple of different permutations on the same set of candidates into one ranking list with a better permutation on candidates. More specifically, there are K permutations {x1, x2, ..., xK } in total, where the elements {X1j, X2j, ..., XNj}, which are in the ranking list xj, denote either relative orders or numeric values for the candidates. In the framework of rank aggregation on score-based permutations, the elements in the ranking lists represent numerical values and a combined list x with the aggregated scores is used to obtain the relative orders for candidates by sorting the values inx.
The Lovasz Bregman (LB) divergence for rank aggregation on the score-based permutations is initially introduced in [5]. The LB divergence is derived from the generalized Bregman divergence parametrized by the Lovasz extension of a submodular function, 1 https://github.com/uwjunqi/Subrank where a submodular function can be defined via the property of diminishing return [6]. The diminishing return suggests that a func- Although general discrete optimization problems are NP-hard, the submodular function allows the discrete optimization problems to be efficiently approximated by heuristic algorithms, e.g., an exact minimization and an approximated maximization [7], in a polynomial time with a performance guarantee. Moreover, the Lovasz extension of a submodular function ensures a convex function [8].
The LB divergence to rank aggregation on score-based permutations is briefly discussed in [5,9], where the LB divergence is capable of measuring the divergence between a score-based permutation and an order-based permutation. This work applies the LB divergence to construct the Normalized Discounted Cumulative Gain (NDCG) [10] loss function for the rank aggregation with an application to the distributed ASR system. The NDCG is a measure of ranking quality to evaluate the effectiveness of ranking algorithms, and it can be taken as an objective function for the rank aggregation of the distributed ASR system. Figure 1 shows a distributed ASR system, where there are 8 well-trained DNN-based acoustic models and the outputs of all models need to be aggregated to enhance the ASR performance. Our previous work [3,11] focused on how to use submodular data partitioning and selecting algorithms to split the full dataset into 8 different non-overlapping subsets, and made use of the subsets to train 8 different DNN-based acoustic models. Then, the scoring lists of 8 acoustic models are aggregated to generate a consensus scoring list based on submodular rank aggregation in this work.
Our contribution in this work includes: (1) the NDCG loss function is constructed based on the LB divergence; (2) linear and Nested structured frameworks are formulated for constructing optimization problems of the submodular rank aggregation; (3) efficient learning and inference algorithms are proposed; (4) the algorithms are applied to a distributed ASR system.
The remainder of the paper is organized as follows: Section 2 introduces the submodular function based on the LB divergence and the related NDCG loss function, and then the linear and nested rank aggregation functions are discussed in Section 3. The experimental results of the submodular rank aggregation for the distributed ASR system are shown in Section 4 and the paper is concluded in Section 5.

LB DIVERGENCE AND THE NDCG LOSS FUNCTION
The LB divergence as a utility function was firstly proposed in [12], and the related definitions for the LB divergence are briefly summarized in Definitions 1, 2, 3.
where V is the ground set with all elements, and |V | is the number of candidates. Definition 2. For a submodular function f and a score-based permutation x, we define a permutation σx such that Definition 3. Given a submodular function f and its associated Lovasz extensionf , we define the LB divergence df (x||σ) for measuring the divergence of permutations between a score-based permutation x and an order-based permutation σ. The LB divergence is defined as (3).
where both h f σ and h f σx are defined via definitions 1 and 2, respectively.
Next, we study how to apply the LB divergence to obtain the NDCG loss function, which is used in the rank aggregation on scorebased permutations.
Corollary 1. For a submodular function f (X) = g(|X|), the LB divergence df (x||σ) associated with f is derived as (6). where is a concave function, |X| denotes the cardinality of the set X, and σx and σ refer to the scorebased and order-based permutations, respectively. It is found that df (x||σ) ∝ L(σ), and the normalized df (x||σ) can be applied as a utility function for the NDCG loss measurement because the normalization term in (5) is constant. Besides, the LB divergence guarantees an upper bound for the NDCG loss function as shown in Proposition 1. Proposition 1. Given a score-based permutation x and a concave function g, the LB divergence df (x||σ) defined as (6) provides a constant upper bound to the NDCG loss function. Specifically, where n is the number of permutation x, ǫ = max x(σx(i))δg(i) is a normalization term, and the upper bound for L(σ) is independent of the permutation variable σ.
Proof. To obtain (7), we firstly show that, given a monotone submodular function f and any permutation σ, there is an inequality (8) for df (x||σ): is a marginal gain. Furthermore, by setting f (X) = g(|X|), we can obtain max j f (j) = g(1), and by applying the submodular diminishing return property min . Thus, by setting L(σ) = 1 Z df (x||σ), and noting that Z ≥ n · min i σx(i)δg(i), the proof for (7) is completed.

THE LEARNING FRAMEWORKS FOR SUBMODULAR RANK AGGREGATION
Section 3 discusses two learning frameworks based on the NDCG loss function as introduced in Section 2, and the associated algorithms are proposed accordingly.

The Linear Structured Framework
The first learning framework based on the LB divergence is built upon a linear structure. Suppose that there are |Q| queries and K score-based permutations {x q 1 , x q 2 , ..., x q K } associated with the query q ∈ Q, ∀q ∈ Q, and π q ∈ {1, 2, ..., N }. The LB divergence df (x q i ||π q ) computes the divergence between x q i and π q . The problem focuses on how to assign the weight vector w = (w1, w2, ..., wK ) T to df (x||π q ) = (df (x q 1 ||π q ), ..., df (x q K ||π q )) T . The objective function of the linear structure is formulated in (9), where λ denotes a regularization term.
The stochastic gradient descent (SGD) algorithm is applied to obtain ∇ sgd as (10), and update the parameter w based on (11) to satisfy the constraints in (9).
The update of (11) goes through all i = 1, ..., K and each query q ∈ Q. Several iterations are repeated until w gets convergence.
Proposition 2. In the linear structured framework, given the number of candidates N and the number of permutations K, the computational complexity in the training stage is O(N K).
The inference process is formulated as follows: given a test data {q, N, K, X}, where q is a query, N and K have been defined as above, and X = {x q 1 , ..., x q K } represents the score-based permutation associated with the query q, we estimate an optimal order-based permutationσ as defined in (12).
where w * refers to the weight vector trained in the training stage. Generally, the problem in (12) is an NP-hard combinatorial problem, but the LB divergence provides a close-form solutionσ = σµ, where µ = K i=1 w * i x q i . The complete inference algorithm is shown in Algorithm 1. Note that the order-based permutationσ is finally obtained by sorting the numerical values in RX and the permutation associated with RX is returned in a decreasing order.

The Nested Structured Framework
The linear structured framework involves several potential problems: the first problem is that the score-based permutations {xi} K i=1 may not interact with each other, since one permutation might be partially redundant with another; the second problem lies in the fact that a permutation xi tends to become dominant over the rest. To overcome these problems, an additional hidden layer is utilized to construct a nested structured framework.
The objective function for the nested structured framework is formulated as (13), where K1 and K2 separately represent the numbers of units in the input and hidden layers, W1 ∈ R K 2 ×K 1 and W2 ∈ R 1×K 2 denote weights for the bottom and upper layers, respectively, and λ1 and λ2 refer to regularization terms. By setting both Φ1 and Φ2 as increasing concave functions, the objective function becomes a concave function and thus it needs to be maximized with respect to W1 and W2.
The update for weights W1 and W2 follows a feed-forward manner that is similar to that employed for a standard Multiple Layer Perceptron (MLP) training.
As to the update for the weights of the bottom layer, the temporal variables δ1(i) and ∇1(i, j) need to be firstly computed by (14) and (15), and W1 is updated by (16), where µ refers to the learning rate.
The procedure of updating W2 starts as soon as the update of W1 is done. The new δ2 and ∇2(i) are separately derived by (17) and (18) based on the message propagation from the bottom layer. Finally, the update for W2 is conducted by (19).
Proposition 3. In the nested structured framework, given the numbers K1 and K2 for the input and hidden layers, respectively, and the number of candidates N , the computational complexity for the entire training process is O (N K1 + K1K2).
The inference for the nested structured framework shares the same steps that the linear structured framework except that the step 2 in Algorithm 1 is replaced by (20).

EXPERIMENTS
This section presents the experimental setups and empirical results on the distributed ASR system. Our distributed ASR system is shown in Figure 1, where there are 8 DNN-based acoustic models and each DNN shares the same initial setup in terms of the model architectures and ReLU activation functions. A full training dataset is partitioned into 8 non-overlapping subsets by means of the robust submodular data partitioning algorithm [3], and each subset is employed to train a particular DNN-based acoustic model, which results in 8 different DNN-based acoustic models. The 8 lists of acoustic scores from the acoustic models need to be aggregated into a final list with a higher ASR accuracy. Instead of adopting the aggregation method based on Adaboost of our previous work [3], we apply the approach of submodular rank aggregation discussed in this work.

Experimental setup
Following the steps in [3], the submodular partitioning functions were composed based on the prior phonetic knowledge that a triphone corresponds to 8 different biphones based on the phonetic knowledge including 'Place of Articulation', 'Production Manner', 'Voicedness' and 'Miscellaneous'. For training each DNN-based acoustic model, the full dataset is split into 8 disjoint data subsets by formulating the problem as a robust submodular data partitioning algorithm in [3]. Our experiments were conducted on the 1300 hours of conversational English telephone speech data from the Switchboard, Switchboard Cellular, and Fisher databases. The development and testing datasets were the 2001 and 2002 NIST Rich Transcription development sets, with 2.2 hours and 6.3 hours of acoustic data, respectively. Data preprocessing includes extracting 39-dimensional Mel Frequency Cepstrum Coefficient (MFCC) features that correspond to 25.6ms of speech signals. Besides, mean and variance speaker normalization were also applied.
The acoustic models are initialized as clustered tri-phones modeled by 3-state left-to-right hidden Markov models (HMMs) [13]. The state emission probability in the HMMs was modeled from the Gaussian mixture model (GMM). The DNN targets consisted of approximately 7800 clustered tri-phone states [14]. All sequential labels corresponding to the training data were generated by forcedalignment based on HMM-GMM. A 3-gram language model, built from the training materials, was used for the decoding.
The units at the input layer of each of the 8 DNNs correspond to a long-context feature vector that was generated by concatenating 11 consecutive frames of the primary MFCC feature followed by a discrete cosine transformation (DCT) [15,16]. Thus, the dimension of the initial long-context feature was 429 and the number was reduced to 361 after DCT. There were 4 hidden layers in total with a setup of 1024-1024-1024-1024 for the DNN construction [17]. The parameters of the hidden layers were initialized via Restricted Boltzman Machine pre-training [18], and the fine-tuned by the Multi-layer Perceptron Back-propagation algorithm. Besides, the feature-based maximum likelihood linear regression was applied to the DNN speaker adaption [19].
When the training process of all the DNN-based acoustic models was done, the final posteriors of the clustered tri-phones associated with the training data should be separately obtained from each of the DNN-based acoustic models. Those posteriors were taken as permutation data for training the submodular rank aggregation models. In the testing stage, the posteriors collected from the 8 DNN-based acoustic models are combined to one permutation that is expected to be as close to the ground truth as possible. As for the configuration of the submodular rank aggregation formulations, the number of permutations K is set to 8, and the dimension of a permutation is configured as 7800, which matches the number of clustered triphone states. Besides, the number of units of the hidden layer in the nested structured framework is set to 120, and there is only one output corresponding to the final aggregation permutation. Table 1 shows the ASR decoding results from each of the DNNbased acoustic models, and Table 2 represents the combined ones based on the different aggregation methods. The results suggest that the submodular rank aggregation methods based on the linear and nested structured formulation obtain a better result than the baseline system based on the Adaboost approach and a simple averaging method. The marginal gain obtained with the nested structured formulation with regards to the linear structured one arises from the potential bias caused by the forced-alignment of the clustered triphones.

CONCLUSIONS
This work proposes a submodular rank aggregation on score-based permutations based on the LB divergence in both linear and nested structured frameworks for a distributed ASR system. Besides, the related learning algorithms based on SGD are used to ensure optimal solutions to the submodular rank aggregation problem. Our experiments on the distributed ASR system show that the nested LB divergence can obtain significantly more gains. However, the gains obtained with respect to the other approaches are lower to varying degrees. In addition, our methods can be scalable to large-scale datasets because of their low computational complexity in the inference procedure. Future work will study how to generalize the nested structure to a deeper one with more hidden layers. Although the convexity/concavity of the objective function with a deep structure can be maintained, the use of the message-passing to deeper layers cannot obtain a satisfying result on the task of distributed ASR. Thus, an improved learning algorithm for training an LB-divergence based objective function with a deeper structure is necessary.


Sensor Fusion for Identification of Freezing of Gait Episodes Using Wi-Fi and Radar Imaging

Parkinson’s disease (PD) is a progressive and neurodegenerative condition causing motor impairments. One of the major motor related impairments that present biggest challenge is freezing of gait (FOG) in Parkinson’s patients. In FOG episode, the patient is unable to initiate, control or sustain a gait that consequently affects the Activities of Daily Livings (ADLs) and increases the occurrence of critical events such as falls. This paper presents continuous monitoring ADLs and classification freezing of gait episodes using Wi-Fi and radar imaging. The idea is to exploit the multi-resolution scalograms generated by channel state information (CSI) imprint and micro-Doppler signatures produced by reflected radar signal. A total of 120 volunteers took part in experimental campaign and were asked to perform different activities including walking fast, walking slow, voluntary stop, sitting down & stand up and freezing of gait. Two neural networks namely Autoencoder and a proposed enhanced Autoencoder were used classify ADLs and FOG episodes using data fusion process by combining the images acquired from both sensing techniques. The Autoencoder provided overall classification accuracy of ~87% for combined datasets. The proposed algorithm provided significantly better results by presenting an overall accuracy of ~98% using data fusion.

disease, the patients' health get badly affected and because of enduring ailment, coerce them to get more frequent sick leaves and eventually it come to stage, where they have no choice but to consider early retirement from work. In this way, they get sufficient time to properly look after health maintain their health in more effective way and prevent any fatal and sudden disorders. keeping their health ample time to preserve usually take early retirement that eventually leads to loss in productivity along extreme healthcare and other societal expenditures.
In medical healthcare, one of the major motor related impairments in that present biggest challenge is freezing of gait (FOG) in Parkinson's patients. In FOG episode, the patient is unable to initiate, control or sustain a gait. The FOG events are usually confined to short period of time followed by regaining control and continue regular walking. Statistic This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ indicate that half of the Parkinson's patients experience these episodes twice a month and third experiences on day to day basis [2]. This results in limited mobility, deterioration in mental well-being and adverse effect on activities of daily livings and the patients are at high risk of falling causing fatal injuries [3]. Critical events such as falls make 20% to 30% of injuries among elderly people [4].
These signs make the occurrence of FOG episodes as serious health concerns for patients. It is thus highly important to deliver means for identifying and reducing the effect of FOG episodes. Rhythmic auditory stimulation (RAS), such as playing marching music and dance therapy, has been shown to be a safe, inexpensive, and effective method in resuming normal gait in PD patients [5].
The non-invasive wireless sensing leveraging wireless channel information (CSI) and micro-Doppler signatures in medical healthcare are two of the most promising solutions to detect the occurrence of FOG episodes in real-time and deliver feedback cue in an attempt to restore the normal gait alert the patient and reduce the risk of critical events such as falls. Providing timely cue to the Parkinson's disease patients has proved to decrease the severity of particular episodes [6] and reduce the overall duration of the episodes by 35% [7]. Several researchers have designed FOG classifiers based on either software devices. The software-based FOG detector [8] presents several constraints such as mapping machine learning algorithms on hardware. It is thus evident that the FOG detection in PD patients that provide hardware implementation is obtrusive and bulky to be worn on body. There are several case studies that have mapped real-time, complex hardware implementation of machine learning classifiers for wearable healthcare applications. The Field Programmable Gate Array (FPGA) based systems were proposed for heartbeat monitoring [9], [10], seizure episodes in epilepsy disease [11]- [13] and activities of daily living [14], [15]. In light of these advances in healthcare sector, this article presents a non-invasive, low-cost and easily deployable FOG detector based on the data fusion of scalogram obtained using Wi-Fi sensing and spectrograms acquired using radar sensor. A novel deep neural network, namely enhanced Autoencoder is proposed that classifies activities of daily living and detects FOG episodes with high accuracy.
The paper is organized as follows: Section II discusses the related work done on freezing of gait detection, section III explains the signal processing for Wi-Fi sensing, generating scalograms from CSI and producing spectrograms from received signals through radar sensor. Section III gives a brief introduction to the neural network used and discusses the proposed enhanced Autoencoder. Section IV provides the experimental setup and how data were acquired, section V provides details about results obtained and section VI concludes the paper.

II. RELATED WORK
Several researchers have addressed the application of software-based systems and wearable devices in healthcare sector to recognize human activities [16], detect inertial movements [17]. An extensive work has been done in research published work in [18], [19] that leverage wireless sensing for medical healthcare application. In addition, several recent research articles have been published on patient monitoring, quantification of Parkinson's disease motor impairments using wearable sensors only. The main issues with these articles are the locations of sensors placed on bodies, the experimental setup and achieved accuracy. A comprehensive review of medical healthcare detection systems have been discussed in [20], [21]. There are several systems that exploit data mining and machine learning algorithms to determine the severity of particular disease. Arora et al. [19] recruited ten clinically diagnosed PD patients and ten healthy volunteers. Android smartphones were deployed on each subject's body and were asked to perform activities such as finger tapping, finger to nose test, and walk back-and-forth. The authors claim that this method could easily discriminate PD patients from healthy volunteers using machine learning algorithms, however due to the limited number of subjects and low classification accuracy, this solution is not suitable for FoG gait detection. Numerous wearable sensors have been for objective assessment of FoG episodes, yet there is small or no agreement with regard to location, number of participants, experiment setup and dataprocessing techniques. Stand-alone tri-axial accelerometer is widely used as in [22], magnetometer or combination of accelerometer and gyroscopes. Wearable sensors non-wearable sensors are used for cardiovascular activity, gait identification and activities of daily living [23]- [27].
The data-processing and classification algorithms are different in all aforementioned systems. Algorithms based on threshold are easily implemented and deliver adequate performance [28], however, these require optimum threshold levels to be tuned on all patients. Support vector machine (SVM), Naïve Bayes (NB), random forest (RF) are commonly machine learning algorithms used for classification tasks. NB and RF are less computional complex as compared to SVM algorithm, but the latter presents higher classification accuracy and robustness against a large amount of data. Capecci et al. [29] used smartphone worn on waist that executes gait test where accelerometer data is used leveraging threshold-based method, obtaining accuracy of 84.4%. Rodríguez et al. [30] used SVM algorithm with data obtained from 21 participants equipped with wearable sensor to determine FoG and provides specificity and sensitivity of 79.0% and 74.7%, respectively. In reference [31], the authors have used a model on convolution neural network using accelerometer data on 21 volunteers providing an accuracy of 92.3%.

III. PRELIMINARIES A. Wireless Wi-Fi Signals & Channel State Information Extraction
The Wi-Fi signals driven by Orthogonal Frequency Division Multiplexing (OFDM) such as IEEE 802.11 a/a/ac can efficiently and effectively overcome the multipath frequency selective fading experienced due in an indoor environment. The OFDM frequency spectrum is split into several orthogonal frequency carriers where the data that is to be transmitted is encoded and mapped using same modulation scheme. The received signal at the receiving side is down-converted into the baseband signal. The sub frequency channels can be converted from time domain into frequency domain using serial-toparallel signal converter and then applying Fast Fourier Transform (FFT) on all frequency channels, as shown in figure 1. The input serial data stream is formatted into the word size required for transmission, and shifted into a parallel format. The data is then transmitted in parallel by assigning each data word to one carrier in the transmission. The reason for converting serial into parallel is due to the fact that all subcarriers are transmitted at once in once burst rather than one after another, as in case for serial transmission. So that each frequency channel carry adequate information. The operating frequency of Wi-Fi router is set to 2.4 GHz.
The commodity devices such as Intel 5300 and Atheros ar5b225, are open-source device drivers that allow recording the channel state information of each frequency channel (carrier) representing the fine-grained physical layer channel measurements, consisting of wireless channel characteristics such as power distortion, multipath fading and shadowing effect.
Let Hi denote the channel state information values of frequency channel i, which is a complex value and is given as: Here |Hi| and Hi denote the amplitude and phase information of ith subcarrier, respectively. The phase of individual subcarrier i, Hi is written as follows: where β indicates the initial phase offset of ith subcarrier of the phase-locked loop and mi is the subcarrier index of ith frequency channel. The environmental noise is represented in terms of Z, and λp, λs, and λc are the phase errors, sample frequency offset and central frequency offset, respectively. The CSI phase information is not adequate due to the random noise in the radio frequency channels because of using off-the-shelf Intel 5300 network interface. Hence, in this article, we have only considered the amplitude inform retrieved using channel state information.

B. Radar Technology & Micro-Doppler Signatures
The radar sensors use radio signals to identify and locate a target. The traditional radar sensor system consists of a trans-mitter and receiver and a typical signal processing functional unit. The sensor when operating, continuously radiates electromagnetic signals and a target within area of interest reflects back the signal that is received by the radar system. Frequency modulated continuous wave (FMCW) radar uses transmission frequency that varies linearly across the waveform resulting in overcoming range-profile issues and is now widely used short-range wireless sensing applications, including activities of daily living, fall detection in healthcare and so on [23]. The FMCW radar operating at 5.8 GHz is extremely robust against interferences from RF signal sources and has the capacity to record human micro-Doppler signatures while maintaining high resolution. The specifications of an FMCW radar sensors used in this work is presented in table I. The main reason for using radar in the region of 5.6 -6 GHz with operating frequency of 5.8 GHz (ISM, unlicensed band) is that the Wi-Fi transmitter was working at 2.4 GHz (ISM band, unlicensed band). In order to avoid co-channel interference, we opted for a Wi-Fi router and radar sensor that should work in an unlicensed band and should operate at different frequencies.
The RF signals transmitted by an FMCW radar sensor can be mathematically denoted as follows: Here, T F is the total duration of a frame as indicated in Figure 1, N F represents the total number of transmitted frames. The transmitted FMCW signal comprising L number of chirps at the i th frame can be written as: Here, f0 is the operating frequency, μ denotes a change of instantaneous frequency of an FCMW chirp signal. The value of μ can be determined bandwidth (B) divided by the duration of a frame (TF).
In equation 6, a indicates the amplitude information in terms of complex values of the reflected signal of the m th target for i th frame, d s is the distance between the corresponding arrays, τ and w (i) l,k (t) represent the range, Doppler information, and direction-of-arrival information, respectively.

IV. THE PROPOSED FOG DETECTION SYSTEM
The proposed freezing of gait detection system architecture is presented in figure 2 that resolves a multi-classification problem to identify the particular episodes and detect ADLs. The proposed system involves a low-cost commercially available 2.4 GHz WiFi router, a 5.8 GHz Ancortek radar sensor. Each body movement induces a unique CSI represented in terms of scalogram extracted from variances of amplitude information and a micro-Doppler signatures generated by received signal reflected by the subject's body. The multiresolution time-frequency scalograms are obtained using continuous wavelet transform and micro-Doppler are acquired using short-time Fourier transform. The Wi-Fi-based and radar-based signatures are used image object to train and test the proposed enhanced classifier to determine the FOG episodes.

V. PRE-PROCESSING
This section discusses the pre-processing for CSI data obtained using Wi-Fi signals and micro-Doppler signatures received using the radar sensor. The channel state information is extracted from the packets retrieved from Internet Control Messages Protocol (ICMP) packets. Consequently, in theory, the total number of CSI packets received are the same as ICMP packets. However, it was observed that received CSI packets were slightly lesser than the number of ICMP transmitted packets. To synchronize and calibrate the frequency of the data collected, we performed the linear transformation on the raw CSI. The IEEE 802.11n Wi-Fi networks use multiple data sub frequency carriers or subcarrier that transmit Wi-Fi signals simultaneously exploiting orthogonal frequency division multiplexing. In principle, each subcarrier carrier independent information. However, each adjacent subcarriers carry similar information at times. For this purpose, we have used principal component analysis to obtain an independent dataset for each human activity. The CSI data packets can be integrated together into multiple independent principle components.

A. Scalogram
Time-frequency scalograms are extracted from CSI amplitude information from each received packet and are used as features to detect FoG episodes. The extracted features are a multiresolution energy density function obtained from 5000 packets using Continuous Wavelet Transform (CWT). The energy density function E(t, f) is obtained by squaring the amplitude information of the CWT function C d (t, f) of a discrete sequence. The time-frequency scalogram can be computed from the CWT C c (t,s) of a continuous-time signal x(t), represented in terms of time t and scale s as in equation 7.
Here ψ(v − t s ) denote the dilation of the wavelet ψ(t). We have scaled the term v − t = τ and scale the value of s as a function of frequency f provided s = g 1 (w) = g 2 (f). The continuous wavelet transform of discrete and continuous signals are represented in equation 8 and 9 as follows: Here x(KT) is a discrete sequence of samples having a period T = 1/F, where F is the sampling frequency. The CWT of a discrete signal can be obtained when x(kT) is replaced with CSI SC (kT), expressed in equation 10.
In equation 10, F = 60 Hz is the sampling frequency of CSI amplitude information and T = 0.02 seconds. The mother wavelet used in this work is the "morse" wavelet. The scalogram E(t,f) can further be described mathematically as follows: The continuous wave transform-based scalograms deliver multiple resolution time-frequency analyses that are primarily ependent on the window size, resulting in various dilation of the mother wavelet. The scalograms provide adequate transient changes in the amplitude information of the CSI packets due to the human body movements and present high resolution as smaller window durations at higher frequency are used. Moreover, the scalograms have the potential to detect smooth features in waveforms due to larger windows durations at lower frequency. Figure 3(a) shows the variations in amplitude information for single-frequency carrier obtained using Wi-Fi sensing. A clear transition in each human activity can be seen, implying each body motion induces a unique imprint in terms of amplitude variation in dB. The scalograms presented in

B. Micro-Doppler Signature
Any subject that is moving within radar range has mechanical rotation/vibration along with its bulk translation, it leads to the generation of frequency modulation on the reflected lectromagnetic signal that produces sidebands around the subject's Doppler frequency shift known as the micro-Doppler effect [24], [25]. Any object in motion at distance in P moves with frequency f v and displacement D v , with function displacement value of, D (t) = D v si n2π f v tcosβcosα p . Let us assume R 0 to be the distance between the radar sensor and the moving target initial position O, the total range between the two will constantly change with respect to time, the target object's micro-movements are expressed as represented as R (t) + D (t). The reflected signal received can be written as: The value of f 0 in equation 7 is the carrier frequency, λ is its wavelength and ρ is the backscattering coefficient. Putting R(t) in equation 7, the received signal can be mathematically described as follows: Here w v = 2π f v , the derivative of the second phase component provides micro-Doppler shift expression which is expressed as: VI. AUTOENCODER AND PROPOSED ENHANCED AUTOENCODER FOR FOG DETECTION One of the major challenges researchers face is the application of deep neural networks when classifying RF signals (Wi-Fi and Radar in our case) is the limited size of the available dataset. RF data acquisition is consumed a lot of time, is expensive and involves a large number of volunteers [26]. It is highly unlikely to record hundreds of thousands of Wi-Fi and radar data and hence novel algorithms have to be introduced so as to avoid underfitting and overfitting problems. In this context, a 14-layer deep convolution neural network architecture has been proposed to classify human gait classification. In this work, we consider a conventional Autoencoder and compare the results with our proposed threelayer enhanced Autoencoder.

A. Auto-Encoder
An Autoencoder neural network that reproduces the input values at the output side with specific limitations. For instance, given The values of W ∼ and b ∼ show the weights and biases at the decoder side, respectively. During the unsupervised pretraining process, the neural network tried to reduce the reconstruction error to its minimum value as: To prevent the Autoencoder neural network, cost function with a sparsity parameter is applied to force the network to learn the correlation among the given input data. With the addition of sparsity parameters, the cost function can thus be written as: where h denotes the number of hidden neurons, β is the sparsity proportion and KL describes Kullback-Leibler divergence and can be expressed as follows: where p j represents the activation function for j th hidden neuron and p denote the value for activation function, h is the number of hidden neurons. After pretraining the network, the decoder is removed from the network and encoder values are placed for training, using a supervised learning method by adding the SoftMax classifier with six neurons after the encoder. An input vector of K is fed as an input to the SoftMax function that primarily normalizes it into a probability distribution consisting of K probabilities, P (y k ||x i ) for k = 1, 2, . . . K, proportional to the exponentials of the input numbers [28], [29]. In probability distribution function, the input value x i corresponds to labels class y k . The probability class p k can be mathematically expressed as: an input matrix x, the network aims to estimate h w (x) ≈ x. The unsupervised algorithm was introduced in order to initialize the weights and biases of an Autoencoder which was extremely efficient effective when a limited number of training data were available. The Autoencoder neural network implements unsupervised pretraining processing by encoding decoding the Input data, respectively. The Autoencoder approximates a nonlinear mapping on the input data matrix x as follows: where σ describes the nonlinear activation function, W and b are the weights and biases at the encoder side, respectively. The features that are encoded in the network are then decoded in order to reconstruct the particular input matrix x using the following function [27].
The optimum values of weights and biases are obtained when the cost function is minimized as: The 'fine-tuning' technique that works on the gradient-based technique, is applied to resolve the values as indicated in

B. Proposed Enhanced Autoencoder
The enhanced Autoencoder integrates the advantages of convolutional filtering in convolutional neural networks having an unsupervised pretraining of an Autoencoder. On the contrary to the topology for an Autoencoder network, instead of the fully connected layers, the encoder of the proposed neural network consist of convolutional layers and the decoder side have deconvolutional layers. The deconvolutional filters are primarily the transposed copies of the convolutional layers as done in this work and are learned from scratch. Moreover, each and every deconvolutional filer is followed up by an unpooling filter [30]. The unpooling function is operated by saving the locations of the largest values when the pooling process occurs that inherently preserve the values while zeroing the remaining ones. The proposed system is ten times faster than conventional Autoencoder and can provide adequate results when limited number of data (observations) are available for training, specifically in the case of RF sensing system such as ours. The spatial locality of the neural network is preserved by accommodating a convolutional function at each neuron. Hence for a specific input value, P, the encoder calculates the values as follows: Here σ indicates the values for activation function, * is the 2-D convolution, F n is the n th 2-D convolutional layer filter and b represents the basis of the encoder. The unsupervised pretraining process that is applied to the neural network aiming to reduce the subsequent following expression: After performing the unsupervised pretraining process, the decoding part is eliminated and the softmax classifier and fully connected layers are included at the end of the neural network. Next, the neural network is fine-tuned by optimization function as done in the convolution neural network. Similarly, the ADAM algorithm in conjunction with the ReLU activation function for optimization of the two fully connected layers having 150 hidden neurons each. The optimization of the hyperparameters of the proposed enhance Autoencoder was performed by grid search method and is discussed in subsequent section. The overall architecture of the proposed enhanced Autoencoder is presented in figure 6. The deep network architecture was implemented using 14 hidden neurons. Selecting the optimum number of hidden neurons involved rigorous experimentations. Three methods were used namely fixed, constructive and destructive. In the fixed approach, a group of neural networks with different numbers of hidden neurons were trained and evaluated on the test available dataset using a different number of randomly selected starting weights. The increment in the number of hidden neurons one, two or more depending on the computational  resources available. Plotting the evaluation criterion (e.g. sum of squared errors) on the test set as a function of the number of hidden neurons for each neural network generally produces a bowl-shaped error graph. The network with the least error found at the bottom of the bowl was selected because it was able to generalize best. This approach was time consuming, but generally worked very well. The constructive and destructive approaches involve changing the number of hidden neurons during training rather than creating separate networks each with a different number of hidden neurons, as in the fixed approach. The constructive approach involves adding hidden neurons until network performance starts deteriorating. The destructive approach is similar except that hidden neurons are removed during training.

VII. EXPERIMENTAL SETTINGS AND DATA ACQUISITION
The data were recorded from two different sources, i.e. from radar sensors and Wi-Fi router. The experiments were performed at a large room at University of Glasgow as shown in figure 8. The radar sensor antennas were kept at a distance of 0.25 meter as shown in figure 8. All the activities were performed with aspect angel parallel to the radar sensor and Wi-Fi transmitter.   lenge the classifier. Accuracy classification of FOG episodes is vital having minimum false alarms and low missed detection because inaccurate detection of the particular events can have extremely adverse effects on the patients. In this context, all of the 120 volunteers were asked to repeat a single activity more than six times, resulting in 4320 observations in total. The duration of each activity was 5 seconds with the exception of walking activity that was recorded for 10 seconds.

A. Data Acquisition Using Wi-Fi
The experimental design to detect FOG episodes in indoor settings (10 meters by 12 meters room at University of Glasgow, UK) is shown in figure 6. The transmitter, in this case, is a Wi-Fi router that operates at 2.4 GHz and is deployed 8 meters (in line-of-sight) away from the receiving antenna. The receiver is an Omni-directional antenna wired with the Intel 5300 network interface card installed in the PCIe slot of a Dell Inspiron desktop computer, Intel® Core™ i7-9700 Processor, 8GB RAM). The experimental procedure involved the acquisition of multiple data frequency carriers where 30 OFDM subcarriers were obtained from each CSI id. These packets contain the variances of amplitude and phase information of specific human activity which was obtained using signal processing techniques and was performed in Matlab 2019 tool in order to get the time-frequency scalograms. Figure 7 shows the perturbations of amplitude information obtained for five human activities obtained from channel state information using Wi-Fi sensing. Figure 7(a) shows the box plot (amplitude level against group of 30 subcarriers) for person walking fast in indoor settings. Huge variation can be seen, where maximum fluctuation occurs between 12 dB and 25 db. As the person stops walking, the change in amplitude level decreases dramatically and slight variations can be observed as in figure 7(b). There was a sudden increase in variances of amplitude information when person start walking very fast as in figure 7(c). For sitting down on chair, the CSI signatures are distinct from rest of the four imprints as shown in figure 7(d). When the person was experiencing FOG episodes, implying that the feet were felt as if glued to the ground and upper body was trying to move. Hence slight variations can be seen in figure 7(e).
The channel state information measurements for activities of daily living and FOG episodes show distinguishable variances in the amplitude information against individual subcarriers as shown in figure 8. The box plot for CSI data against individuals subcarriers is shown in figure 9, indicating the first I Q R = Q3 C S I sc − Q1 C S I sc (28) min _CSIsc = Q1_CSI sc − 1.5(IQR) The box plot as in figure 8, primarily presents the statistical information regarding the variances of amplitude information of CSI packets across all 30 subcarriers. It is evident that the box plot that statistical features for individual's frequency carrier including quartiles, interquartile ranges and median do not provide adequate information to distinguish activities of daily living and may be similar for specific subcarriers.

B. Data Acquisition Using Radar
The Ancortec radar sensor used to collect data transmitted signals with power approximately +20 dBm at 5. 8 GHz, having a bandwidth of 400 MHz bandwidth. Two Yaggi antennas were used as transmitter and receiver with gain equal to 10 dBi that had horizontal and vertical beamwidth of 60 degrees. The yaggi antenna is primarily manufactured conductors having high reflectivity potential. It has numerous applications due to its unidirectional pattern, high gain and broad bandwidth. It is designed by folding dipole directors where the dipole is electrically connected with the feeder. The dipole is used here is a resonant with the dipole length, that is half the wavelength (half of the wavelength of 5.8 GHz frequency range) and is fed to the feeder. The radiator is set to 5% less than the folded dipole material and is deployed at the forefront to a distance of lower than quarter wavelength of operating waveform. The RF signals directors are designed lesser outwards in order to give yaggi antenna deployed at transmitter and receiver side taper in the radiation and reception direction that essentially makes the operating antenna extremely directional for detecting activities of daily living. The radiation pattern of yaggi antenna used in proposed system is presented in figure 10.
The radar sensor was powered by Universal Serial Bus (USB) cable since its power consumption limited to USB standards. In addition, by analyzing the life span and autonomy of the proposed system in real-world realistic deployment, it would always be connected to a computer for data acquisition and operational task, or to the electricity mains in indoor settings.
The FMCW radar sensor presents both a range profile and Doppler information. In this work, we have only considered the latter information due to computational purpose and ADL and FOD can be accurately identified merely using this information. For data acquisition, the radar was put on an in the same room where Wi-Fi system was deployed. The participants were asked to perform activities within a short range of 1-meter to about 3-meter range and the radar data were obtained with Wi-Fi data simultaneously. The two antennas were deployed in such a way that it would keep the torso of the volunteers in the center of radar beam enhance strength of reflected strength. The recorded data were processed using Short Time Fourier Transform (STFT) to get the spectrograms and produce micro-Doppler signatures.
The Range-Time-Intensity graph can be obtained by stacking up the reflected signals in a matrix format and applying Fast Fourier Transform (FFT) function along the fasttime direction to produce range cells of human action. Next, STFT algorithm is applied to the range profile that consists the target's, in our case people in action, to generate their micro-Doppler signatures [31]. The STFT algorithms apply a sequence of FFTs algorithms with narrow, overlapping windows along the total duration of the collected data; the absolute squared value of the complex matrix is the known as the spectrogram, that is a plot of velocities of body parts in action (obtained through the Doppler effect) with respect to time. A notch MTI filter is applied to mitigate the contribution of static objects targets near 0 Hz such as walls, ceiling, floor, and furniture. Figure 11 shows the micro-Doppler signatures activities of daily living and FOG episodes obtained at the room in University of Glasgow. The positive values of micro-Doppler components are movements towards the radar, while the negative values are the movements away.

VIII. RESULTS AND DISCUSSIONS
In this article, the neural network models are implemented in Python using Keras that exploits Tensorflow using tensor manipulation library. Both Autoencoder and proposed enhanced Autoencoder classification algorithms are tested using scalograms of Wi-Fi data and spectrograms of radar sensor leveraging tenfold cross-validation of both datasets as described in section earlier section. Both neural networks are trained for 300 epochs having a minibatch size of 80. The validation accuracy for both datasets is obtained by splitting 25% of the training data as the validation set and the evaluated the models after the completion of each iteration using the validation datasets.
The adaptive moment estimation algorithm is used to optimize the pretraining process and for fine-tuning having a learning rate of 0.001. Leveraging grid search technique, we have identified the optimum values for width and depth without overfitting as described in Table II (Wi-Fi sensing), Table III (radar sensor) and Table IV (data fusion; combing scalograms and spectrograms). We have opted to use a three-layer Autoencoder having layers of 100, 50 and 25, respectively. The best classification accuracy with optimized parameters are shown in red.
For all three scenarios (Wi-Fi sensing, radar sensing and data fusion: combined datasets), the best classification performance was obtained for 100,50 and 25 layers. The Wi-Fi sensing based on scalograms classification provided an accuracy of 84.5%, the spectrogram classification for radar sensing delivered an accuracy of 85.5.%. When the two datasets were combined, the accuracy of 87.1% was achieved where a small improvement was observed.
We further examine the optimization of hyperparameters of proposed enhanced Autoencoder was performed using grid search, as presented in Table V to Table VII. The enhanced Autoencoder was implemented with three convolutional layers. Both convolutional and deconvolution layers were populated with 30 numbers of 9 by 9 and 3 by 3 concatenated filters. The best classification accuracy using scalograms with Wi-Fi Sensing, 92.3% was obtained with width of 100, depth of 2, having filter size of 9 by 9 and 3 by 3. In addition, the classification accuracy of radar sensing, 94.7%, was slightly better than Wi-Fi sensing as shown in Table VI. When the same network configurations were used, and the combined datasets of both devices were used, a significant change in the performance was observed. The accuracy of the system increased to 98.1% when classification all human activities. Table VII shows the highest classification accuracy (in red) obtained for width of 100 and depth of 2.
The confusion matrix of data fusion for ADLs recognition (walking slow, voluntary stop, walking fast, sitting down and standing up, and FOG episodes) and identifying FOG episodes for Autoencoder and proposed enhanced Autoencoder are shown in Table VI and Table VII, respectively. In Table VI, the Autoencoder provided adequate classification accuracy. However, similar activities such as walking fast & walking slow, voluntary stop and FOG episodes were deliberately chosen to confuse the classifier, consequently challenging its performance. For instance, the walking fast was classified with accuracy of around 90%, however it slightly misclassified with walking slow; indicating an error of ∼10%. A similar pattern was observed for another pair of activities namely voluntary stop and FOG episodes. The particular freezing of gait event was identified with an accuracy of 85% but the algorithm detected false negative rate of 14% for voluntary stop.
A significant improvement in classification accuracy of individual activities can be seen in Table VII. The proposed algorithm detected the walking fast, sitting down and standing up activities with 100% accuracy. The two similar pair of activities, that were presented as to confuse the proposed algorithm, were also classified with accuracies of ∼99% IX. CONCLUSION This paper presented freezing of gait detection Parkinson's disease patients that primarily causes motor impairments leading to high risks of falls. In this context, activities of daily living namely walking fast, walking slowly, voluntary stop, sitting down on chair/standing up from a chair and freezing of gait episodes were monitored using two non-invasive sensing technique. The two methods for data acquisition were Wi-Fi-based sensing and radar-based sensing techniques. The multi-resolution scalograms were produced from CSI data and spectrograms were obtained from received signal using radar sensor. An extensive experimental campaign was conducted involving more than 100 volunteers with age range from 30 to 76 years. For classification purpose, two deep neural networks such as an Autoencoder and proposed convolutional neural network based enhanced Autoencoder were used to classify activities of daily and detect FOG events. It was observed that the Autoencoder delivered overall classification accuracy of ∼87% for combined datasets, however the proposed algorithm outperformed the same classifier providing classification accuracy of 98.1% for the same datasets.