extfieldsofstudy:
Computer Science
Medicine

s2fieldsofstudy:
Computer Science
Medicine

text:
E-Cardiac Care: A Comprehensive Systematic Literature Review

The Internet of Things (IoT) is a complete ecosystem encompassing various communication technologies, sensors, hardware, and software. IoT cutting-edge technologies and Artificial Intelligence (AI) have enhanced the traditional healthcare system considerably. The conventional healthcare system faces many challenges, including avoidable long wait times, high costs, a conventional method of payment, unnecessary long travel to medical centers, and mandatory periodic doctor visits. A Smart healthcare system, Internet of Things (IoT), and AI are arguably the best-suited tailor-made solutions for all the flaws related to traditional healthcare systems. The primary goal of this study is to determine the impact of IoT, AI, various communication technologies, sensor networks, and disease detection/diagnosis in Cardiac healthcare through a systematic analysis of scholarly articles. Hence, a total of 104 fundamental studies are analyzed for the research questions purposefully defined for this systematic study. The review results show that deep learning emerges as a promising technology along with the combination of IoT in the domain of E-Cardiac care with enhanced accuracy and real-time clinical monitoring. This study also pins down the key benefits and significant challenges for E-Cardiology in the domains of IoT and AI. It further identifies the gaps and future research directions related to E-Cardiology, monitoring various Cardiac parameters, and diagnosis patterns.

Introduction
Following the available information as confirmed by the World Health Organization (WHO) [1], cardiovascular disease claims a large number of causalities across the globe [2] and is responsible for approximately 80% of sudden deaths. Moreover, in more than 15% of the deaths, cardiac arrhythmia is considered the chief reason. Thus, promoting cardiovascular health is vital and requires an overhaul of healthcare systems [3].
The rapidly expanding Internet of Things (IoT) [4] technology has the capability to monitor and control the critical human functions, irrespective of where the individual is located or what they are doing. Medical IoT (MIoT) is a cutting-edge technology that functions by exploiting the advantages of the Internet at a very affordable cost with minimum effort. The MIoT-based cardiac system guarantees monitoring the physical symptoms [5] of cardiac patients, such as temperature, Blood Pressure (BP), Oxygen Saturation (SPO2), Electrocardiogram (ECG), Heart Rate (HR) [6], and linked environmental parameters effectively and without any failure. The MIoT cardiac care framework is a customized paradigm that meets the requisite medical and safety standards of pervasive cardiac healthcare, including serious heart-related issues.
Various cardiac (heart) abnormalities can be detected through an Electrocardiogram (ECG) which is a medical testing platform that keeps track of electrical activity the heart generates as it contracts. An electrocardiograph is a device that records patient's ECG. An ECG is a valuable tool for identifying problems associated with heart rate or heart rhythm.
It offers assistance to the physician in determining whether a patient is having a heart attack or has had one in the past. An ECG is usually the first option for a cardiac test because of its proven dependability. An ECG is helpful to determine if one's pulse is difficult to feel (bradycardia), or it is too fast (tachycardia) to count accurately. An ECG can also show heart rhythm irregularities, i.e., arrhythmia. The main types of arrhythmia are mentioned in Table 1.

Types of Arrhythmia Explanation
Tachyarrhythmias A fast heart rhythm with a rate of more than 100 beats per minute.
Bradyarrhythmias Slow heart rhythms that may be caused by disease in the heart's conduction system.

Supraventricular arrhythmias
Arrhythmias that begin in the atria (the heart's upper chambers). "Supra" means above; "ventricular" refers to the lower chambers of the heart, or ventricles.

Ventricular arrhythmias
Arrhythmias that begin in the ventricles (the heart's lower chambers).
Similarly, atrial fibrillation, atrial flutter, and premature or extra beats are the other types of cardiac issues. Figure 1 shows waveforms for different arrhythmia types. Atrial fibrillation refers to a rapid, disorganized, and irregular heart rhythm., while atrial flutter is an atrial arrhythmia generated by a fast circuit in the atrium. Compared to atrial fibrillation, atrial flutter is typically more organized and regular. A comprehensive review of E-Cardiology, which encompasses the Internet of Things, artificial intelligence, and cardiology could help understand the essential building blocks of an IoT-based cardiac care system and intelligent diagnosis of various cardiovascular diseases. It can also help to develop a complete picture of various hardware devices (sensors), AI techniques, and communication technologies adopted by the existing studies in the field of intelligent cardiac healthcare.
Following the introduction, Section 2 of this paper briefly discusses related works; Section 3 highlights the contributions made by this paper. Section 4 elaborates on the review methodology adopted for conducting the survey. Section 5 gives the outcomes of the selected studies with a detailed analysis of the research questions (RQs). This section is further divided into four subsections. Section 6 contains a discussion. Section 7 summarises the conclusions.

Related Works
This section presents a brief explanation of the related surveys in the field of IoT-based cardiac healthcare.
The primary goal of the study [7] was to collect the latest facts, figures, and evidence on the use of preprocessing techniques for heart disease classification. The review study also summarised the impact of the most frequently used preprocessing tasks and techniques and the performance in the field of cardiology. This review paper covered the literature from 2000 to June 2019.
A survey on IoT and AI in healthcare was presented by [8] for 2007 to February 2018. The paper highlighted the top application classifications, which included wearables, sensor networks, connectivity options, and disease detection and treatment. This review identified gaps and provided future research directions related to technology and design. However, this survey analysed only three online databases.
A review article on data mining techniques frequently used in the field of cardiology until 2015 was presented in [9]. The performance comparison of various data mining models in cardiology were also discussed in this review paper.
The authors in [10] presented a survey on the Internet of Things (IoT) for healthcare using mobile computing. This systematic study investigated how mobile computing assisted IoT in a healthcare environment. Moreover, the intention of this paper was to analyse the impact of mobile computing on IoT technology in Smart hospitals and the field of healthcare. This study covered the literature between 2011 and 2019.
Another study [11] proposed a substantial review of various IoT applications in a life-saving environment, as well as various other fields in Smart cities. It also contrasted IoT with M2M and highlighted some drawbacks of IoT technology. This review article covered 2013 to 2018 through the Scopus database.
Another study [12] presented literature on (IoT) technologies and several projects for healthcare in 2018. This paper provided a review of primary medical IoT sensors and an overview of state-of-the-art IoT infrastructure essential for healthcare. It focused on the latest IoT technologies for healthcare services, such as cloud computing, big data, RFID, WSN, Bluetooth, Wi-Fi, and other vital medical sensors. However, this study lacks a systematic review.
The study [13] highlighted various IoT applications and was presented in 2022. The study focused on IoT adoption in Pakistan and France in 2020. This systematic study highlighted the barriers and possibilities for the implementation of IoT applications. It also indicated the influence of COVID-19 on IoT adoption in the healthcare domain.
The [14] systematic review discussed telemedicine and healthcare IoT (HIoT). It covered 146 articles between 2015 and 2020. The articles were divided into five categories after a technical analysis. In addition to the benefits and limitations of the selected methods, a comprehensive comparison of evaluation techniques, tools, and metrics was also included. This study presented a summary of healthcare applications of IoT (HIoT).
The discussion so far is limited to only a particular aspect of Smart healthcare/E-Cardiology and does not genuinely attempt to cover the domain holistically. When we say "entire domain", it means AI-based IoT, which encompasses preprocessing techniques and also various communication technologies. According to the deficiencies of the existing review papers, we provide a comprehensive systematic literature review for the following reasons: • The latest research articles need to be covered to assess the current state of the art. • The present studies do not cover all the aspects of E-Cardiology.
The following section highlights the contributions made by this review study, thus bringing novelty to this systematic review study.

1.
This review paper highlights the influence of IoT, communication technologies, AI models, and preprocessing techniques in cardiac healthcare using our review protocol. Moreover, this study covers the complete and latest infrastructure for E-Cardiology, including its benefits and challenges. Thus, this systematic review covers almost all aspects of E-Cardiology which have not been discussed before in such a comprehensive way under one umbrella. 2.
The study presents the systematic analysis of the most recent studies (2016 to 2021) to investigate our formulated research questions.

3.
This paper incorporates monitoring of vital CCU parameters, ECG analysis, and classification of various heart disorders, thus giving a thorough picture of E-Cardiology.

4.
This review study provides recommendations and future guidelines for researchers and cardiologists as well.
The next section discusses the research methodology adopted for our SLR.

Review Methodology
A systematic literature review (SLR) paradigm is followed in this paper for reviewing papers from the most reliable resources, as shown in Figure 2. Principally, the research work, applications, and monitoring/detection techniques provided by AI-aided MIoT in cardiac care are considered. The primary studies have been then passed through a quality assessment process for the study analysis to produce the best fit results. The following subsections briefly describe the detail of each step involved in our review protocol.

Defining Review Strategy
The application of medical IoT in cardiac care is a compelling field of study for the researchers, so the primary focus of this SLR was to formulate the research questions exploring how medical IoT is affecting cardiac care and the significance of artificial intelligence in the diagnosis and detection of various heart diseases.
The review questions in Table 2 indicate how MIoT and AI are contributing to cardiac healthcare systems in Smart hospitals. This question investigates major benefits, and current challenges of IoT-based cardiac healthcare system.

Defining Search Strategy
Once the research questions were designed, the next step was to indicate and state the search strategy to be followed precisely. Therefore, the primary literature mentioned in Appendix A (Table A1) was identified using three search strings which were used in the five digital databases, namely IEEE Xplore, ACM Digital Library, SpringerLink, ScienceDirect, and Google Scholar. These are the most popular online data resources in the domain of computer science and information technology. Second, these digital libraries were used as sources for previous systematic literature reviews related to computer science and E-Cardiology [15].
Our search span included the period of 2016 to 2021. The criteria used for the selection of search terms or keywords is mentioned below [16]: • The important terms were extracted from the research questions. • Synonyms and alternate spellings were identified for the key terms. • Keywords were identified from various books and relevant research articles. • For synonyms or alternating spellings, the Boolean operator OR was used. • Boolean AND operator was used to interlink significant terms.
After the critical analysis of the key terms, three search strings were formed in order to extract the relevant information. These search strings were checked on each of the aforementioned databases by changing their patterns to retrieve the best relevant results. The three search strings are given in Table 3.

Inclusion and Exclusion Criteria
To identify and include studies relevant to answer the RQs, inclusion and exclusion criteria were developed as described in the section "Defining Review Strategy". To find the most appropriate publications, we defined the inclusion and exclusion criteria as mentioned in Table 4.

Inclusion Criteria Exclusion Criteria
The papers published in English were chosen on priority. The papers published in other languages were not selected.
The most recently published research papers, i.e., 2016 to 2021, were singled out for studies. Gray literature was excluded from the study list.
Papers describing an overview of current approaches that implement modern tools and techniques in E-Cardiology were selected.
Papers not defining the topic appropriately were excluded.
The main aim was to target the primary studies such as original research papers. Duplicated material was removed.
The authors evaluated each forthcoming paper to decide whether it should be included or excluded. The selection of papers was accomplished by following the three steps mentioned below.
The first step included the removal of duplicated and redundant papers; perusing the keywords, abstracts, and titles of research articles was the next step. Reading of full length research papers was carried out in the last step. Accordingly, the inclusion and exclusion criteria were implemented to their full effect. The articles that attracted difference of opinion were discussed and reviewed again by the authors, either using the full text or the partial text, until a consensus was achieved on an agreed-upon draft.

Quality Assessment Criteria
In this step, based upon the coherence and relevancy, we analyzed all the collected studies to address the defined research questions. A deep analysis of each paper was made, and based on our research questions, 134 papers were selected. Out of those 134 research papers, the papers having considerable citation count, appearing in good impact factor journals, and being delivered at highly ranked conferences were finally selected, thus leaving a total of 104 papers for the review, shown in Appendix A (Table A1).

Quantitative Analysis
The last step of our review protocol design was conducted to execute necessary statistical analysis on quantitative data. In this step, we quantitatively summarised and analyzed the results extracted from various sources such as conferences, journals, and book sections. Then, we carried out some quantitative statistical analysis of the findings to explore more about our research questions (RQs) and trends. Figure 3 gives a thorough overview of our screening and assessment method for the statistical analysis of our literature. Five databases were chosen for the review, as illustrated in this figure. A total of 502 documents were chosen for review and analysis. The majority of papers were discovered to be duplicates. Thus, 203 records were eliminated before screening. Papers were removed for a few different reasons. Articles were chosen in the screening process based on a planned inclusion and exclusion strategy. Following the screening, 104 papers were chosen based on inclusion and exclusion standards. The next section highlights the "Outcomes" of this systematic review. The cardiac healthcare monitoring system in an IoT sphere encompasses the various IoT sensory modules and technologies attached to the patient, receiving sensory data, and sending data to the cloud for further monitoring, processing, and decision making. In an IoT-based cardiac healthcare monitoring system, the sensors, such as heart rate/pulse sensor, temperature sensor, blood pressure sensor, blood oxygen sensor, and ECG sensor, obtain sensory parametric values from the patient, transmit data through specific communication technologies to the cloud, apply machine learning practices to the learned parametric values, and generate alerts to the specialist suggesting timely action when warranted.

(i) Heart Rate Sensory Unit
Heart rate monitoring plays a crucial role in patient cardiac abnormalities diagnosis, detection, and classification. Several cardiac ailments and disorders occur due to a patient's high or low heart rate. Normal beats per minute (bpm) are 60-100. Less than 60 bpm is considered to be low and greater than 100 is considered to be high bpm. We discovered comprehensive studies that used several types of heart beat sensors for bpm monitoring. The studies [17,18] used heart beat pulse sensors to measure patient heart rates in a real-time environment. In this article [19], the KY 093 module was used to obtain heart rate values. Using a MAX30100 pulse oximeter, the authors in [20] collected heart beat information. The publications [21][22][23] utlized an ECG module AD8232 to obtain the patients heart rate data in real time for monitoring purposes. Table 5 mentions recent studies on heart rate sensors. Each heart rate sensor has its own set of properties. This table shows some important characteristics of several heart beat sensor variants such as pins, type, operating voltage, low current supply, accuracy, and so on.

(ii) Temperature Sensory Unit
Body temperature is an essential parameter for the development of cardiac healthcare monitoring. Various analog and digital temperature sensors are available for determining body temperature. Temperature can be measured in celsius or fahrenheit. Temperatures above 37.5 or 38.3 celsius are considered high. The temperature sensor LM35 is referenced in [24][25][26][27] for health monitoring. The authors in [18,28] used an 18DS20 sensor for temperature monitoring in a real-time environment. Table 6 shows several of the temperature sensors, along with descriptions. The LM35 and 18DS20 sensors are the most widely employed temperature sensors in the research studies that we analysed.

(iii) Blood Pressure (BP) Sensory Unit
BP monitoring is a fundamental biological measure for the detection and diagnosis of cardiac incongruities and anomalies. BP values can be obtained using various sensory units and devices. Systolic and diastolic values are captured by BP sensors or devices to be examined by a physician. Normal BP is less than 120/80 mmHg, while low BP, called "hypotension", is below 90/60 mmHg, and high BP, called "hypertension", is above 140/90 mmHg. Our research discovered a publication on E-Cardiology that dealt with cardiac patients' BP. In 2017 [29], a digital BP monitor (OMRONHBP1300) was used to monitor and automatically detect cardiac arrhythmia. The paper published in 2019 [30] examined predicting cardiac ailments in E-Cardiology using ECG, cholesterol, and BP. The MPX10 BP sensor was utilised in the 2020 publication [27] for patient health monitoring. Table 7 shows lists of sensors and devices utilized in the past few years for BP monitoring of cardiac patients, along with their comparable attributes.
The multiple modules of BP sensors and devices used in past studies, as well as different sensors of other cardiac parameters, are shown in Table 7.   Acc-accuracy; BPM-beats per minute; GND-ground; INT-integrated IR; LED-infrared light-emitting diode; IRD-IR LED to driver; LO-leads off; N/A-not applicable; N-no; RD-red LED to driver; SCL-serial clock; SDA-serial data; SDN-shutdown control input; V-voltage; VCC-voltage common collector; VIN-voltage input; Y-yes.

(iv) Oxygen Sensory Unit
Blood oxygen can be monitored using several IoT-based blood oxygen sensory units, such as pulse oximetry sensors, to obtain oxygen saturation levels along with a patient's heart rate. Ready-made wearable devices are also available to measure blood oxygen saturation levels. Blood oxygen is measured in percentage. The normal blood oxygen saturation is 90 to 100%. The study [37] describes a pervasive healthcare monitoring service system that uses an SpO2 device to measure oxygen saturation. The MAX30100 pulse oximeter has proven to be useful in measuring blood oxygen levels in cardiac patients [20]. Our findings and the literature on IoT-based cardiovascular healthcare monitoring used the sensors mentioned in Table 8 to measure oxygen saturation. This table lists several important and common features of oxygen saturation sensors and devices, such as addressed parameters, voltage, type, accuracy, pins, range, and so on. Table 8 shows that blood oxygen sensors/devices are used for cardiac patients in very few studies.

(v) ECG Unit
ECG is the most crucial biological parameter for monitoring, detecting, predicting, and classifying cardiac irregularities and variations in the human heart. The ECG AD8232 module was used in the studies [21,22,[38][39][40] to monitor ECG and detect cardiac anomalies in cardiovascular patients. Heart abnormalities were detected with the use of the ECG AD8233 module [41]. In Table 9, recent papers published on multiple ECG sensors and devices are mentioned along with their necessary and comparable features. Low supply current, electrodes, the sampling rate, right leg drive shut down, single supply operation, high pass filter, output, operating temperature, pins, and other features of various ECG modules are considered as the most notable attributes. Table 10 shows detailed and comprehensive literature analysed to find IoT-based cardiovascular sensors and devices used in previous studies from 2016 to 2021. This table demonstrates that the majority of research on ECG has been conducted using the ECG AD8232 module to detect anomalies in cardiac patients.

[81]
Wearabale Smart ECG device (UB-DNNS) Communication technologies and protocols can be defined as a set of rules, technologies, semantics, equipment, and programs used to transfer, process, communicate, and receive information. Communication technologies and protocols vary depending upon the technology and network type devised, developed, or utilized. Some of the protocols and communication technologies are discussed in this section. The publications mentioned in Table 11 address the communication technologies and protocols used in previous selected studies for the development of E-Cardiology, monitoring, detection, and classification. BL is a wireless technology for short-range communication and exchanging data between mobile and fixed devices. BL has a transmission power of 1 mw-100 mw and a 1 Mbps data rate. Its data transmission range is 30 feet. The wearable healthcare monitoring devices (wearable fitness watches and pulse oximeters) may have the BL features integrated. BL technology was also employed in previous research [19,39,41,58,66,73] for data transmission for E-Cardiology. In prior literature on E-Cardiology monitoring, BL technology was determined to be the most commonly used technology. Ethernet is a wired communication networking protocol that can be used in local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). Ethernet allows communication through data cables. The publications [58,83] used an Ethernet-wired technology for connectivity support between various hardware modules implemented for cardiovascular disease diagnosis. One existing research study found that Ethernet-wired communication is rarely used in E-Cardiology. GSM is a cell-based or mobile communication modem that works as a mobile communication system. GSM technology is also used in E-Cardiology to send SMS messages or dial calls. GPS, which helps people to find their position on Earth, consists of networks of satellites and receivers or devices that determine location.
The communication technologies and protocols used in E-Cardiology in previous research studies and findings are detailed in Table 11.
BT-Bluetooth; GSM-global system for mobile communications; GPS-global positioning system; GPRSgeneral packet radio service; MQTT-message queuing telemetry transport; N-No; Sc.P-security protocols; SMS-short message service; SMPP-short message peer-to-peer; SP/PC-Smart phone/personal computer; TCP/IP-transmission control protocol/Internet protocol; WIFI-wireless fidelity; Y-yes.

RQ 3: Which Pre-Processing Techniques Are Used in E-Cardiology, along with the Most Widely Used AI Classifiers/Models?
RQ 3 is divided into two subsections. The first subsection investigates and compares various AI Models for the classification and prediction of CVD. This part explores various studies that use different machine learning and deep learning models for CVD prediction. Our study also provides a comprehensive explanation about the algorithms and methodologies used for prediction and classification techniques and the different datasets and performance metrics that we used to evaluate the models. Furthermore, the data preprocessing techniques used with different classifiers are also indicated in the second subsection below.

AI Classifiers/Models and E-Cardiology
The prediction of CVD is a much discussed topic of research in the realm of healthcare. AI-based prediction systems can be of great help in detecting disease at an earlier stage which can reduce risk associated with disease progression. The concept of AI is not new in cardiac electrophysiology with automated ECG interpretation. It has existed in some form or other since the 1970s [87].
Artificial Intelligence (AI) is the reflection of human cognitive functions from the surroundings acquired by applying algorithms, pattern matching, cognitive computing, and deep learning to achieve specific objectives [88]. The ongoing progress in AI, primarily in the sub-domains of machine learning (ML) and deep learning (DL), have caught the attention of physicians hoping to develop newly integrated, dependable, and potent methods for ensuring standard healthcare in the critical field of cardiology.
Machine learning (ML) is a subset of AI to "teach" computers to analyze huge datasets in a quick, accurate, and efficient manner by using complex computing and statistical algorithms [89]. Supervised ML is more successful in predicting survival compared to the traditional clinical risk scores [90].
The study [91] proved that the accuracy of disease prediction can be increased by using an unsupervised type of ML for obstructive coronary artery disease in nuclear cardiology.
Deep learning (DL) is a supervised ML methodology that relies on neural networks and is known for the automated algorithms required to extract meaningful patterns from data collections [92]. In the medical context, the most widespread deep learning algorithms are artificial neural networks (ANN), multilayer perceptron (MLP), convolution neural networks (CNN/ConvNet), recurrent neural networks (RNN), radial basis function network (RBFN), deep belief networks, and deep neural networks (DNN) [88]. Compared with traditional supervised ML, the real strength of DL is that it is an effective, powerful, and flexible approach to representing complicated raw input data that does not demand manual feature engineering. For instance, while addressing the issue of automated ECG interpretation, early conventional supervised ML techniques depended on human-defined ECG features. In contrast, the modern DL model extracts patterns within raw ECGs to detect sinus rhythm and various other arrhythmias with a performance that equals the result of any cardiologist [93].
The significant areas of cardiac healthcare that can benefit from ML/DL techniques are prognosis, diagnosis, classification, treatment, and clinical workflow. Table 12 presents an overview of different AI algorithms extracted from the literature review on heart disease diagnosis/classification. A comparative analysis of different AI techniques frequently used in Smart cardiology for the prognosis/diagnosis of various CVDs is given in Table 13.
As suggested by WHO, by 2030 almost 23.6 million individuals will die from heartrelated causes [94]. CVDs are the main cause, but they can be cured and prevented. To reduce the risk involved, analysis is fundamental. The difficult part is accurate diagnosis [95]. Table 14 summarises the most recent work performed in the field of artificial intelligence related to CVDs.

Data Preprocessing Techniques in E-Cardiology
This section identifies and evaluates studies that applied data preprocessing techniques in cardiac disease classification. Data Preprocessing (DP) in AI is a critical stage that enhances the quality of data to achieve meaningful insights and is the initial step in the development of an AI model. Conventionally, real-world data is not in an appropriate format and contains errors or outliers. It usually lacks specific attribute values/trends, thus resulting in an inadequate AI model. Data preprocessing solves this problem by cleaning and organizing raw data to tailor it to the needs of building and training AI models. Hence, data preprocessing in AI is a data mining approach that reshapes raw data into a readable format that is readily available for an AI model to meet the high standards of performance [120]. Consequently, the algorithm can easily interpret the data's features.There are four primary ways of data preprocessing, i.e., (1) data cleaning, (2) data integration and formatting, (3) data transformation, and (4) data reduction.
Different preporocessing techniques used in past studies for diagnosing heart disease and other types of arrhythmia are also mentioned in Table 14, along with AI models. This table also lists the task performed by the preprocessing technique.

RQ 4: What Are the Major Issues and Challenges in Current E-Cardiology?
After conducting comprehensive research, we identified some significant benefits and major challenges in the field of MIoT to answer our RQ 4. These challenges and benefits have been emphasized on the basis of studies conducted by different researchers in the domain of MIoT and E-Cardiology. Based upon selection and rejection criteria, only valid and reliable papers were selected, as mentioned earlier. We incorporated only the latest benefits and challenges that were found to be unique in the domain of IoT and AI regarding healthcare and cardiology. A pictorial representation of these benefits and challenges is shown in Figure 4.

Benefits of E-Cardiology
Internet of Things (IoT) develops a linkage between "things", such as devices, gadgets, vehicles, and sensors. Likewise, the medical Internet of Things (MIoT)-based cardiovascular healthcare system monitors the physical symptoms [5] of cardiac patients at a very reasonable cost. These physical symptoms include temperature, blood pressure (BP), SPO2, and heartbeat, along with ECG [6] and associated numerical measurements. Significant benefits of IoT-based cardiology from various studies are noted in Table 15.   Table 16. Artificial intelligence (AI) is another significant aspect of E-Cardiology. Until now, AI in cardiac electrophysiology has exhibited promising results. Primary advantages of AI-based cardiology are discussed in Table 17.

Challenges of E-Cardiology
Internet of Things (IoT) comes with various challenges as shown in Table 15. In addition to these challenges, ref. [129] proposed some other vital challenges, such as fixation of sensors, body impact on signal propagation, and synchronization, that may affect critical health services such as cardiac care. The MIoT-based healthcare systems are expected to produce a vast amount of data. Moreover, these sensors and devices are linked through networks, thus enabling real-time transmission of data. Therefore, hackers may attempt to target it. Moreover, the timely availability of medical data will affect the patient's life. Consequently, it is crucial to have real-time information with lower latency over the network [124].
Artificial intelligence (AI) has brought a revolution in the field of healthcare. It has especially made a great contribution in the domain of cardiac care, such as timely prediction and diagnosis of cardiovascular diseases (CVDs), ECG analysis, and arrhythmia classification. However, despite all these milestones, it carries some challenges. Table 17 describes critical issues and challenges of AI-based cardiology from different past studies.
It is hoped that these challenges can be met and that through MIoT and AI we can achieve new levels of technical and medical standards in the field of cardiac healthcare. Figure 5 shows the distribution of the selected studies (104 papers) according to our four research questions (RQs). The pie chart shows that 47 studies overlap RQ1 and RQ2, whereas 33 studies address RQ3 and 24 address RQ4. The outcomes of this systematic literature review suggest a noticeable increase in the research conducted in IoT using artificial intelligence in E-Cardiology. The research activities also incorporate monitoring CCU parameters, ECG analysis, and diagnosis/classification of various heart diseases. This extensive review study reveals that IoT sensors utilized for E-Cardiology are based upon analog sensors, digital sensors, and different wearable sensors or device modules. For fitness tracking and health monitoring activities, a variety of ready-made and wearable watches are also available. E-Cardiology patients can use a variety of wearable gadgets to track various cardiac healthcare characteristics. To establish expert E-Cardiology systems, several communication technologies and protocols for transmissions over a defined range need to be instituted and maintained. Our findings on communication technologies and protocols for the implementation of IoT-based cardiovascular healthcare monitoring and detection systems show that the following technologies provide viable tools to use in MIoT systems: Bluetooth (BL/BLE), ethernet, global system for mobile (GSM), global positioning system (GPS), global packet radio service (GPRS), message queuing telemetry transport (MQTT), short message service (SMS), email, zigbee, transmission control protocol/Internet protocol (TCP/IP), wireless fidelity (Wifi), security protocols, broadband/Internet, cloud, Smart cars, Smart phones/computers, etc. Our findings show that deep learning is often used in cardiac imaging procedures, particularly in echocardiography [88]. Furthermore, CNNs have been evaluated and found useful in calculating coronary artery calcium in cardiac CT angiography [101]. Though deep learning techniques are garnering attention in the field of Smart cardiology, we may infer that instead of depending on a particular AI model, hybrid techniques are expected to produce better results. From the results of our study, it may also be inferred that SVM was more frequently used in cardiac care than was deep learning. However, in recent years, deep learning has emerged as a more powerful and reliable tool for the detection and diagnosis of various heart diseases. It was also noted that data reduction appears to be a major concern of researchers when applying data mining/data preprocessing approaches to predict CVD. This literature review includes 24 studies devoted to major issues and challenges in E-Cardiology. These studies conclude that a major benefit of MIoT in cardiac healthcare is that it generates timely and accurate data, which results in better healthcare outcomes. One vital advantage of using ML techniques is the ability to fuse various types of data [143]. We also assessed the results and noted that mIoT devices do not possess the requisite data protocols and standards [131]. Therefore, many issues must be addressed to ensure IoT privacy and security [144,145], which is one of the major challenges of the IoT era. Moreover, the continuous monitoring of critical indicators requires reduced energy consumption and a longer battery life [146] to prevent a break of communication. This is also one of the significant challenges of MIoT. The medical data gathered using MIoT is seldom standardized and often fragmented. The data in legacy IT systems are usually generated with incompatible formats. Thus, the great challenge of interoperability needs to be addressed as well [124,147]. However, to move forward in the field of MIoT, fearing AI is no option.

Discussion
Instead, we should work toward the smooth digitization of healthcare infrastructure [148]. Obviously, various benefits of AI cannot be implemented and utilized correctly without integrating AI into clinical decision making effectively and responsibly [149]. Figure 6 refers to the research work conducted based on the number of papers per annum. It shows the year-wise trends in publications in the field of E-Cardiology. It also indicates that the maximum number of papers selected for this survey was from 2019.

Gaps, Future Recommendations
In a future work, all the vital cardiac parameters can be combined with an intelligent cardiac care unit (CCU) to develop a complete picture of E-Cardiology. These parameters/indicators may be comprised of temperature, blood pressure, oxygen saturation, heart rate, and ECG analysis. In addition, in differentiating between normal and abnormal heartbeats, a Smart CCU can also be used to detect QRS complexes in electrocardiographic (ECG) signals to determine the presence of a cardiac malady and different arrhythmias. Integrated and wearable IoT solutions, which address all the necessary cardiac parameters of a heart patient, need to be implemented. The results and accuracy of the devices/sensors used in the development of an IOT-based cardiac system cannot be compromised. The implemented system must be tested, evaluated, and approved under the supervision of cardiologists. Advanced communication technologies, including secure network protocols, must be implemented. Data accessibility features, such as widespread data access, must be possible in a secure environment so that the data confidentiality and integrity are maintained. Ubiquitous access is also an important factor and can be achieved by storing the digital data on a cloud server.

Limitations of the Review Study
This review of literature has some limitations. First, many papers were conference proceedings; therefore some parameters remained inaccessible since their authors did not mention them in detail. Second, some of the studies on AI-based IoT architecture in cardiac healthcare could not be located even after following a comprehensive search protocol, such as gray literature and reports that were not published in the databases which we selected for review. Therefore, we suggest that an additional systematic study be conducted to cover the related literature from other important databases.

Conclusions
This review study outlines a total of 104 primary studies from 2016 to 2021 based on our filtering process for supporting the proposed research. Quality assessment of the selected studies was conducted for the formulated research questions after a rigorous analysis.
This work mentions different sensors and communication technologies being used in cardiology. Moreover, this review study also describes various preprocessing techniques and AI algorithms used in the existing studies to diagnose and classify CVD and ECG analysis. This systematic review also provides comparative analysis of various existing techniques in the field of AI, medical sensors, and communication technologies. Finally, this study targets various advantages and issues indicated in the existing literature in the field of E-Cardiology. The interaction of MIoT and artificial intelligence makes cardiac healthcare more manageable by making various applications, services, communication protocols, third party APIs, and IoT sensors available. E-Cardiology guarantees more privacy and security to the IoT devices which are prone to hackers. Furthermore, AI-based diagnosis of various cardiovascular diseases in E-Cardiology helps save time, enabling cardiologists to focus more on treatment. This systematic work presents a review protocol to analyse how IoT applications assist cardiac healthcare and how various artificial intelligence (AI) models contribute to present and prospective research work of IoT in E-Cardiology. This study also indicates how different communication technologies bring privacy and security to IoT devices related to cardiac healthcare. The purpose of this paper is to highlight the influence of IoT, communication technologies, and AI techniques in cardiac healthcare in light of our systematic literature review protocol. Therefore, one can say that this systematic review covers the complete and latest infrastructure of E-Cardiology, along with its benefits and challenges which have not been examined before in such a comprehensive way.

Conflicts of Interest:
The authors declare no conflict of interest.

Selected Literature
Summary of the selected literature consisting of 104 papers used in this systematic literature review is given in the Table A1.


Estimation of Reliable Proposal Quality for Temporal Action Detection

Temporal action detection (TAD) aims to locate and recognize the actions in an untrimmed video. Anchor-free methods have made remarkable progress which mainly formulate TAD into two tasks: classification and localization using two separate branches. This paper reveals the temporal misalignment between the two tasks hindering further progress. To address this, we propose a new method that gives insights into moment and region perspectives simultaneously to align the two tasks by acquiring reliable proposal quality. For the moment perspective, Boundary Evaluate Module (BEM) is designed which focuses on local appearance and motion evolvement to estimate boundary quality and adopts a multi-scale manner to deal with varied action durations. For the region perspective, we introduce Region Evaluate Module (REM) which uses a new and efficient sampling method for proposal feature representation containing more contextual information compared with point feature to refine category score and proposal boundary. The proposed Boundary Evaluate Module and Region Evaluate Module (BREM) are generic, and they can be easily integrated with other anchor-free TAD methods to achieve superior performance. In our experiments, BREM is combined with two different frameworks and improves the performance on THUMOS14 by 3.6% and 1.0% respectively, reaching a new state-of-the-art (63.6% average mAP). Meanwhile, a competitive result of 36.2% average mAP is achieved on ActivityNet-1.3 with the consistent improvement of BREM. The codes are released at https://github.com/Junshan233/BREM.

INTRODUCTION
With the conversion of mainstream media information from text and images into videos, the number of videos on the Internet grows rapidly in recent years. Therefore video analysis evolves into a more important task and attracts much attention from both academy and industry. As a vital area in video analysis, temporal action detection (TAD) aims to localize and recognize action instances in untrimmed long videos. TAD plays an important role in a large number of practical applications, such as video caption [11,30] and contentbased video retrieval [3,8].
Recently, a number of methods have been proposed to push forward the state-of-the-art of TAD, which can be mainly divided into three types: anchor-based [16,18,31], bottom-up [14,15,37], and anchor-free [12,26,35] methods. Although anchor-free methods show stronger competitiveness than others with simple architectures and superior results, they still suffer from the temporal misalignment between the classification and localization tasks. Current anchor-free frameworks mainly formulate TAD into two tasks: localization and classification. The localization task is designed to generate action proposals, and the classification task is expected to predict action category probabilities which is naturally used as ranking scores in non-maximum suppression (NMS). However, classification and localization tasks usually adopt different training targets. The feature that activates the classification confidence may lack information beneficial to localization, which inevitably leads to misalignment between classification and localization. To illustrate this phenomenon, we present a case on THUMOS14 [10] in Fig. 1, where a proposal with the highest classification score fails to locate the ground truth action. This suggests that the classification score can't accurately represent localization quality. Under this circumstance, accurate proposals may have lower confidence scores and be suppressed by less accurate ones when NMS is conducted. To further demonstrate the importance of accurate score, we replace predicted classification score of action proposals with the actual proposal quality score, which is tIoU between proposal and corresponding ground-truth. As shown in Tab. 1, mAP is greatly improved, which suggests that accurate proposals may not be retrieved due to inaccurate scores.
Recent attempts adopt an additional branch to predict tIoU between proposal and the corresponding ground truth [12] or focus on the center of an action instance [35]. Although notable improvement is obtained, there is still a huge gap between the performance of previous methods and ideal performance. We notice that previous methods mainly rely on the region view which only considers global features of proposals and ignore local appearance and motion evolvement, which increases the difficulty of recognizing boundary location accurateness, especially for actions with long duration.
In this paper, we propose a new framework that gives insights into moment and region views simultaneously to align two tasks by estimating reliable proposal quality. First, we propose Boundary Evaluate Module (BEM) to acquire boundary qualities of proposals from a moment view. Specifically, BEM focuses on local appearance and motion evolvement for predicting the boundary quality of each temporal location which reflects the distance between the current location to the location of the action boundary. Then, the quality of the generated proposal is calculated by its boundary qualities. However, the duration of realistic actions can vary from a few seconds to minutes and the localization quality of short actions is more sensitive to the boundary error than long actions. To address this, multi-scale boundary quality is adopted in BEM in a divide-and-conquer way which assigns a suitable scale for each proposal depending on its duration. For the region view, we propose a simple but effective module named Region Evaluate Module (REM), which employed the sampled features in proposals as the proposal feature representation and refines proposals. In particular, REM obtains aligned features by sampling at three locations within the action proposals, which contain more contextual information beneficial to estimate reliable proposal quality and accurate boundary. The proposed Boundary Evaluate Module and Region Evaluate Module (BREM) are generic, and they can be integrated with other anchor-free TAD methods to achieve better results. To validate the effectiveness of BREM, we conduct experiments on two popular mainstream datasets THUMOS14 [10] and ActivityNet-1.3 [9]. By combining BREM with a basic anchor-free TAD framework proposed by [33], we achieve an absolute improvement of 3.6% AP@Avg on THUMOS14. When integrating with the stateof-the-art TAD framework ActionFormer [35], we achieve a new state-of-the-art (63.6% AP@Avg) on THUMOS14 and competitive result (36.2% AP@Avg) on ActivityNet-1.3.
Overall, the contributions of our paper are following: 1) Boundary Evaluate Module (BEM) is proposed to predict multi-scale boundary quality and offer proposal quality from a moment perspective. 2) By introducing Region Evaluate Module (REM), the aligned feature of each proposal are extracted to estimate localization quality in a region view and further refine the locations of action proposals.
3) The combination of BEM and REM (BREM) makes full use of moment view and region view for estimating reliable proposal quality and it can be easily integrated with other TAD methods with consistent improvement, where a new state-of-the-art result on THUMOS14 and a competitive result on ActivityNet-1.3 are achieved.

RELATED WORK
Anchor-Based Method. Anchor-based methods rely on predefined multiple anchors with different durations and the predictions refined from anchors are used as the final results. Inheriting spirits of Faster R-CNN [22], R-C3D [31] first extracts features at each temporal location, then generates proposals and applies proposal-wise pooling, after that it predicts category scores and relative offsets for each anchor. In order to accommodate varied action durations and enrich temporal context, TAL-Net [6] adopts dilation convolution and scale-expanded RoI pooling. GTAN [18] learns a set of Gaussian kernels to model the temporal structure and a weighted pooling is used to extract features. PBRNet [16] progressively refines anchor boundary by three cascaded detection modules: coarse pyramidal detection, refined pyramidal detection, and fine-grained detection. These methods require predefined anchors which are inflexible because of the extreme variation of action duration.
Bottom-up Method. Bottom-up methods predict boundary probability for each temporal location, then combines peak start and end to generate proposals. Such as BSN [15], it predicts start, end, and actionness probabilities and generates proposals, then boundarysensitive features are constructed to evaluate the confidence of whether a proposal contains an action within its region. BMN [14] employs an end-to-end framework to generate candidates and confidence scores simultaneously. BU-TAL [37] explores the potential temporal constraints between start, end, and actionness probabilities. Some methods, such as [19,34,38] adopt generated proposals by BSN or BMN as inputs and further refine the boundary and predict more accurate category scores. Our method is inspired by bottom-up frameworks, but we utilize boundary probability to estimate proposal quality instead of generating proposals.
Anchor-Free Method. Benefiting from the successful application of the anchor-free object detection [21,27], anchor-free TAD methods have an increasing interest recently which directly localize action instances without predefined anchors. A2Net [33] explores the combination of anchor-based and anchor-free methods. AFSD [12] is the first purely anchor-free method that extracts salient boundary features using a boundary pooling operator to refine action proposals and a contrastive learning strategy is designed to learn better boundary features. Recent efforts aim to use Transformer for TAD. For example, RTD-Net [26] and TadTR [17] formulate the problem as a set prediction similar to DETR [4]. ActionFormer [35] adopts a minimalist design and replaces convolution networks in the basic anchor-free framework with Transformer networks. Our method belongs to anchor-free methods and is easily combined with anchor-free frameworks to boost the performance.

METHOD
Problem Formulation. An untrimmed video can be depicted as a frame sequence = { } =1 with frames. Action annotations in video consists of action instances Ψ = { , } =1 , where = ( , ) are timestamp of start and end of the -th action instance respectively and is the class label. The goal of temporal action detection is to locate boundaries of actions and recognize categories which cover Ψ as precisely as possible.
Overview. For an untrimmed video denoted as = { } =1 , a convolution backbone (e.g., I3D [5], C3D [28].) is used to extract 1D temporal feature ∈ R / × , where , , denote video frame, feature channel and stride. Then, up-sample is used to for acquiring frame level feature . Multi-scale boundary quality of start and endˆ/ are predicted by (Sec. 3.2). Parallel, several temporal convolutions are used on to generate the hierarchical feature pyramid. For each hierarchical feature, a shared detection head is applied to predict coarse proposals and category confidences. After that, the aligned feature is extracted for each coarse proposal to refine boundaries and scores (Sec. 3.3). The boundary quality of each proposal is interpolated onˆ/ according to the temporal location of boundaries.

Basic Anchor-free Detector
Following recent object detection methods [27] and TAD methods [12,33], we build a basic anchor-free detector as our baseline, which contains a backbone, a feature pyramid network, and heads for classification and localization.
We adopt I3D network [5] as the backbone since it achieves high performance in action recognition and is widely used in previous action detection methods [12,37]. The feature output of backbone is denoted as ∈ R / × . Then, is used to build hierarchical feature pyramid by applying several temporal convolutions. The hierarchical pyramid features are denoted as { ∈ R / × } =1 , where means -th layer of feature pyramid and is the stride for the -th layer.
The heads for classification and localization consist of several convolution layers which are shared among each pyramid feature. For details, for -th pyramid feature, classification head produces category scoreˆ∈ R / ×C , where C is the number of classes. And localization head predicts distance between current temporal location to action boundaries, denoted as Following AFSD [12], the quality branch is also adopted in the baseline model which is expected to suppress low quality proposals.
Based on this baseline model, we further propose two modules named Boundary Evaluate Module (BEM) and Region Evaluate Module (REM) to address the issue of misalignment between classification confidence and localization accuracy. Noteworthily, the proposed BEM and REM are generic and easily combined not only with the above baseline framework but also with other anchor-free methods that have a similar pipeline. The details of BEM and REM would be explained in the rest of this section.

Boundary Evaluate Module
As discussed in Sec. 1, the misalignment between classification confidence and localization accuracy would lead detectors to generate inaccurate detection results. To address this, we propose Boundary Evaluate Module (BEM) to extract features and predict action boundary quality maps from a moment view which is complementary to the region view, thus it can provide more reliable quality scores of proposals.
Single-scale Boundary Quality. Boundary quality maps provide localization quality scores for each temporal location. The quality score is only dependent on the distance from the current location to the location of the action boundary of ground truth.
In practice, we set predefined anchors 1 at each temporal location, denoted as { } =1 , where = [ − /2, + /2] denotes the anchor at -th temporal location and is the predefined anchor size. For a video with action ground truth { , , , } =1 , we define start and end region for -th instance as = [ , − /2, , + /2] and = [ , − /2, , + /2]. The boundary quality maps for start boundary and end boundary , ∈ R are calculated by where tIoU is temporal IoU. The parameter controls the region size, examples for small and large are shown in Fig. 3 denoted as Small scale and Large scale separately. In this way, each score in the quality map indicates the location precision of the start or end boundary. In the inference phase, proposal boundary quality is acquired by interpolation at the corresponding temporal location. Previous works [16,36] formulate the prediction of boundary probability as a binary classification task that can't reflect the relative probability differences between two different locations. However, we define precise boundary quality using tIoU between the predefined anchor and boundary region. Moreover, previous works define positive locations by action length (e.g., locations lie in [ − /10, + /10] are positive samples in [16] and [36], where and are action length and start location of ground-truth). Thus, the model has to acquire the information of the duration of actions. But it is difficult because of the limited reception field, especially for long actions. So, the definition of boundary quality in Eq. 2 is regardless of the duration of actions. Another weakness of previous works is that they define the action boundary using a small region which leads to that only the proposal boundary closing to the ground-truth boundaries being covered. In this work, we can adjust to control the region size. We demonstrate that small region size is harmful to performance in our ablation.
Multi-scale Boundary Quality. Actions with different duration require different sensitivity to the boundary changes. Fig. 3 helps us to illustrate this. If we use Small scale, a short proposal and a long proposal (blue lines) with the same localization error of start boundary acquire the same boundary qualities of 0.41, but the actual tIoU of the long proposal is 0.9. Similarly, if we use Large scale, these two proposals acquire boundary qualities of 0.95, but the actual tIoU of the short proposal is 0.57. Thus, single-scale boundary quality is suboptimal for varied action duration. The scale should dynamically adapt the duration of actions. To address this, we expand the single-scale boundary quality maps into quality maps with multi-scale anchors. Thus, for a proposal, we can choose a suitable anchor depending on its duration (as yellow points show in Fig. 3).
In detail, start and end boundary quality maps are extended to two dimensions corresponding to temporal time steps and anchor scales, denoting as , ∈ R × , where is the number of predefined anchors. We predefine multiple anchors with different size at each temporal location, denoting as { } =1 , where = { ,1 , · · · , , } denoting predefined anchors. The anchor size is defined as representing evenly spaced number from to , where and indicate the maximum and minimum anchor scale. In this paper, is set as 1 that corresponds to the interval time between adjacent input video frames and depends on the distribution of duration of the actions in datasets. We conduct ablation studies about the selection of in Sec. 4.2. Thus the -th anchor at is , = [ − /2, + /2]. As for a ground-truth ( , , , ), its start and end region for -th anchor can be denoted as , = [ , − /2, , + /2] and , = [ , − /2, , + /2]. Then the multi-scale quality maps , ∈ R × are calculated by In the inference phase, the boundary quality of the proposal is obtained by bilinear interpolation according to the boundaries location and the proposal duration (See Sec.3.4). Implementation. To predict multi-scale boundary quality maps, as shown in Fig.2, the backbone feature is first fed into an upsampling layer and several convolution layers to get the framelevel feature ∈ R × with a higher temporal resolution, which is beneficial to predict quality score of the small anchor. Because the anchor scales may have a large range and different scales need different receptive fields, we adopt a parameter-free and efficient method to generate features. In detail, we use linear interpolation in each anchor to obtain the multi-scale anchor feature map, denoted as ∈ R × × × . In particular, for ( , ) ∈ R × , we uniformly sample features in the scope [ − /2, + /2] from which ensures that the receptive field matches the anchor size. This procedure of interpolation can be efficiently achieved by matrix product [14]. After the multi-scale anchor feature map is obtained, we apply max pooling on the sampled features and a 1 × 1 convolution to extract anchor region representation : where ∈ R × × . Finally, two boundary score maps are obtained based on as follows: where (·) and (·) are convolution layers and (·) is sigmoid function.
Training. We denote label maps forˆandˆas , ∈ R × respectively. The label maps is computed by Eq. 4. We take points where / > 0 as positive. L2 loss function is adopted to optimize BEM, which is formulated as follows: where N / is the set of positive points.

Region Evaluate Model
BEM estimates the localization quality of proposals in the moment view that focuses more on local appearance and motion evolvement.
Although it achieves considerable improvement, as illustrated in Tab. 4, we believe that feature of the region view can provide rich context information which is beneficial to the prediction of localization quality. Therefore, we propose Region Evaluate Module (REM), as shown in the right part of Fig. 2, which first predicts coarse action proposals and then extracts features of proposals to predict localization quality scores, action categories, and boundary offsets. Specifically, REM predicts coarse action offset (ˆ,ˆ), action categoriesˆand quality scoreˆfor each temporal location (omitting subscript standing for temporal location for simplicity). For a location with coarse offset prediction which indicates the distance to start and end of the action boundaries, the corresponding proposal can be denoted asˆ= ( −ˆ, +ˆ). Then three features are sampled from pyramid feature at { −ˆ, , +ˆ} via linear interpolation and aggregated by a fully-connected layer. This procedure is illustrated in Fig. 4. Based on the aggregated feature, BEM produces refined boundary offsets (Δˆ, Δˆ), quality scores˜and category scores˜. The final outputs can be obtained by where , , are final action proposal, action category score and location quality score respectively andˆ=ˆ+ˆ.
Training. The loss of REM is formulated as: ℓ =l +l +l +l +l +l , where , are loss weight.l andl are focal loss [13] for category prediction.l andl are loss of quality prediction, which is implemented by binary cross entropy loss. tIoU between proposal and corresponding ground-truth is adopted as target of quality prediction:l where is ground-truth for location .l is generalized IoU loss [23] for location prediction of initial proposal andl is L1 loss for offset prediction of the refining stage: where N indicates the ground-truth action locations, and Δ = 2 · ( −ˆ)/ˆ,ˆis coarse proposal length.

Training and Inference
Training details. Since there are mainly two different strategies for video feature extraction, including online feature extraction [12,31] and offline feature extraction [14,15,19], we adopt different training methods for them. For frameworks using the online feature extractor, BEM and REM are trained jointly with the feature extractor in an end-to-end way. The total train loss function is where is used to balance loss. As for methods with the offline feature extractor, since BEM is independent of other branches, we individually train BEM and other branches, then combine them in the inference phase for better performance.
Inference. The final outputs of REM is calculated by Eq. 8. Thus, the generated proposals can be denoted as {( , , ) } =1 , where = ( , ) and is the number of proposals. In order to obtain boundary quality, we define a function that generates index of appropriate anchor scale in multi-scale boundary quality map according to the action duration, denoted as ( ). We adopt a simple linear mapping: where is a predefined mapping coefficient. For a proposal, controls the anchor size used by it. We explore the influence of in our ablation. Then start and end boundary quality are acquired by bilinear interpolation, where is bilinear interpolation and = − is the length of proposal. After fusing these scores, the final proposals is denoted as {( · · √ , · , , )} =1 .

EXPERIMENTS
Dataset. The experiments are conducted on two popularly used datasets, THUMOS14 [10] and ActivityNet-1.3 [9]. THUMOS14 contains 200 untrimmed videos in the validation set and 212 untrimmed videos in the testing set with 20 categories. Following previous works [14,15,37], we train our models on the validation set and report the results on the testing set. ActivityNet-1.3 contains 19,994 videos of 200 classes with about 850 video hours. The dataset is split into three subsets, about 50% for training, and 25% for validation and testing. Following [14,15,32], the training set is used to train the models, and results are reported on the validate set.
Implementation Details. For THUMOS14 dataset, we sample 10 frames per second (fps) and resize the spatial size to 96 × 96. Same as the previous works [12,14], sliding windows are used to generate video clips. Since nearly 98% action instances are less than 25.6 seconds in the dataset, the windows size is set to 256. The sliding windows have a stride of 30 frames in training and 128 frames in testing. The feature extractor is I3D [5] pre-trained in Kinetics. The mean Average Precision (mAP) is used to evaluate performance. The tIoU thresholds of [0.3 : 0.1 : 0.7] are considered for mAP and average mAP. If not noted specifically, we use Adam as optimizer with the weight decay of 10 −3 . The batch size is set to 8 and the learning rate is 8 × 10 −4 . As for loss weight, , , are set to 5, 1 and 0.5. The anchor scale and mapping coefficient in BEM are {1, 50, 20} and 2. In the testing phase, the outputs of RGB and Flow are averaged. The tIoU threshold of Soft-NMS is set as 0.5.
On ActivityNet-1.3, each video is encoded to 768 frames in temporal length and resized to 96 × 96 spatial resolution. I3D backbone is pre-trained in Kinetics. mAP with tIoU thresholds {0.5, 0.75, 0.95} and average mAP with tIoU thresholds [0.5 : 0.05 : 0.95] are adopted. Optimizer is Adam with weight decay of 10 −4 . Batch size is 1 and learning rate is 10 −5 for feature extractor and 10 −4 for other components. As for loss weight, , , are set to 5, 1 and 1 repestively. The anchor scale and mapping coefficient in BEM are {1, 130, 22} and 2. The tIoU threshold of Soft-NMS is set to 0.85. In order to validate the generalizability of our method, we also evaluate the performance when integrating BREM with methods using the offline feature extractor. ActionFormer [35] is the latest anchor-free TAD method that shows strong performance. Thus we integrate BREM with ActionFormer to validate the effectiveness of BREM. The implementation details are shown in our supplement.
The results on the testing set of THUMOS14 are shown in Tab. 2. Our baseline achieves 53.1% AP@Avg outperforming most of the previous methods. Based on the strong baseline, BREM absolutely improves 3.6% from 53.1% to 56.7% on AP@Avg. It can be seen that the proposed BREM acquires improvement on each tIoU threshold compared with the baseline. Especially on high tIoU thresholds, BREM achieves an improvement of 5.4% on AP@0.7. Similarly, integrating BREM with ActionFormer [35] provides a performance gain of 1.3% on AP@0.5 and yields a new state-of-the-art performance of 63.6% on AP@Avg.  The results on ActivityNet-1.3 validation set are shown in Tab. 3. Integrating BREM with baseline (Base) reaches an average AP of 34.3%, which is 0.7% higher than baseline. And BREM achieves an average AP of 36.2% when combined with ActionFormer (AF) using the pre-training method from TSP [1], which is the best result using the features from [29]. It is worthy to note that BREM brings considerable improvement on middle tIoU thresholds, outperforming ActionFormer by 1.6% on AP@0.75. TCANet [19] is the only model better than ours, but it uses the stronger SlowFast feature [7] and refines proposals generated by a strong proposal generation method [14].

Ablation Study
We conduct ablation experiments on THUMOS14 for the RGB model based on the baseline to validate the effectiveness of our method. The AP at tIoU=0.5, 0.6 and 0.7, and average AP in [0.3 : 0.1 : 0.7] are reported. Each experiment is repeated three times and the average result is presented to obtain more convincing results.
Effectiveness of Model Components. In order to analyze the effectiveness of the proposed BEM and REM, each component is applied in the baseline model gradually. Meanwhile, the result of the combination of BEM and REM is also presented to demonstrate they are complementary to each other. All results are shown in Tab. 4. Obviously, BEM boosts the average AP by 2.2%. The significant improvement brought by BEM confirms that BEM helps to preserve better action proposals based on the more accurate quality score of boundary localization. Meanwhile, REM improves the average AP by 1.2%. This suggests that aligned features are beneficial for refining more accurate boundaries, classification, and quality scores. By combining BEM and REM, the performance is further improved Effectiveness of Boundary Quality. In order to demonstrate the effectiveness of boundary quality, we first analyze its importance by introducing single-scale boundary quality. Then the comparison between single-scale and multi-scale boundary quality is conducted to validate the necessity of introducing more anchor scales. Finally, different settings of boundary anchors are explored. Results are shown in Tab. 5. For single-scale boundary quality with anchor size=4, the AP@Avg drops from 44.2% to 42.6%. We conjecture that the reason is that the estimated boundary quality at the most temporal locations can not reflect the actual location quality because of the small anchor size (see Fig. 3 Small scale). Increasing the anchor size boosts the performance. The best result is reached with anchor size=28, and further increasing the anchor size harms the performance. For multi-scale boundary quality, we gradually increase the largest anchor size ( ). As shown in Tab. 5, increasing improves the performance, and saturation is reached when = 50 because there are few long actions in the dataset thus too large anchors are rarely used. The above results suggest that our single-scale boundary quality can help preserve better predictions in NMS, but a suitable anchor size has to be carefully chosen. Contrary to single-scale boundary quality, multi-scale boundary quality introduces further improvement by dividing actions into different appropriate anchor scales depending on their duration. It can be seen that the anchor size of {1, 50, 20} brings a 1% improvement compared with single-scale boundary quality. Furthermore, it is less sensitive to the choice of anchor size.
Effectiveness of REM. Based on the aligned feature, REM refines the location, category score, and localization quality score of each action proposal. We gradually remove each component to show its effectiveness. The results are shown in Tab. 6. Removing offset, quality, and classification drop the performance by 0.8%, 0.1%, and 0.5% respectively. Refinement of location and category score bring more noticeable improvement to the model than quality score. We preserve quality score refinement in our final model since it can stable the performance and only increases negligible computation. Previous work [12] extracts salient boundary feature by boundary max pooling, while we extract the region feature of the proposal by interpolation which is more efficient and shows competitive performance.
Ablation study on regional feature extraction method in REM. We explore different feature extraction methods in REM, 1) FC: all sampled features in each anchor region are concatenated and a fully connected layer is applied to convert them to the target dimension. 2) Mean: the mean operation is applied to all sampled  Ablation study on mapping coefficient in BEM. The mapping coefficient in BEM controls the corresponding anchor size of the proposal in the inference phase (See Eq. 13). For a proposal, it will use a smaller scale anchor if enlarging . We vary the mapping coefficient ∈ {0.5, 1.0, 2.0, 3.0} in the inference phase and report the results in Tab. 8. The performance is stable if equals to 1.0 or 2.0. Smaller and larger will decrease the performance since the anchor size and the duration of action are not appropriately matched, which also confirms the importance of multi-scale boundary quality.

CONCLUSION
In this paper, we reveal the issue of misalignment between localization accuracy and classification score of current TAD methods. To address this, we propose Boundary Evaluate Module and Region Evaluate Module (BREM), which is generic and plug-and-play. In particular, BEM estimates the more reliable proposal quality score by predicting multi-scale boundary quality in a moment perspective. Meanwhile, REM samples region features in action proposals to further refine the action location and quality score in a region perspective. Extensive experiments are conducted on two challenging datasets. Benefiting from the great complementarity of moment and region perspective, BREM achieves state-of-the-art results on THUMOS14 and competitive results on ActivityNet-1.3.

A EXPERIMENTAL DETAIL
The proposed Boundary Evaluate Module and Region Evaluate Module (BREM) is generic, and it is easily combined with other anchor-free frameworks to achieve better results. Fig. 2 illustrates the overall architecture of anchor free methods combined with BREM, where BEM is integrated as an extra component and REM is adopted to replace the original detection head. In our experiments, BREM is combined with two typical frameworks, a basic anchor-free framework (denoted as Base) and ActionFormer [35]. The implementation details are described in this section.

A.1 The architecture of Boundary Evaluate Module
The implementation of Boundary Evaluate Module (BEM) is shown in Tab 9. The input feature of BEM is frame level feature ∈ R × with the time resolution same as the input of backbone, which preserves more detail information of appearance and motion. In the experiments, the number of sample points is set to 14 for the balance of efficiency and effectiveness.
Finally, the refined action prediction is obtained via Eq. 8 in the main paper.

A.3 Combination with Base
Detailed architecture. We adopt a previous successful feature pyramid proposed by AFSD [12] and its architecture is shown in Fig. 6. I3D [5] is used to extract the semantic feature of videos and the feature of 4 ℎ and 5 ℎ stage (C4 and C5) are used to generate a feature pyramid. Meanwhile, an up-sample layer and a convolutional layer are used to produce the frame-level feature. Finally, pyramid features and frame level feature are fed into REM and BEM, respectively.     , and the channel of all convolutional layers is 512. ⊕ denotes element-wise summation.

A.4 Combination with ActionFormer
Detailed architecture. Contrary to Base, ActionFormer [35] uses off-the-shelf features. Frame level feature is generated by applying a 1 with kernel size=3 and stride=1 on pre-encoded video features. Then is fed into BEM and the original head of ActionFormer is replaced with REM.