## Regional quality estimation for echocardiography using deep learning

Gilles Van De Vyver , Svein-Erik M˚søy , H˚vard Dalen 1 a 1 a 1,2 , Bjørnar Leangen Grenne 1,2 , Espen Holte 1,2 , Sindre Hellum Olaisen , John Nyberg , Andreas Østvik 1 1 1,3 , Lasse Løvstakken , 1 and Erik Smistad 1.3

1 Norwegian University of Science and Technology, Trondheim, Norway 2 St. Olavs hospital, Trondheim, Norway 3 Health Research, SINTEF, Trondheim, Norway

## Abstract

Automatic estimation of cardiac ultrasound image quality can be beneficial for guiding operators and ensuring the accuracy of clinical measurements. Previous work often fails to distinguish the view correctness of the echocardiogram from the image quality. Additionally, previous studies only provide a global image quality value, which limits their practical utility. In this work, we developed and compared three methods to estimate image quality: 1) classic pixel-based metrics like the generalized contrast-to-noise ratio (gCNR) on myocardial segments as region of interest and left ventricle lumen as background, obtained using a U-Net segmentation 2) local image coherence derived from a U-Net model that predicts coherence from B-Mode images 3) a deep convolutional network that predicts the quality of each region directly in an end-to-end fashion. We evaluate each method against manual regional image quality annotations by three experienced cardiologists. The results indicate poor performance of the gCNR metric, with Spearman correlation to the annotations of ρ = 0 24. . The end-to-end learning model obtains the best result, ρ = 0 69, comparable to . the inter-observer correlation, ρ = 0 63. . Finally, the coherence-based method, with ρ = 0 58, outperformed . the classical metrics and is more generic than the end-to-end approach. The image quality prediction tool is available as an open source Python library at https://github.com/GillesVanDeVyver/arqee .

## Keywords :

Cardiac segmentation, Ultrasound, image quality, Coherence, Signal-To-Noise Ratio

## 1 Introduction

Image quality is one of the main challenges in ultrasound imaging and can differ significantly between patients and imaging equipment. In echocardiography, many factors influence image quality such as the ultrasound scanner, the patient, and the probe. Several quantitative measurements using the images are performed. However, this requires image quality good enough for the given measurement. Different measurements have different image quality requirements, for instance, left ventricular (LV) volume, ejection fraction (EF), and strain measurements require good image quality in the entire myocardium. On the other hand, mitral annular plane systolic excursion (MAPSE) only requires good image quality in the annulus. Good image quality should generally provide measurement values with low uncertainty. Estimating image quality can be useful in the following ways:

- · To guide operators to achieve as good image quality as possible while scanning.
- · To automatically select the best images, recordings, and the best cardiac cycles to use for a given measurement.
- · As quality assurance, e.g. to warn the user when an image is not good enough for a measurement, and to automatically approve/disapprove individual myocardial segments based on image quality.
- · In data mining projects, to exclude cases with insufficient quality for reliable measurements.

We distinguish between two types of quality of ultrasound images: view quality/correctness and image quality. In this work, we will focus on image quality specifically. For view correctness, previous work has demonstrated that 3D ultrasound can serve as training data to automatically identify the transducer rotation and tilt in relation to the desired standard view and can guide the user to the correct position [1-3]. For

image quality, the classic ultrasound signal-processing metrics are the contrast ratio (CR) [4], contrast-to-noise ratio (CNR) [5], and generalized CNR (gCNR) [6]. These three metrics need a region of interest (ROI) and a background region to compare against. More recently, global image coherence (GIC) [7] has been proposed as a general quality metric that does not require the selection of these two regions. The image coherence measures how well the signals of the transducer elements align after delay compensation, with more alignment corresponding to clearer and sharper images. From the above mentioned methods, only the GIC can be used directly and automatically for measuring image quality separately as it does not require selecting an ROI and noise region. However, this approach requires channel data, which is not readily available in practice and does not give regional metrics.

Several automatic methods for measuring ultrasound image quality have been published applicable to cardiac imaging. Abdi et al. [8] used a recurrent neural network to predict the global quality of cardiac cine loops. The criteria for quality assessment take both image quality and view correctness into account. In subsequent studies [9, 10], they used an architecture that performs both view classification and global quality prediction simultaneously. The image quality metric is a global criterion based on the manual judgment of the clarity of the blood-tissue interface. Labs et al. [11] used a multi-stream neural network architecture where each stream takes in a sequence of frames and predicts a specific quality criterion. The criteria are global and take both view correctness and image quality into account. Karamalis et al. [12] detect attenuated shadow regions with random walks resulting in a pixel-level confidence map. Unlike the other methods above, this method is not based on deep learning. It provides a local, pixel-level metric, but it only measures the visibility of regions and not the quality of their content.

All of the automatic methods mentioned above have the limitation that they only provide a global image quality evaluation and/or do not assess the image quality separate from the view correctness.

## 2 Methods

In this work, we developed and compared three fully automatic methods to asses regional image quality in cardiac ultrasound separate from the view correctness:

- · Classical ultrasound image quality metrics, such as contrast ratio and contrast to noise ratio, applied in cardiac regions automatically extracted using deep-learning segmentation.
- · Deep-learning predicted ultrasound coherence, which is a measure of how coherent a signal is received by the transducer elements, together with deep-learning segmentation.
- · End-to-end prediction of regional image quality.

The rest of this section first presents the datasets used to develop these methods, and then presents each of the three methods.

## 2.1 Datasets

## 2.1.1 VLCD

The Very Large Cardiac Channel Data Database (VLCD) consists of channel data from 33280 frames from 538 recordings of 106 study participants [7]. It contains parasternal short axis (PSAX), parasternal long axis (PLAX), apical long axis (ALAX), apical two-chamber (A2C), and apical four-chamber (A4C) views. We split the VLCD dataset on the study participant level into train, validation, and test sets, 70%, 15%, and 15% respectively.

## 2.1.2 HUNT4

The Nord-Trøndelag Health Study dataset (HUNT4Echo) is a clinical dataset including among others PSAX, PLAX, ALAX, A2C, and A4C views. Each recording contains 3 cardiac cycles. We use two subsets of the HUNT4Echo dataset.

- · Segmentation annotation dataset A fraction of 311 study participant exams, the segmentation annotation set [13], contains single frame segmentation annotations in both ED and ES as pixel-wise labels of the left ventricle (LV), left atrium (LA), and myocardium (MYO) in ALAX, A2C, and A4C views.

Table 1: Definitions of image quality labels for annotating cardiac regions of interest. We only look at the signal quality of the region that is inside the ultrasound sector. If more than 50% is outside the sector, the region is treated as out-of-sector and excluded from analysis.

| image qual- ity label   | Definition                                                                                                                                                           |   Quality score |
|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|
| Not visible             | Less than 50% of what is inside the sector is visible.                                                                                                               |               1 |
| Poor                    | Low signal-to-noise ratio and/or smearing/low resolution, but can still see at least 50% of the my- ocardium/annulus. Diffuse border between annulus and cavity.     |               2 |
| Ok                      | Medium signal-to-noise ratio and/or some smearing/lower resolution, can see at least 70% of the my- ocardium/annulus.                                                |               3 |
| Good                    | Good signal-to-noise ratio and clear border in at least 80% of the myocardium/annulus. Very little smearing/good resolution. Clear border between annulus and cavity |               4 |
| Excellent               | Within the best 10% of what you have seen on this system.                                                                                                            |               5 |

- · Regional image quality dataset For this work, we created an additional dataset of image quality labels. The local image quality labels are manual annotations that asses the image quality of the cardiac regions of interest on a subset of the HUNT4 dataset in ALAX, A2C, and A4C views. Section 2.2 describes the annotation process in more detail.

## 2.2 Regional image quality annotation on HUNT4

An annotation tool was developed specifically for this project using the open-source Annotation Web software 1 [14]. The tool was made to enable clinicians to annotate regional image quality as efficiently and accurately as possible. The tool is freely available and can be adapted to other image quality projects. The image quality annotations were performed by three experienced cardiologists using the following protocol:

- 1. Annotate the end-diastole (ED) and end-systole (ES) frame of each recording, and optionally other frames if the image quality changes significantly during the recording.
- 2. If the majority of the cardiac regions of interest is out-of-sector, label it as out-of-sector. Otherwise, label the part of the region that is inside the sector according to the definitions in Table 1. We ignore the out-of-sector regions in the remainder of this work.

For the first round of annotations, each of the three clinicians annotated the same 10 frames from 5 recordings of 2 study participants. We used this dataset to calculate the inter-observer variability. For the second round of annotations, the three clinicians collectively annotated 458 frames from 158 recordings of 65 study participants. The annotations from the second round form the regional image quality dataset . This dataset was split randomly at the study participant level into train, validation, and test sets, allocating 70%, 15%, and 15% of the data to each set respectively.

## 2.3 Regional image quality estimation

## 2.3.1 Classical ultrasound image quality metrics

For the classical image quality metrics, deep-learning segmentation is used to extract the annulus regions and each of the myocardial segments as regions of interest and the LV as the background region. Appendix A gives more details about the procedure for dividing the segmentation into regions. The four classical ultrasound image quality metrics below were tested. We apply histogram matching [15, 16] to a Gaussian distribution ( µ = 127 , σ = 32) for the B-Mode grayscale images before calculating pixel-based quality metrics.

- · Pixel intensity is the average pixel intensity value in each region.
- · Contrast Ratio ( CR ) [4] is defined as

$$C R = \frac { \mu _ { \text{segment} } } { \mu _ { \text{LN} } }$$

where µ segment is the average intensity in each region and µ LV is the average intensity inside the LV lumen.

1 https://github.com/smistad/annotationweb

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-65b16ef3-72f4-497d-a141-b9ceaee88426.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-03426d83-dea7-472f-96e4-b9c0e24f58b9.jpg)

(a) Segmentation

(b) Probability density functions

Figure 1: Calculating gCNR for a region of the myocardium. The left side of the figure shows the segmentation with the MYO and annulus points divided into regions. The right side shows the probability density functions of the segment and background pixels used to calculate the gCNR. In this example, we use the mid region on the left side as ROI and the full LV lumen as background. These correspond to the green and red masks respectively in the left part of the figure.

## · Contrast to Noise Ratio ( CNR ) [5] is defined as

$$C N R = \frac { \mu _ { \text{segment} } - \mu _ { \text{LV} } } { \sqrt { \sigma _ { \text{segment} } ^ { 2 } + \sigma _ { \text{LV} } ^ { 2 } } }$$

where σ segment is the standard deviation in each region and σ LV is the standard deviation inside the LV lumen.

- · Generalized CNR ( gCNR ) [6] is defined as the maximum performance that can be expected from a hypothetical pixel classifier based on intensity using a set of optimal thresholds. It is calculated as

$$\ g C N R = 1 - \frac { 1 } { 2 } \sum _ { i = 0 } ^ { M A X _ { i } } \min \{ p _ { \text{segment} } ( i ), p _ { \text{LV} } ( i ) \}$$

where p segment ( x ) is the probability density function of the pixel intensities inside the region the gCNR is calculated for, p LV ( x ) the probability density function of the pixel intensities inside the LV lumen, and MAX i the maximum possible pixel intensity. Fig. 1 shows an example of the probability density functions for one of the regions as ROI and the LV lumen as background.

## 2.3.2 Local, deep-learning predicted image coherence as quality metric

We use the VLCD dataset to calculate the coherence factor [17] for each pixel in the ultrasound image. This factor is the ratio between the amplitude of the sum of the received signals to the sum of the amplitudes of those signals,

$$C F = \frac { \sum _ { n = 1 } ^ { N } S _ { i } } { \sum _ { n = 1 } ^ { N } | S _ { i } | }$$

where S i is the delayed signal for the i -th transducer element. This is equivalent to taking the coherent sum of the signal and dividing it by the incoherent sum of each signal. Thus, the coherence factor measures of how well the complex signals of all transducer elements align. The remainder of the signal-processing chain is the native processing of the GE HealthCare Vivid E95 system 2 but without the log compression. The result is a coherence image with the same dimension as the B-Mode image. The final preprocessing step applies gamma normalization with γ = 0 5 on the coherence images, .

$$t _ { i, n o r m a l i z e d } = t _ { i } ^ { \gamma }$$

2 Gundersen et al. [18] describe this signal-processing chain in more detail.

Table 2: Characteristics of the coherence prediction network. The general architecture is a U-Net. The 'number of channels' row indicates the number of channels at the first, bottom, and last convolution of the U-Net respectively.

| Input size Number of channels Number of output Lowest resolution Upsampling scheme Normalization scheme Batch Size Optimizer Initial learning Rate Scheduler Loss function Inter-layer Activation Final layer Activation Epochs   | 256 × 256                                                                                                                                                                                                                                    |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|                                                                                                                                                                                                                                   | 16 ↓ 128 ↑ 16                                                                                                                                                                                                                                |
| channels                                                                                                                                                                                                                          | 1                                                                                                                                                                                                                                            |
|                                                                                                                                                                                                                                   | 4 × 4                                                                                                                                                                                                                                        |
|                                                                                                                                                                                                                                   | 2 × 2 repeats                                                                                                                                                                                                                                |
|                                                                                                                                                                                                                                   | BatchNorm                                                                                                                                                                                                                                    |
|                                                                                                                                                                                                                                   | 32                                                                                                                                                                                                                                           |
|                                                                                                                                                                                                                                   | Adam                                                                                                                                                                                                                                         |
|                                                                                                                                                                                                                                   | 1e-2                                                                                                                                                                                                                                         |
|                                                                                                                                                                                                                                   | None                                                                                                                                                                                                                                         |
|                                                                                                                                                                                                                                   | Negative structural similarity index measure (SSIM) [20]                                                                                                                                                                                     |
|                                                                                                                                                                                                                                   | Gaussian error linear unit (GELU) [21]                                                                                                                                                                                                       |
|                                                                                                                                                                                                                                   | Sigmoid                                                                                                                                                                                                                                      |
|                                                                                                                                                                                                                                   | 500 Rotations ( - 45 ◦ ≤ angle ≤ 45 ◦ ), horizontal mirroring, gamma correc-                                                                                                                                                                 |
| Augmentations                                                                                                                                                                                                                     | tion (0 . 75 ≤ γ ≤ 1 . 25, only on B-mode), scaling (0 . 75 ≤ magnification ≤ 1 . 25), contrast/brightness adjustments (0 . 2 ≤ gain ≤ 2, 0 ≤ bias ≤ 128, only on B-mode). Each augmentation is applied individually with 0 . 5 probability. |

where t i are the pixels of the target coherence image. The corresponding B-mode images are generated from the channel data using the same, native signal-processing pipeline.

The HUNT4 dataset, like most ultrasound datasets, does not include channel data. Therefore, VLCD was used to train an image-to-image network that takes as input the grayscale B-mode image and predicts the coherence image, which is then used to calculate local image coherence. We use a lightweight U-Net architecture inspired by the U-Net 1 architecture in [19], with characteristics listed in Table 2. As coherence is related to image quality, we only apply augmentations that do not influence the quality of the image. Furthermore, the coherence should be invariant to different gain settings, so we additionally augment with brightness adjustments on the B-mode image while keeping the target coherence image unchanged. During training and validation of the coherence prediction model, we sample a random frame from each recording in the train and validation set respectively during each epoch. During testing, we use all frames in the test set. The local image coherence quality metric of a region is the average pixel value of all pixels corresponding to the region in the coherence image. This is the same as the pixel intensity metric above but applied to the coherence image instead of the B-Mode image.

## 2.3.3 End-to-end deep-learning quality prediction

The end-to-end learning approach trains a convolutional neural network on the regional image quality dataset to predict the quality of each region directly. We treat the problem as a regression task where the model predicts a score for each segment. Table 1 shows the correspondence between quality scores and annotation labels. As architecture, we use a MobileNetV2 [22] that predicts the image quality labels of all regions simultaneously. Appendix B describes the ablation study conducted to justify this setup.

## 3 Experimental setup

## 3.1 Evaluation of coherence prediction from B-mode

We use the structural similarity index (SSIM) [20], peak signal-to-noise ratio (PSNR) 3 , and relative pixel error (RPE) to evaluate the coherence image prediction network. We define the RPE as

$$R P E = \frac { | t _ { i } - p _ { i } | } { m a x ( t _ { i }, \epsilon ) }$$

where t i are the pixel values of the target coherence image, p i the pixel values of the predicted coherence image, and ϵ = 1 e -4. Section 4.1 contains the results of this experiment.

## 3.2 Evaluation of quality metrics

The correlation and accuracy of each quality metric were measured by comparing them to the expert annotations on the test set of the regional image quality dataset. For the classic image quality methods and regional coherence

3 The maximum pixel value for coherence images is 1.

Table 3: Average metric values of the coherence prediction network on the test set of the VLCD.

method, linear regression models were used to map the quality metric values to image quality labels. The train and validation set were used together to fit the linear regression model and evaluate it on the test set. Section 4.2 contains the results of this experiment.

## 3.3 Comparison to inter-observer variability

We compare the end-to-end, local image coherence, and gCNR-based model to the inter-observer variability on the data obtained in the first round of annotation, explained in subsection 2.2. For the local image coherence and gCNR-based models, the same linear model as subsection 3.2 was used to map the metrics to quality labels. The inter-observer variability is calculated from the aggregate of the three unique pairwise score errors between each of the three annotators:

$$e _ { \text{inter-observer} } = e _ { 1 2 } \cup e _ { 2 3 } \cup e _ { 1 3 }$$

where e ij is the score difference between operator i and j. The error metrics of the automatic methods are calculated from the aggregate of the pairwise score errors between the output of the method and each of the three annotators:

$$e _ { M } = e _ { 1 M } \cup e _ { 2 M } \cup e _ { 3 M }$$

where e iM is the score difference between operator i and method M . Section 4.3 contains the results of this experiment.

## 3.4 Relation to variability in clinical measurements

This experiment evaluates whether there is a relation between the predicted quality and the agreement between different methods for clinical measurements. The hypothesis is that with lower image quality the variability, and thus the uncertainty, of the measurements between methods and between experts increases. More specifically, this analysis compares peak global longitudinal strain (GLS) and ejection fraction (EF) measurements obtained either fully automatically with AI tools or manually by using GE HealthCare EchoPAC software on HUNT4 [13,23]. For AI estimation of GLS and EF, the deep-learning methods proposed by Østvik et al. [24] and Smistad et al. [25] were used respectively. The study participants in HUNT4 used for model development were excluded from the analysis. For GLS, the predicted quality is the average quality of all segments over the full recording. For EF, the predicted quality is the average quality of all segments in the end-diastole (ED) and end-systole (ES) frames of all cycles in the recording. Section 4.4 contains the results of this experiment.

## 4 Results

## 4.1 Results of coherence prediction from B-mode

Table 3 summarizes the average metric values on the test set. Fig. 2 shows an example of the best, median, and worst-case predictions according to the relative pixel error. Fig. 3 illustrates how the predicted coherence images are almost independent of the brightness/gain and contrast/dynamic range of the input B-Mode images. The main finding is that the difference between the estimated and ground truth coherence images is small and thus the predicted coherence images can be used to obtain coherence-based quality metrics for B-mode for which the channel data is not available.

## 4.2 Results of quality metrics

Table 4 summarizes the results of the evaluation of the quality metrics. Fig. 4 shows box plots of the quality metrics per image quality label for the end-to-end, coherence, gCNR, and intensity models. Fig. 5 shows examples of B-mode images with varying quality together with labels from the annotators and automatic quality metrics. The main finding is that the end-to-end model performs the best, followed by the local image coherence metric. The classical ultrasound image quality metrics perform poorly.

Figure 2: Best, median, and worst case of image-to-image coherence prediction task with relative pixel errors of 4.0, 5.4, and 9.1% respectively. The first column shows the input B-mode image. The second column shows the coherence image as predicted by the image-to-image network. The third column shows the ground truth coherence image as calculated from the channel data. Finally, the rightmost column shows the color-coded difference of the target minus the predicted image.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-cc894e3b-c639-4b1c-ad2a-d02c39b99b2e.jpg)

Figure 3: Effect of brightness on coherence prediction. The first row shows a B-Mode image from the regional image quality dataset, brightened and darkened with gamma correction ( γ = 0.9 and 1.1). The second row shows the predicted coherence images generated by giving the corresponding input from the first row to the network. The predicted coherence is unaffected by the adjustments in brightness, apart from the saturation effect in the brightened image reducing the information in the input, as can be seen in the basal part of the inferolateral wall.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-4237c15f-a139-41ba-9127-523c01624a06.jpg)

Table 4: Comparison of quality metrics on the test set of the regional image quality dataset, i.e. the second round of annotations. Accuracy is defined as the ratio of agreement between the automatic measurement, rounded to the nearest quality category, and the annotation on the segment level.

| Quality metric        |   Spearman correlation | Mean absolute error (MAE)   | Accuracy   |
|-----------------------|------------------------|-----------------------------|------------|
| Pixel intensity       |                   0.3  | 1.31 ± 0.91                 | 20%        |
| CR                    |                   0.28 | 1.32 ± 0.92                 | 22%        |
| CNR                   |                   0.25 | 1.38 ± 0.96 0.              | 20%        |
| gCNR                  |                   0.2  | 1.50 ± 0.96                 | 15%        |
| Local image coherence |                   0.48 | 1.07 ± 0.80                 | 29%        |
| End-to-end learning   |                   0.65 | 0.59 ± 0.45                 | 48%        |

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-7997aa0f-d08e-4f1a-8331-61d094912f2a.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-332e8a9e-29fc-47ce-b915-301b66dbc20c.jpg)

(a) End-to-end model

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-627a8fa4-936e-4de8-8948-4b6a63a33b25.jpg)

(b) Local image coherence

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-939c20af-6114-4ce7-8642-9cbb52478026.jpg)

(c) gCNR

(d) Pixel intensity

Figure 4: Box plots of quality metrics versus regional quality labels on the test set of the regional image quality dataset. The predictions of the end-to-end model have the strongest correlation to the quality labels. The dotted line represents the linear regression model that maps the quality metrics to quality labels, as described in section 3.2. The inference output of the end-to-end model can be used directly without additional linear model.

Figure 5: Example cases of annotations and automatically predicted regional quality from the test set. The visualization uses the regional quality metrics to color-code the output of divided segmentation output. The end-to-end model predicts the regional qualities directly from the B-mode without using the segmentation output. The local image coherence metric uses the segmentation output to select ROI. The gCNR metric uses the segmentation output to select ROI and background region. The background region, i.e., the LV lumen, is not shown in the image.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-24551a25-6dfc-45f7-9282-9473500bde01.jpg)

## Comparison to inter-observer variability

Figure 6: Bar plot comparing inter-observer variability to automatic methods. A method with lower variability will have the most occurrences with low score errors. Here we can observe that the variability of the end-to-end model is on par with the inter-observer variability, while the two other methods are not.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-30792054-b602-48d0-b356-3a4c6500e3b2.jpg)

Table 5: Comparison of automatic methods to inter-observer variability on the first round of annotations. For this data, there are three annotation values for each quality label. Each of the automatic methods is compared against each of the three annotators and the results are averaged. For the inter-observer variability, the labels are compared pairwise between each of the three annotators.

| Method                                      | Spearman correlation   | MAE                                 | Accuracy    |
|---------------------------------------------|------------------------|-------------------------------------|-------------|
| End-to-end model Local image coherence gCNR | 0.69 0.58 0.24         | 0.59 ± 0.59 1.19 ± 1.01 2.23 ± 1.58 | 46% 27% 13% |
| Inter-observer                              | 0.63                   | 0.67 ± 0.60                         | 40%         |

## 4.3 Results of comparison to inter-observer variability

Fig. 6 shows the bar plot comparing the automatic methods to the inter-observer variability and Table 5 lists the corresponding average metric values. Using the Wilcoxon signed-rank test [26] and a significance level of p = 0 05, we find that the difference in Mean absolute error (MAE) between each of the methods is . statistically significant. The difference between the inter-observer MAE and the MAE of each of the methods is also statistically significant, i.e. the inter-observer MAE is higher than the MAE of the end-to-end model and lower than the MAE of the other two models.

## 4.4 Results of relation to variability in clinical measurements

Fig. 7 shows box plots visualizing the agreement between the measurements obtained automatically and with EchoPAC for each predicted quality category. The standard deviations in these plots represent how well the AI estimates agree with the manual references. The main finding is that the limits of agreement are narrower for higher qualities.

## 5 Discussion

## 5.1 Challenges and considerations

Assessing image quality based on human perception is inherently a subjective task, even when supported by clear definitions of image quality categories. This creates challenges for training and evaluation as there is no ground truth as the reference labels are a subjective estimation themselves. Therefore, it is not realistic to expect the automatic models to agree with reference labels as well as on tasks with well-defined ground truth labels. This, together with a rather fine scale of image quality categories, explains the low accuracies in Tables 4 and 5. Fig. 6 shows how the end-to-end model has on average less error than the annotators between each other. This indicates the model has learned to produce quality labels that strike a middle ground between the

## Effect of quality on agreement of AI GLS with EchoPAC GLS

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-f6f8aa0c-d953-44a2-94f2-c66c4386736f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-f9aab2cf-a88c-41e4-978f-c3df80e06d38.jpg)

## Effect of quality on agreement of AI EF with EchoPAC EF

(a) AI GLS [24] vs EchoPAC GLS (2D Strain tool) [23].

(b) AI EF [25] vs EchoPAC EF [13].

Figure 7: Box plots of the difference between clinical measurement values obtained automatically by AI [24,25] and reference measurements obtained manually using GE HealthCare EchoPAC on the HUNT4 data [13, 23] per image quality category, as predicted by the end-to-end model. The decrease in standard deviation with better image quality indicates a better agreement between the methods for higher image quality. Additionally, there is a noticeable change in bias between different quality categories. We believe this effect is partly caused by physiological differences correlated with image quality and is out of scope for this work.

subjective assessments of the annotators.

The end-to-end learning model overestimates low-quality regions and underestimates high-quality regions, as can be seen in Fig. 4a. This means that the model can only explain a limited amount of variability in the image quality labels and is a result of minimizing the mean squared error (MSE) while dealing with subjective, and thus noisy reference values with fixed boundaries. We can eliminate this effect by fitting a linear model on the validation set that maps predicted image quality to image quality labels and applying it when doing inference on the test set. This increases MSE but gives more uniform performance over the image quality labels.

One reason for the weak correlation between the annotations and the pixel-based methods is the rough selection of ROI and background region. Fig. 4 shows how the average metrics of the classic pixel-based and coherence metrics increase for each quality label until the good label, and then drop again for excellent . This is because on the one hand in these high-quality images, the blood speckles can be visible inside the LV lumen, which is used as background region, and on the other hand the myocardium tissue, which is used as ROI, is less blurred resulting in a smaller spread of pixels with high intensity. This can be seen for the anterolateral wall and apex in the rightmost column of Fig. 5. One possibility to only select regions belonging to the tissue is to perform automatic pixel selection methods like Otsu thresholding [27] or percentile filters, but in our experiments this reduced the performance even further.

## 5.2 Design choices

The different methods in this study have a trade-off between accuracy and versatility. The default end-to-end network gives the best results but requires specific image quality labels for the task. Next, the coherence-based method is more generic and can potentially be applied more generally without the need for view-specific image quality annotations. Rindal et al. [7] showed that the GIC is not significantly different between apical views, but is higher for apical views than parasternal views. Thus, while a single image-to-image model can learn to predict coherence for different views, the mapping from coherence to image quality should be done for each group of views separately. Another advantage is that coherence can be used to give a global image quality metric without the need for a segmentation model. Finally, the pixel-based methods can be applied automatically in the most general way given a segmentation model to select ROI and background regions but also give the lowest accuracy.

The ablation study of the end-to-end learning model showed that increasing the complexity of the model did not improve the performance. This is a result of the relatively small dataset size and the specific task of the model. For a more general model of image quality prediction with more varied input, e.g. one model for all cardiac views, a larger dataset and more complex model may be required.

Figure 8: Screenshot of the real-time demo application. The left side shows the input B-Mode image. The right side shows the output of the segmentation color-coded by the output of the end-to-end quality network. The color codes are the same as in Fig. 5.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-74c6962d-5ec5-48a1-bcfb-bfb656a57ecc.jpg)

## 5.3 Clinical use

Image quality estimation can be the first step towards a method for giving reliability estimates to clinical measurements and quality control of fully automatic methods. Fig. 7 shows that the variability in clinical measurements goes down with higher predicted quality. However, image quality is only one source of variability, so a reliability model would also need to include view correctness and other factors that determine whether a given input is difficult to assess.

More direct use cases of the quality prediction model include the automatic selection of the best frame to perform a clinical measurement when multiple options are available, data cleansing in data mining, and automatic disapproval of segments for regional strain analysis. All the methods explored in this work are computationally efficient and can be run in real-time while scanning, and can thus be used as a guidance tool to enable clinicians to acquire images with better image quality.

## 6 Real-time demo application

To showcase the functionality of the end-to-end, real-time quality network, a real-time application was developed using the FAST framework [28]. The demo is a split-screen application that shows the B-Mode input to the left and the segmentation regionally color-coded by the quality as predicted by the end-to-end network to the right. Fig. 8 provides a screenshot of the application in use. We provide a demo video [29] illustrating the application in action while a clinician operates a GE Vivid E95 scanner. The video can be accessed at https://doi.org/10.6084/m9.figshare.26413984.v2 .

## 7 Conclusion

In this work, we developed and compared different deep-learning methods for regional image quality estimation in cardiac ultrasound. We show that classic pixel-based methods, such as (g)CNR, together with automatic image segmentation, give low agreement with the quality assessment of cardiologists. We developed a U-Net model to predict the coherence factor for each pixel in the ultrasound image and showed that the resulting coherence image can be used to assess the image quality in a pixel-based way with better performance than the classic measures. The best results, below inter-observer variability, are obtained by using an end-to-end deep-learning model. Finally, we show higher predicted quality is associated with lower limits of agreement between fully automatic and manual methods for the clinical measurements EF and GLS.

## References

- [1] D. Pasdeloup, S. H. Olaisen, A. Østvik, S. Sabo, H. N. Pettersen, E. Holte, B. Grenne, S. B. Stølen, E. Smistad, S. A. Aase et al. , 'Real-time echocardiography guidance for optimized apical standard views,' Ultrasound in Medicine &amp; Biology , vol. 49, no. 1, pp. 333-346, 2023.
- [2] R. Droste, L. Drukker, A. T. Papageorghiou, and J. A. Noble, 'Automatic probe movement guidance for freehand obstetric ultrasound,' in Medical Image Computing and Computer Assisted Intervention-MICCAI

|      | 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part III 23 . Springer, 2020, pp. 583-592.                                                                                                                                                                                                                                                                                                                                                                                                                                |
|------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [3]  | A. Østvik, E. Smistad, S. A. Aase, B. O. Haugen, and L. Lovstakken, 'Real-time standard view classification in transthoracic echocardiography using convolutional neural networks,' Ultrasound in medicine & biology , vol. 45, no. 2, pp. 374-384, 2019.                                                                                                                                                                                                                                                                                                  |
| [4]  | S. Smith, H. Lopez, and W. Bodine Jr, 'Frequency independent ultrasound contrast-detail analysis,' Ultrasound in medicine & biology , vol. 11, no. 3, pp. 467-477, 1985.                                                                                                                                                                                                                                                                                                                                                                                   |
| [5]  | M. Patterson and F. Foster, 'Improvement and quantitative assessment of b-mode images produced by annular array/cone hybrids,' in Acoustical Imaging . Springer, 1984, pp. 477-477.                                                                                                                                                                                                                                                                                                                                                                        |
| [6]  | A. Rodriguez-Molares, O. M. H. Rindal, J. D'hooge, S.-E. M˚søy, a A. Austeng, M. A. L. Bell, and H. Torp, 'The generalized contrast-to-noise ratio: A formal definition for lesion detectability,' IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control , vol. 67, no. 4, pp. 745-759, 2019.                                                                                                                                                                                                                                            |
| [7]  | O. M. H. Rindal, T. G. Bj˚stad, a T. Espeland, E. A. R. Berg, and S. E. M˚søy, a 'A very large cardiac channel data database (vlcd) used to evaluate global image coherence (gic) as an in-vivo image quality metric,' IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control , 2023.                                                                                                                                                                                                                                                     |
| [8]  | A. H. Abdi, C. Luong, T. Tsang, J. Jue, K. Gin, D. Yeung, D. Hawley, R. Rohling, and P. Abolmaesumi, 'Quality assessment of echocardiographic cine using recurrent neural networks: Feasibility on five standard view planes,' in Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20 . Springer, 2017, pp. 302-310.                                                                                                          |
| [9]  | C. Luong, Z. Liao, A. Abdi, H. Girgis, R. Rohling, K. Gin, J. Jue, D. Yeung, E. Szefer, D. Thompson et al. , 'Automated estimation of echocardiogram image quality in hospitalized patients,' The International Journal of Cardiovascular Imaging , vol. 37, pp. 229-239, 2021.                                                                                                                                                                                                                                                                            |
| [10] | N. Van Woudenberg, Z. Liao, A. H. Abdi, H. Girgis, C. Luong, H. Vaseli, D. Behnami, H. Zhang, K. Gin, R. Rohling et al. , 'Quantitative echocardiography: real-time quality estimation and view classification implemented on a mobile android device,' in Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation: International Workshops, POCUS 2018, BIVPCS 2018, CuRIOUS 2018, and CPM 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16-20, 2018, Proceedings . Springer, 2018, pp. 74-81. |
| [11] | R. B. Labs, A. Vrettos, J. Loo, and M. Zolgharni, 'Automated assessment of transthoracic echocardiogram image quality using deep neural networks,' Intelligent Medicine , vol. 3, no. 03, pp. 191-199, 2023.                                                                                                                                                                                                                                                                                                                                               |
| [12] | A. Karamalis, W. Wein, T. Klein, and N. Navab, 'Ultrasound confidence maps using random walks,' Medical image analysis , vol. 16, no. 6, pp. 1101-1112, 2012.                                                                                                                                                                                                                                                                                                                                                                                              |
| [13] | S. Olaisen, E. Smistad, T. Espeland, J. Hu, D. Pasdeloup, A. Østvik, S. Aakhus, A. R¨sner, o S. Malm, M. Stylidis, E. Holte, B. Grenne, L. Løvstakken, and H. Dalen, 'Automatic measurements of left ventricular volumes and ejection fraction by artificial intelligence: clinical validation in real time and large databases,' European Heart Journal - Cardiovascular Imaging , p. jead280, 10 2023. [Online]. Available: https://doi.org/10.1093/ehjci/jead280                                                                                        |
| [14] | E. Smistad, A. Østvik, and L. Løvstakken, 'Annotation web-an open-source web-based annotation tool for ultrasound images,' in 2021 IEEE International Ultrasonics Symposium (IUS) . IEEE, 2021, pp. 1-4.                                                                                                                                                                                                                                                                                                                                                   |
| [15] | N. Bottenus, B. C. Byram, and D. Hyun, 'Histogram matching for visual ultrasound image comparison,' IEEE transactions on ultrasonics, ferroelectrics, and frequency control , vol. 68, no. 5, pp. 1487-1495, 2020.                                                                                                                                                                                                                                                                                                                                         |
| [16] | R. C. Gonzalez, Digital image processing . Pearson education india, 2009.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| [17] | K. W. Rigby, 'Method and apparatus for coherence filtering of ultrasound images,' Jun. 8 1999, uS Patent 5,910,115.                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| [18] | E. L. Gundersen, E. Smistad, T. S. Jahren, and S.-E. M˚søy, a 'Hardware-independent deep signal processing: A feasibility study in echocardiography,' IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control , 2024.                                                                                                                                                                                                                                                                                                                      |
| [19] | S. Leclerc, E. Smistad, J. Pedrosa, A. Østvik, F. Cervenansky, F. Espinosa, T. Espeland, E. A. R. Berg, P.-M. Jodoin, T. Grenier et al. , 'Deep learning for segmentation using an open large-scale dataset in 2d echocardiography,' IEEE transactions on medical imaging , vol. 38, no. 9, pp. 2198-2210, 2019.                                                                                                                                                                                                                                           |

| [20]   | Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, 'Image quality assessment: from error visibility to structural similarity,' IEEE transactions on image processing , vol. 13, no. 4, pp. 600-612, 2004.                                                                                                                                                        |
|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [21]   | D. Hendrycks and K. Gimpel, 'Gaussian error linear units (gelus),' arXiv preprint arXiv:1606.08415 , 2016.                                                                                                                                                                                                                                                              |
| [22]   | M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, 'Mobilenetv2: Inverted residuals and linear bottlenecks,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4510-4520.                                                                                                                                          |
| [23]   | J. Nyberg, E. O. Jakobsen, A. Østvik, E. Holte, S. Stølen, L. Lovstakken, B. Grenne, and H. Dalen, 'Echocardiographic reference ranges of global longitudinal strain for all cardiac chambers using guideline- directed dedicated views,' JACC: Cardiovascular Imaging , vol. 16, no. 12, pp. 1516-1531, 2023.                                                          |
| [24]   | A. Østvik, I. M. Salte, E. Smistad, T. M. Nguyen, D. Melichova, H. Brunvand, K. Haugaa, T. Edvardsen, B. Grenne, and L. Lovstakken, 'Myocardial function imaging in echocardiography using deep learning,' ieee transactions on medical imaging , vol. 40, no. 5, pp. 1340-1351, 2021.                                                                                  |
| [25]   | E. Smistad, A. Østvik, I. M. Salte, D. Melichova, T. M. Nguyen, K. Haugaa, H. Brunvand, T. Edvardsen, S. Leclerc, O. Bernard, B. Grenne, and L. Løvstakken, 'Real-time automatic ejection fraction and fore- shortening detection using deep learning,' IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control , vol. 67, no. 12, pp. 2595-2604, 2020. |
| [26]   | F. Wilcoxon, 'Individual comparisons by ranking methods,' Biometrics Bulletin , vol. 1, no. 6, pp. 80-83, 1945. [Online]. Available: http://www.jstor.org/stable/3001968                                                                                                                                                                                                |
| [27]   | N. Otsu, 'A threshold selection method from gray-level histograms,' IEEE transactions on systems, man, and cybernetics , vol. 9, no. 1, pp. 62-66, 1979.                                                                                                                                                                                                                |
| [28]   | E. Smistad, M. Bozorgi, and F. Lindseth, 'Fast: framework for heterogeneous medical image computing and visualization,' International Journal of computer assisted radiology and surgery , vol. 10, pp. 1811-1822, 2015.                                                                                                                                                |
| [29]   | G. V. D. Vyver, 'Regional quality estimation for echocardiography using deep learning - real-time demo,' 7 2024. [Online]. Available: https://figshare.com/articles/media/Regional quality estimation for echocardiography using deep learning - real-time demo/26413984                                                                                                |
| [30]   | F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, 'nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,' Nature methods , vol. 18, no. 2, pp. 203-211, 2021.                                                                                                                                                |
| [31]   | --, 'nnunet,' https://github.com/MIC-DKFZ/nnUNet, 2023.                                                                                                                                                                                                                                                                                                                 |
| [32]   | G. V. D. Vyver, S. Thomas, G. Ben-Yosef, S. H. Olaisen, H. Dalen, L. Løvstakken, and E. Smistad, 'Toward robust cardiac segmentation using graph convolutional networks,' IEEE Access , vol. 12, pp. 33 876-33 888, 2024.                                                                                                                                               |
| [33]   | M. Tan and Q. Le, 'Efficientnet: Rethinking model scaling for convolutional neural networks,' in Interna- tional conference on machine learning . PMLR, 2019, pp. 6105-6114.                                                                                                                                                                                            |
| [34]   | S. Eppel, 'Classifying a specific image region using convolutional nets with an roi mask as input,' arXiv preprint arXiv:1812.00291 , 2018.                                                                                                                                                                                                                             |
| [35]   | L. Hou, C.-P. Yu, and D. Samaras, 'Squared earth mover's distance-based loss for training deep neural networks,' arXiv preprint arXiv:1611.05916 , 2016.                                                                                                                                                                                                                |
| [36]   | J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248-255.                                                                                                                                                      |

## A Extraction of cardiac regions of interest

We use the nnU-Net [30,31] architecture to segment the cardiac images. The nnU-Net is used out of the box using the default configuration but without the final ensemble step. Instead, we train and validate on a single predefined 80% train, 10% validation, and 10% test split from the HUNT4 segmentation annotation dataset. Table 6 summarizes the characteristics of the nnU-Net architecture. This model is described in more detail and compared to other segmentation models in our previous work [32]. We use two nnU-Nets, one for apical two(A2C) and four-chamber (A4C) views, and one for apical long axis (ALAX) views. The nnU-Net for A2C and A4C views segments the left ventricle (LV), left atrium (LA) and myocardium (MYO). The nnU-Net for ALAX

Table 6: Characteristics of the nnU-Net [30,31] used. The 'number of channels' row indicates the number of channels at the first, bottom, and last convolution of the U-Net respectively. The number of output channels reflects the number of structures the model segments. For A2C/A4C views, the nnU-Net segments the LV, MYO, and LA. For ALAX views, the nnU-Net additionally segments the AO.

Input size

256 × 256

Number of channels

32 ↓ 512 ↑ 32

Number of output channels

3 for A2C/A4C views, 4 for ALAX views

Lowest resolution

4 × 4

Upsampling scheme

Deconvolutions

Normalization scheme

InstanceNorm

Batch Size

49

Optimizer

Adam

Initial learning Rate

1e-2

Scheduler

Polynomial

Loss function

Dice &amp; cross-entropy

Inter-layer activation

Leaky Relu

Final layer Activation

Softmax

Epochs

500

Augmentations

Rotations, scaling, Gaussian noise, Gaussian blur, brightness, contrast, simulation of low resolution, gamma correction and mirroring. For more details, see [31].

views additionally segments the aorta (AO).

The segmentation of the MYO is divided into eight regions using the following algorithm:

- 1. Extract the annulus points. For A2C/A4C views, these are the points where the MYO meets the LA. For ALAX views, these are the points where the MYO meets the LA and AO. Points A and B are the annulus points in Fig. 9.
- 2. Extract the apex of the LV, defined as the furthest points from the base points within the lv lumen. This is point C in Fig. 9.
- 3. Divide both the left and right part of the endocardium border, defined as the border between the LV and MYO regions, into three parts with equal length. This gives points D, E, F and G in Fig. 9.
- 4. Find the closest points on the outer MYO border for points C, D, E, F and G. These are points H, I, J, K and L in Fig. 9.
- 5. Fill in the regions by connecting the points via the contour, resulting in the MYO divided into six regions.
- 6. Draw circles 4 with a radius of 2 millimeters around the annulus points, points A and B in Fig. 9. We use these additional two regions to asses the local image quality of the annulus points in the image. The result are the eight regions, as in Fig. 9.
- 7. Remove any parts of regions that fall outside of the sector. The apical top regions in Fig.1a, i.e., the yellow and white masks, are examples of this. If more than 50% of all pixels inside the region fall outside the sector, we exclude the region from analysis.

The goal is to automatically quantify the image quality in each of these eight regions. The LV lumen is used as background region.

## B Ablation study of the end-to-end learning model

In the ablation study, we evaluated the impact of modifying the architecture of the end-to-end learning model. Three different network architectures were tested: Cardiac View Classification (CVC) network [3], MobileNetV2 [22], and EfficientNet [33]. We tested approaching the problem both as a classification and regression task, with only the final dense layer and loss function being changed accordingly. Additionally, three basic network attention variations were tested using the automatic segmentation output. The default model did not use any attention and predicts each label directly, as in Fig.10a. For the other two versions, the region masks extracted from the segmentation were dilated with a square dilation filter of size 50x50 pixels and used as an additional input to the networks which then predict the label of one region at a time. The dilation filter reveals the direct vicinity around each region so the boundary between tissue and background becomes visible. The first variant used this dilated mask as hard attention by blacking out the other parts of the image, as shown

4 Due to the unequal pixel spacing in depth and width, the annulus regions become ovals in the 256x256 segmentation maps. When plotting the images with equal spacing in width and depth, these regions become circles again.

Figure 9: Division of MYO into regions. A and B represent the MYO base points. C represents the LV apex. D, E, F and G are obtained by dividing the inner LV border into equal parts. H, I, J, K and L are obtained by finding the closest corresponding point on the outer MYO border.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-c29a6c2f-a523-455f-a00a-f7da74f02a81.jpg)

Table 7: Training configuration of the end-to-end learning model

Input size

256x256

Batch size

16

Optimizer

Adam

Initial learning rate

1e-4

Scheduler

None

Loss function regression

Mean squared error (MSE)

Loss function classification

Squared earth mover's distance [35]

Epochs

500

Augmentations

Rotations ( - 30 ◦ ≤ angle ≤ 30 ◦ ), horizontal mirroring (also mir- ror labels), gamma correction (0 9 . ≤ γ ≤ 1 1) . and scaling (0 85 . ≤ magnification ≤ 1 15). . Each augmentation is applied individually with 0 5 probability. .

in Fig. 10b. For the second variant, the masks were used as soft attention as input to a side branch of the network, as proposed by Eppel [34]. For this version, we created an attention map and added it element-wise to the output of the first layer, corresponding to version 'c' in Eppel [34]. Fig. 10c shows this configuration.

The ablation study consists of two parts. Both parts used the training configuration listed in Table 7 and data from the regional image quality dataset. In the first part, we examined the effect of changing the convolutional backbone and the effect of framing the problem as a classification or regression task. Table 8 compares the predictions with the annotations on the test set for the different configurations using the default end-to-end model. In the second part of the ablation study, the backbone was fixed to MobileNetV2 [22] and the problem was framed as a regression task. Next, the different variants shown in Fig. 10 were compared. Table 9 summarizes the results on the test set. The default end-to-end model with no attention, the MobileNetV2 [22] architecture, and the problem framed as a regression task gave the best results and were thus used for this paper .

Table 8: Results on the test set for the first part of the end-to-end learning ablation study. The Cardiac View Classification (CVC) [3] network is trained from scratch. The other networks are pretrained on ImageNet [36]. Each method is compared against annotations from the test set of the regional image quality dataset.

| Task           | Model                                                                                   | Spearman correlation     | Mean absolute error (MAE)                                   | Number of parame- ters in millions   |
|----------------|-----------------------------------------------------------------------------------------|--------------------------|-------------------------------------------------------------|--------------------------------------|
| Regression     | CVC [3] MobileNetV2 [22] EfficientNet-B0 [33] EfficientNet-B4 [33] EfficientNet-B7 [33] | 0.62 0.65 0.63 0.63 0.63 | 0.60 ± 0.45 0.59 ± 0.45 0.61 ± 0.45 0.60 ± 0.44 0.59 ± 0.46 | 9.6 2.2 4.0 18 64                    |
| Classification | CVC [3] MobileNetV2 [22] EfficientNet-B0 [33] EfficientNet-B4 [33] EfficientNet-B7 [33] | 0.38 0.48 0.46 0.46 0.37 | 0.91 ± 0.81 0.80 ± 0.71 0.80 ± 0.76 0.83 ± 0.73 0.91 ± 0.79 | 9.7 2.3 4.1 18 64                    |

Figure 10: Different variants of the end-to-end learning model. The default model outputs a list of all quality scores at once, while the other two variants output only a single quality score. The end-to-end soft attention model corresponds to version 'c' in Eppel [34].

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00591v4-figure-de81c249-4cde-4b75-91e7-a5b095aa7494.jpg)

Table 9: Results on the test set for the second part of the end-to-end learning ablation study using MobileNetV2.

| Model variant              |   Spearman correlation | Mean absolute error (MAE)   |
|----------------------------|------------------------|-----------------------------|
| Default (no attention)     |                   0.65 | 0.59 ± 0.45                 |
| Variant 1 (hard attention) |                   0.47 | 0.68 ± 0.60                 |
| Variant 2 (soft attention) |                   0.53 | 0.67 ± 0.57                 |
Gilles Van De Vyver Svein-Erik Måsøy Håvard Dalen Bjørnar Leangen Grenne Espen Holte Sindre Hellum Olaisen John Nyberg Andreas Østvik Lasse Løvstakken Erik Smistad
Regional quality estimation for echocardiography using deep learning
Automatic estimation of cardiac ultrasound image quality can be beneficial
for guiding operators and ensuring the accuracy of clinical measurements.
Previous work often fails to distinguish the view correctness of the
echocardiogram from the image quality. Additionally, previous studies only
provide a global image quality value, which limits their practical utility. In
this work, we developed and compared three methods to estimate image quality:
1) classic pixel-based metrics like the generalized contrast-to-noise ratio
(gCNR) on myocardial segments as region of interest and left ventricle lumen as
background, obtained using a U-Net segmentation 2) local image coherence
derived from a U-Net model that predicts coherence from B-Mode images 3) a deep
convolutional network that predicts the quality of each region directly in an
end-to-end fashion. We evaluate each method against manual regional image
quality annotations by three experienced cardiologists. The results indicate
poor performance of the gCNR metric, with Spearman correlation to the
annotations of rho = 0.24. The end-to-end learning model obtains the best
result, rho = 0.69, comparable to the inter-observer correlation, rho = 0.63.
Finally, the coherence-based method, with rho = 0.58, outperformed the
classical metrics and is more generic than the end-to-end approach. The image
quality prediction tool is available as an open source Python library at
https://github.com/GillesVanDeVyver/arqee.