Federated Learning Hyper-Parameter Tuning from a System Perspective

Federated learning (FL) is a distributed model training paradigm that preserves clients' data privacy. It has gained tremendous attention from both academia and industry. FL hyper-parameters (e.g., the number of selected clients and the number of training passes) significantly affect the training overhead in terms of computation time, transmission time, computation load, and transmission load. However, the current practice of manually selecting FL hyper-parameters imposes a heavy burden on FL practitioners because applications have different training preferences. In this paper, we propose FedTune, an automatic FL hyper-parameter tuning algorithm tailored to applications' diverse system requirements in FL training. FedTune iteratively adjusts FL hyper-parameters during FL training and can be easily integrated into existing FL systems. Through extensive evaluations of FedTune for diverse applications and FL aggregation algorithms, we show that FedTune is lightweight and effective, achieving 8.48%-26.75% system overhead reduction compared to using fixed FL hyper-parameters. This paper assists FL practitioners in designing high-performance FL training solutions. The source code of FedTune is available at https://github.com/DataSysTech/FedTune.

INTRODUCTION
F EDERATED Learning (FL) has been applied to a wide range of applications such as mobile keyboard [1], speech recognition [2], and human stroke prevention [3] on top of mobile devices and Internet of Things (IoT) [4], [5]. Compared to other model training paradigms (e.g., centralized machine learning [6], conventional distributed machine learning [7]), FL has unique properties with regards to (1) Massively Distributed: the number of clients is much larger than the clients' average number of data points; (2) Unbalanced Data: clients have a different amount of data points; and (3) Non-IID Data: each client's data may not represent the overall distribution [8]. In addition to the common hyper-parameters of model training such as learning rates, optimizers, and mini-batch sizes, FL has unique hyper-parameters, including aggregation methods and participant selection [9], [10]. Nonetheless, many FL algorithms, e.g., FedAvg [8], have been proved to converge to the global optimum even different FL hyper-parameters are adopted [11] [12].
Although FL hyper-parameters do not affect FL convergence (i.e., the same final global model accuracy), they significantly affect the training overheads of reaching the final model. Specifically, Computation Time (CompT), Transmission Time (TransT), Computation Load (CompL), and Pengfei Hu is the corresponding author. Transmission Load (TransL) are the four most crucial system overhead: • Computation Time (CompT). It measures how long an FL system spends in model training. Overall model training time must be short for applications that require fast adaptation to environments (e.g., security problems).
• Transmission Time (TransT). It represents how long an FL system spends in model parameter transmission between clients and servers. Without a high-speed transmission solution, transmission time can overwhelm the overall FL training since models are generally large and multiple rounds of model parameter transmissions are required.
• Computation Load (CompL). It is the number of Floating-Point Operation (FLOP) that an FL system consumes. For low-profile devices such as IoT nodes, the computation load must be small as these devices arXiv:2211.13656v1 [cs.LG] 24 Nov 2022 are equipped with low computational resources.
• Transmission Load (TransL). It is the total data size transmitted between the clients and the server. This is critical when data transmission is expensive or its power consumption is a concern. For example, outdoor applications usually rely on cellular communications, which need to pay a considerable price for transmitting a large amount of data. Fig. 1 summarizes the different application scenarios faced by FL practitioners in determining FL hyperparameters. Application scenarios have different training preferences in terms of CompT, TransT, CompL, and TransL. Consider the following examples: (1) attack and anomaly detection in computer networks [13] is time-sensitive (CompT and TransT) as it needs to adapt to malicious traffic rapidly; (2) smart home control systems for indoor environment automation [14], e.g., heating, ventilation, and air conditioning (HVAC), are sensitive to computation (CompT and CompL) because sensor devices are limited in computation capabilities; (3) traffic monitoring systems for vehicles [15] are communication-sensitive (TransT and TransL) because cellular communications are usually adopted to provide city-scale connectivity; (4) precision agriculture based on IoT sensing [16] is not time-urgent but requires energyefficient solutions (CompL and TransL); (5) healthcare systems, e.g., fall detection for elderly people [17], require both fast response and small energy consumption (CompT, TransT, CompL, and TransL); and (6) human stampedes detection/prevention [18] require time, computation, and communication efficient systems.
A few approaches have studied FL training performance under different hyper-parameters [19]. However, they do not consider CompT, TransT, CompL, and TransL together, which is essential from a system's perspective. Besides, they almost all use the number of training rounds for comparison, i.e., round-to-accuracy [12], [20] and convergence rate [8], [11]. The corresponding observations cannot be directly applied to guide FL system design in our context. This is because clients in FL are heterogeneous in terms of their different amounts of local training data (i.e., unbalanced data), computation speeds, and transmission rates, and thus each training round has a different time length, computation cost, and transmission cost. For example, selecting more clients in each training round results in a better round-toaccuracy performance [8], [11]. But it does not necessarily mean a better time-efficiency as the time length of each training round increases with the number of selected clients, nor a better transmission-efficiency as more clients need to transmit model parameters in each training round. In addition, it is challenging to tune multiple hyper-parameters in order to achieve diverse training preferences, especially when we need to optimize multiple system aspects. For example, it is unclear how to select hyper-parameters to build an FL training solution that is both CompT and TransL-efficient. This paper targets a new research problem of optimizing the hyper-parameters for FL from a system perspective. To do so, we formulate the system overheads of FL training and conduct extensive measurements to understand FL training performance. Our measurement results illustrate how to Work Description Single trial System FTS [21] optimize client models DP-FTS-DE [22] trade-off privacy and utility Auto-FedRL [23] improve model accuracy [24] improve training robustness FedEx [25] NAS based framework FLoRA [26] NAS based framework FedTune (Ours) a lightweight framework This paper is organized as follows. First, we introduce related work in Section 2. Then, we clarify the problem and provide intuitive explanations of our preliminary measurement results in Section 3. The details of FedTune algorithm is given in Section 4. Last, we discuss the limitations/opportunities in Section 6 and conclude this paper in Section 7.

RELATED WORK
Hyper-Parameter Optimization (HPO) is a field that has been extensively studied [27]. Many classical HPO algorithms, e.g., Bayesian Optimization (BO) [28], successive halving [29], and hyperband [30], are designed to optimize hyper-parameters of machine learning models. However, they cannot be directly applied to FL because FL has unique hyper-parameters such as the client-side and server-side aggregation methods, and FL is a different training paradigm from centralized machine learning and conventional distributed machine learning. Designing HPO methods for FL is a new research area. Only a few approaches have touched FL HPO problems. Table 1 lists several representative approaches, where we highlight whether the work can execute in an online and single trial manner and whether the work targets system overhead in FL training. For example, BO has been integrated with FL to improve different client models [21] and strength client privacy [22]. Several approaches apply reinforcement learning to adjust FL hyper-parameters [23], [24], which introduces extra complexity and loss of generality. FedEx is a general framework to optimize the round-to-accuracy of FL by exploiting the Neural Architecture Search (NAS) techniques of weight-sharing, which improves the baseline by several percentage points [25]; FLoRA determines the global hyper-parameters by selecting the hyper-parameters that have good performances in local clients [26]. Recently, a benchmark suite for federated hyper-parameter optimization [31] is designed, whose effectiveness remains to be investigated.
For two reasons, existing approaches cannot be directly applied to our scenario of optimizing FL hyper-parameter for different FL training preferences. First, CompT (in seconds), TransL (in seconds), CompL (in FLOPs), and TransL (in bytes) are not comparable with each other. Incorporating training preferences in HPO is not trivial. Second, hyperparameter tuning needs to be done during the FL training. No "comeback" is allowed as the FL model keeps training until its final model accuracy. Otherwise, it will cause significantly more system overhead.

UNDERSTANDING THE PROBLEM
We first quantify the system overhead of FedAvg to illustrate the problem. FedAvg minimizes the following objective (1) where f i (w) is the loss of the model on data point (x i , y i ), that is, f i (w) = (x i , y i ; w), K is the total number of clients, P k is the set of indexes of data points on client k, with n k = |P k |, and n is the total number of data points from all clients, i.e., n = K k=1 n k . Due to the large number of clients in a typical FL application (e.g., millions of clients in the Google Gboard project [1]), a common practice is to  randomly select a small fraction of clients in each training round. In the rest of this paper, we refer to the selected clients as participants and denote M as the number of participants in each training round. Each participant makes E training passes over its local data in each round before uploading its model parameters to the server for aggregation. Afterward, participants wait to receive an updated global model from the server, and a new training round starts.

System Model
Assume that clients are homogeneous regarding hardware (e.g., CPU/GPU) and network (e.g., transmission speeds). Let b k,r indicate whether client k participates at the training round r. Then, we have K k=1 b k,r = M , i.e., each round selects M participants. The number of training rounds to reach the final model accuracy is denoted by R, which is unknown a priori and varies when different sets of FL hyper-parameters are used in FL training. CompT, TransT, CompL, and TransL can be formulated as follows.
• Computation Time (CompT). If client k is selected in a training round, its training time can be represented by C 1 ·E·n k , where C 1 is a constant. It is proportional to its number of data points (i.e., n k ) because n k decides the number of local updates (number of mini-batches) for one epoch, and each local update includes one forward-pass and one backward-pass. The computation time of the training round r is determined by the slowest participant and thus is represented by C 1 · E · max K k=1 b k,r · n k . In total, the computation time of an FL model training can be formulated as • Transmission Time (TransT). Each participant in a training round needs one download and one upload of model parameters from and to the server [19]. Thus, the transmission time is the same for all participants in any training round, i.e., a constant C 2 . The total transmission time is represented by  • Computation Load (CompL). Client k has C 3 · E · n k computation load if it is selected in a training round, where C 3 is a constant. The computation load of the training round r is the summation of each participant's computation load and thus is C 3 ·E· K k=1 b k,r · n k . We can formulate the overall computation load as • Transmission Load (TransL). Since each training round selects M participants, the transmission load for a training round is C 4 · M where C 4 is a constant. The total number of training rounds is R, and thus, the total transmission load of an FL training is represented by As we assume that clients have homogeneous hardware and network, clients have the same C 1 , C 2 , C 3 , and C 4 . Thus, these constants do not affect the comparison of training overhead under different hyper-parameters when the same model is used. In the experiments, we assign the model's number of FLOPs for one input to C 1 and C 3 , and the model's number of parameters to C 2 and C 4 .

Measurement Setup
We conduct measurements to study the system overheads when different FL hyper-parameters are used for training. The measurements aim to help us better understand FL training and are the basis of our automatic tuning algorithm. We use the Google speech-to-command dataset [32], which classifies one second's audio clip into 35 commands such as yes, no, right, and up. The dataset includes audio clips that are crowd-sourced from 2618 clients. As officially suggested in [32], we use 2112 clients' data for training and the remaining 506 clients' data for testing. Fig. 2(a) shows the distribution of the number of clients versus the number of data points. It is clear that clients' data are heterogeneous: many clients have only one data point, while others can have up to 316 data points. Fig. 2(b) plots the histogram of each class's number of data points, which shows that the overall data distribution is unbalanced. The speech-tocommand dataset demonstrates the three data properties of FL: massively distributed, unbalanced, and non-IID. In the measurement study, we investigate the FL training overhead in terms of the following three hyper-parameters.

•
The number of participants (i.e., M ). It is well-known that more participants in each training round have a better round-to-accuracy performance [8]. However, the time efficiency is not necessarily better because the time length of each training round also increases with more participants. Besides, it is unclear how computation and communication overheads behaves for different number of participants. In the measurement study, we set M to 1, 10, 20, and 50.

•
The number of training passes (i.e., E). Increasing the number of training passes as a method to improve communication efficiency has been adopted in several works, such as FedAvg [8] and FedNova [11]. However, how does the number of training passes affect the time overhead and computation overhead is unclear. In the measurement study, we set E to 0.5, 1, 2, 4, 8, where 0.5 means that only half of each client's local data are used for local training in each round.
• Model complexity. We also investigate how the model complexity influences the training overheads if a target accuracy is met. Although it is common knowledge that smaller models have better time and computation performance in other paradigms of model training, we are the first to report the four system overhead versus model complexity in the FL setting. We use ResNet [33] to build different models, as listed in Table 2.

Illustration of FL Training
To gain an intuitive understanding of FL training, we plot FL training profiles in Fig. 3 where a different number of participants M is used on each round. We use ResNet-18 and set the target model accuracy to 0.8. In this measurement, E, C 1 , C 2 , C 3 and C 4 are all set to 1, for illustration purpose. Results are normalized to the largest overhead. As expected, more participants lead to a better accuracyto-round performance ( Fig. 3(a)). However, if the FL training is stopped too early, it is misleading to conclude that more participants result in higher final model accuracy. Instead, they all reach the same model accuracy as FL hyperparameters do not invalidate the convergence. Correspondingly, more participants have a better accuracy-to-CompT performance, as shown in Fig. 3(b). However, the performance gap between a different number of participants for the accuracy-to-CompT is less significant than the accuracyto-round performance. This is because the time duration of each training round increases when more participants are selected on each training round (Fig. 3(c)). In this measurement, there are no obvious accuracy-to-CompT difference when 20 and 50 participants are selected. On the other hand, fewer participants lead to better accuracy-to-CompL performance, as shown in Fig. 3(d), which means the effectiveness of each unit computation operation is higher when fewer participants are involved in the FL training. The accuracyto-TransT ( Fig. 3(e)) is the same as the accuracy-to-round ( Fig. 3(a)) since we adopt C 3 = 1 for plotting. That is, more participants have a better accuracy-to-TransT performance. Fig. 3(f) shows that the accuracy-to-TransL performance has the opposite trend, i.e., more participants result in a worse performance.

Measurement Results
In Section 3.3, we illustrate CompT, CompL, TransT, and TransL performance when a different number of partic- Training aspect M E Model complexity ipants are adopted in FL training. This section presents results from more extensive measurements for the number of participants M , the number of training passes E, and the model complexity. The results are averaged over three runs. In the experiments, we use ResNet-18 and normalize their overheads. As we can see, more participants lead to smaller CompT, i.e.; it takes a shorter time to converge. However, the difference is insignificant among 10, 20, and 50 participants, especially when the number of training passes is large. In addition, we can see that larger E has worse CompT. There is no apparent difference between E = 0.5 and E = 1 though. In a nutshell, the common knowledge of more participants is faster for FL model training is valid. However, the gain of more participants is insignificant when the number of participants is moderate. In addition, it is preferred to adopt a small number of training passes to achieve good time efficiency.
• Transmission Time (TransT). Fig. 4(b) plots TransT, which clearly shows that TransT favors larger M and E. Since TransT is dependent on the number of training rounds R (Eq. (3)), it is equivalent to the metric of round-to-accuracy. Our measurement result is consistent with common knowledge (e.g., [12]) that more participants and more training passes have a better round-to-accuracy performance. We can also observe that when M is small, e.g., 1, TransL is much worse than the other cases.
• Computation Load (CompL). Fig. 4(c) shows CompL. We make the following observations: (1) More participants result in worse CompL. The results indicate that the gain of faster model convergence from more participants does not compensate for the higher computation costs introduced by more participants. (2) CompL is increased when more training passes are used. This is probably because that larger E diverges the model training [20], and thus, the data utility per unit of computation cost is reduced.
• Transmission Load (TransL). Fig. 4 Table 2 tabulates the models for comparing training overheads versus model complexity. In this experiment, we select one participant (M = 1) to train one pass (E = 1) on each training round. Fig. 5 shows the normalized CompT, TransT, CompL, and TransL for different models. The xaxis is the target model accuracy, and the y-axis is the corresponding overhead to reach that model accuracy. Since only one client and one training pass are used on each round, CompT and CompL have the same normalized comparison, and so are TransT and TransL. The results show that smaller models are better in terms of all training aspects. In addition, it is interesting to note that heavier models have higher increase rates of overheads versus model accuracy. This means that model selection is especially essential for high model accuracy applications.

Summary of System Overhead
Based on our measurement study, we summarize systems overheads versus FL hyper-parameters in Table 3. As we can see, CompT, TransT, CompL, and TransL conflict with each other in selecting the optimal M and E. Regarding model complexity, smaller models have better system overheads if the model accuracy is satisfied. Please note that Table  3 is consistent with existing work (e.g., [12]), but is more comprehensive. Thus, Table 3 is also valid for other datasets and ML models.

Intuitive Explanation of Measurement Results
We present intuitive explanation of the measurement results to help FL practitioners better understand FL hyperparameters. Fig. 6 visualizes the intuition.

•
The number of participants M . In Fig. 6(a), the top scheme and the bottom scheme have the same computation cost (two local training) and communication cost (four transmissions). However, the bottom scheme has a better overall CompL and TransL at the expense of degraded CompT and TransT. The bottom scheme is better probably because the clients in the bottom scheme always work on the updated global model, whereas the clients in the top scheme work on the same global model. In other words, narrowand-deep FL schemes have better computation load and transmission load than wide-and-shallow FL schemes.
• The number of training passes E. In Fig. 6(b), both the top scheme and the bottom scheme take E total training passes. However, the bottom scheme has better CompT and CompL at the expense of higher communication costs. The results indicate that the usefulness per local update decreases with the number of local updates. Therefore, for computationsensitive FL applications, large E should be avoided.
• Model complexity. Fig. 6(c) shows that a smaller model has better CompT, TransT, CompL, and TransL than a heavier model, as long as the accuracy requirement is met. In addition, our results in Fig. 5 indicate that if the target model accuracy is low, then using a heavy model does not introduce significantly more overhead than a lightweight model. However, if the goal is to achieve a high-accuracy model, carefully selecting a model complexity is essential, as overlarge models cause significantly more system overhead.

FEDTUNE: AUTOMATIC TUNING OF FEDERATED LEARNING HYPER-PARAMETERS FROM A SYSTEM PERSPECTIVE
Due to the conflicting FL hyper-parameters for CompT, TransT, CompL, and TransL, practitioners bear the burden of selecting a set of FL hyper-parameters, which if not chosen well may lead to poor system performance. For example, it is unclear how to set FL hyper-parameters that are both CompT and TransT friendly, since these two system aspects prefer different number of training passes. We propose FedTune to adjust FL hyper-parameters during the FL training automatically. FedTune considers the application's preference for CompT, TransT, CompL, and TransL, denoted by α, β, γ, and δ, respectively. We have α+β +γ +δ = 1. For example, α = 0.6, β = 0.2, γ = 0.1, and δ = 0.1 represent that the application cares most about CompT, somewhat about TransT, and the least on CompL and TransL.

Problem Formulation
For two sets of FL hyper-parameters S 1 and S 2 , FedTune defines the comparison function I(S 1 , S 2 ) as where t 1 and t 2 are CompT for S 1 and S 2 when achieving the same model accuracy. Correspondingly, q 1 and q 2 are TransT, z 1 and z 2 are CompL, and v 1 and v 2 are TransL. If I(S 1 , S 2 ) < 0, then S 2 is better than S 1 . A set of hyper-parameters is better than another set if the weighted improvement of some training aspects (e.g., CompT and CompL) is higher than the weighted degradation, if any, of the remaining training aspects (e.g., TransT and TransL). The weights are training preferences on CompT, TransT, CompL, and TransL. However, the training overhead for different sets of FL hyper-parameters are unknown a priori. As a result, directly identifying the optimal hyper-parameters before FL training is impossible. Instead, we propose an iterative method to optimize the next set of hyper-parameters. Given the current set of hyper-parameters S cur , the goal is to find a set of hyper-parameters S nxt that improves the training performance the most, that is, minimizes the following objective function: where t cur , q cur , z cur , and v cur are CompT, TransT, CompL, and TransL under the current hyper-parameters S cur ; t nxt , q nxt , z nxt , and v nxt are CompT, TransT, CompL, and TransL for the next hyper-parameters S nxt . We focus on the number of participants M and the number of training passes E, since training overhead is monotonous with model complexity. Therefore, we need to optimize S nxt = {M nxt , E nxt }.

S nxt Optimization
To find the optimal S nxt , we take the derivatives of G(S nxt ) over M and E, obtaining Algorithm 1: FedTune update.  We illustrate how to approximate ∆M . The process for ∆E is similar. Considering that each step makes a small adjustment of M , ∂t nxt /∂M can be represented by (+1)×|t nxt −t cur |, where (+1) means CompT prefers larger M according to Table 3. To estimate |t nxt − t cur |, we apply a linear function η t−1 ×|t cur −t prv | where η t−1 = |tcur−tprv| |tprv−tprvprv| (t prvprv is the CompT at two steps before). Thus, η t−1 represents the slope of the linear function. Similarly, we have η q−1 , η z−1 , η v−1 for TransT, CompL, and TransL when calculating their derivatives over M . As a result, ∆M can  be approximated as Similarly, we can calculate ∆E as where ζ t−1 , ζ q−1 , ζ z−1 , and ζ v−1 are the parameters for calculating the derivatives of CompT, TransT, CompL, and TransL over E. FedTune automatically updates η t−1 , η q−1 , η z−1 , η v−1 , ζ t−1 , ζ q−1 , ζ z−1 , and ζ v−1 during FL training. At each step, FedTune updates the parameters that favor the current decision. For example, if M cur is larger than M prv , FedTune updates η t−1 and η q−1 as CompT and TransT prefer larger M (line 17); otherwise, FedTune updates η z−1 and η v−1 .

Decision Making and Parameter Update
Furthermore, FedTune incorporates a penalty mechanism to mitigate bad decisions. Given the previous hyperparameters S prv and the current hyper-parameters S cur , FedTune calculates the comparison function I(S prv , S cur ) (line 15). A bad decision occurs if the sign of I(S prv , S cur ) is positive (line 18). In this case, FedTune multiplies the parameters that are against the current decision by a constant penalty factor, denoted by D (D ≥ 1). For example, if I(S prv , S cur ) > 0 and M cur > M prv , FedTune updates η t−1 and η q−1 as explained before, but also multiplies η z−1 and η v−1 by D (line 19-20).

EVALUATION
In this section, we provide the evaluation results of FedTune. First, we explain our experiment setup in Section 5.1. Next, we present the overall performance in Section 5.2. Then, we conduct trace analysis in Section 5.3. Last, the penalty mechanism is studied in Section 5.4.

Experiment Setup
Benchmarks and Baseline. We evaluate FedTune on three datasets: speech-to-command [32], EMNIST [34], and Cifar-100 [35], and three aggregation methods: FedAvg [8], Fed-Nova [12], and FedAdagrad [36]. We set equal values for the combination of training preferences α, β, γ and δ (see the first column in Table 4). Therefore, for each dataset, we conduct 15 combinations of training preferences. We set target model accuracy for each dataset and measure CompT, TransT, CompL, and TransL for reaching the target model accuracy. We regard the practice of using fixed M and E as the baseline and compare FedTune to the baseline by calculating Eq. (6). In the evaluation, the positive performance means FedTune reduces the system overheads and the negative performance means the degradation. We implemented FedTune in PyTorch. All the experiments are conducted in a server with 24-GB Nvidia RTX A5000 GPUs.

Training Setup.
(1) speech-to-command dataset. It classifies audio clips to 35 commands (e.g., 'yes', 'off'). We transform audio clips to 64-by-64 spectrograms and then downsize them to 32-by-32 gray-scale images. As officially suggested [32], we use 2112 clients' data for training and the remaining 506 clients' data for testing. We set the minibatch size to 5, considering that many clients have few data points. We use ResNet-10 and the target model accuracy of 0.8. (2) EMNIST dataset. It classifies handwriting (28-by-28 gray-scale images) into 62 digits and letters (lowercase and uppercase). We split the dataset based on the writer ID. We randomly select 70% writers' data for training and the remaining for testing. We use a Multiplayer Perception (MLP) model with one hidden layer (200 neurons with ReLu activation). We set the mini-batch size to 10 and the target model accuracy of 0.7. (3) Cifar-100 dataset. It classifies 32by-32 RGB images to 100 classes. We randomly split the dataset into 1200 users, where each user has 50 data points. Then, we randomly select 1000 users for training and the remaining 200 users for testing. We set the mini-batch size to 10. ResNet-18 is used, and the target model accuracy is set to 0.2 (due to our limited computational capability, we set a low threshold for Cifar-100).
For all datasets, we normalize the input images with the mean and the standard deviation of the training data before feeding them to models for training and testing. Both M and E are initially set to 20. FedTune is activated when the model accuracy is increased by at least 0.01 (i.e., = 0.01). The penalty factor D is set to 10

Overall Performance
Results for Diverse Datasets. Table 5 shows the overall performance of FedTune for different datasets when FedAvg is applied. We set the learning rate to 0.01 for the speech-tocommand dataset and the EMNIST dataset, and 0.1 for the Cifar-100 dataset, all with the momentum of 0.9. We show the standard deviation in parenthesis. As shown, FedTune consistently improves the system performance across all the three datasets. In particular, FedTune reduces 22.48% system overhead of the speech-to-command dataset compared to the baseline by averaging the 15 combinations of training preferences. We also observe that the FL training benefits more from FedTune if the training process needs more training rounds to converge. Our experiments with EMNIST (small model) and Cifar100 (low target accuracy) only require a few dozens of training rounds to reach their target model accuracy, and thus their performance gains from FedTune are not significant. The observation is consistent with the decision-making process in FedTune, which increases/decreases hyper-parameters by only one at each step. We leave it as future work to augment FedTune to change hyper-parameters with adaptive degrees.
Results for Different Aggregation Methods. Table 6 shows the overall performance of FedTune for different aggregation methods when we use the speech-to-command dataset and the ResNet-10 model. We set the learning rate to 0.1, β 1 to 0, and τ to 1e-3 in FedAdagrad. As shown, FedTune achieves consistent performance gain for diverse aggregation methods. In particular, FedTune reduces the system overhead of FedAdagrad by 26.75%.

Trace Analysis
We present the details of traces when the speech-tocommand dataset and the FedAdagrad aggregation method are used. Table 4 tabulates the results, where we show the application preference (α, β, γ, and δ), the system overheads (CompT, TransT, CompL, and TransL), the final M and E when the training is finished, and the overall performance. We report the average performance, as well as their standard deviations in parentheses. The first row is the baseline, which does not change hyper-parameters during the FL training. As we can see from Table 4, FedTune can adapt to different training preferences. Specifically, FedTune reduces the system overhead up to 70.51% when the application only cares about computation load (i.e., γ = 1). Only one preference (0.33, 0.33, 0, 0.33) results in a slightly degraded performance. On average, FedTune improves the overall performance by 26.75% for the speech-to-command dataset and the FedAdagrad aggregation method. Fig. 7 illustrates one trace of M and E during the FL training for each application preference. The experiment is conducted with the speech-to-command dataset and the FedAdagrad aggregation method. The experiment result clearly shows that FedTune can automatically adjust FL hyper-parameters during the FL training while respecting the application's preference on system overhead. We also observe that FedTune does not keep increasing or decreasing M and E. Instead, in many cases, it intelligently tunes the M and E during the different phases of the FL training.

Study of Penalty Mechanism
We investigate our penalty mechanism by conducting experiments with the Google speech-to-command dataset and Fig. 9. Performance comparison of FedTune without penalty mechanism and with penalty mechanism for the Speech-to-command dataset and FedAvg aggregation method.
First, we conduct experiments to explore whether the penalty mechanism can mitigate the degradation. Results are averaged over three runs. Fig. 8 plots the performance of FedTune for the degraded cases versus penalty factors, where error bars represent the standard deviation. Although the penalty mechanism does not guarantee positive performance for the degraded cases, it mitigates the degradation. Fig. 8 also shows that FedTune remains stable for a moderate value of penalty factors. Empirically, we set the penalty factor to 10 in FedTune. Fig. 9 compares the performance of FedTune when the penalty factor is 10 (full-fledged) versus 1 (no penalty) for different training preferences. Results are averaged by three runs and the error bars represent the standard deviation. Overall, FedTune increases the performance gain of the FedAvg aggregation method from 17.97% to 22.48%. In addition, FedTune with the penalty mitigates the highly degraded cases and thus is more practical than the nonpenalty counterpart. We also observe that FedTune with penalty mechanism is more stable, with the averaged standard deviation of 7.77% versus 14.14% in the non-penalty counterpart.

DISCUSSION
FedTune has promising performance in tuning FL hyperparameters. As one of the first work of its kind, FedTune has some limitations/opportunities that deserve further exploration.
Heterogeneous Devices. This paper assumes that clients are homogeneous, i.e., the same hardware and network. In practice, however, client devices are heterogeneous. Measurements of the computation capabilities of mobile devices and their network throughput exhibit order-of-magnitude difference [37], [38], [39]. As a result, for clients even with the same amount of local data points, they result in different computation and transmission costs. Currently, FedTune only tunes the system-wide hyper-parameters, which might not be optimal for heterogeneous devices that may require device-level hyper-parameter tuning. However, compared to the overwhelming and laboring effort of manually determining the FL hyper-parameters for each device, tuning system-wide hyper-parameter seems to be a more practical approach for FL practitioners. We leave it as future work to evaluate FedTune on heterogeneous devices.
Extensions. Many FL algorithms have been proposed to tackle the limitations of FedAvg. FedTune could be extended to the following scenarios. (1) Participant selection. Compared to the random selection of participants, guided participant selection that considers clients' data utility and device utility can improve overall training performance [9]. Popular alternatives are to only wait for participants that are finished before a deadline [40] or only wait for the first M participants [11]. (2) Adaptive training passes across participants. Due to the heterogeneity of clients, setting the same number of training passes E for all participants in each training round is not optimal. To support different E across participants, FedNova [12] relies on re-weighting of aggregation while FedProx [20] adds a proximal term to stabilize the convergence. We plan to incorporate these functionality to further improve the performance of Fed-Tune. (3) Investigate more FL hyper-parameters. Although we only take the number of participants, the number of training passes, and the model complexity as examples to illustrate the workflow of FedTune, we believe FedTune is a general framework, which can be applied to other FL hyperparameters, e.g., the mini-batch size.

CONCLUSION
FL involves high system overhead in the training process, which hinders its research and real-world deployment. We argue that optimizing system overhead for FL applications is extremely valuable. To this end, we propose FedTune to adjust FL hyper-parameters, catering to the application's training preferences automatically. Our evaluation results show that FedTune is general, lightweight, flexible, and is able to significantly reduce system overhead in FL training.


Involuntary breathing movement pattern recognition and classification via force based sensors

The study presents a novel scheme that recognizes and classifies different subphases within the involuntary breathing movement (IBM) phase during breath-holding (BH). We collected force data from eight recreational divers until the conventional breakpoint (CB). They were in a supine position on force plates. We segmented their data into the no-movement (NM) phase aka easy phase and IBM phase (comprising several events or sub-phases of IBM). The acceleration and jerk were estimated from the data to quantify the IBMs, and phase portraits were developed to select and extract specific features. The K means clustering was performed on these features to recognize different sub-phases within the IBM phase. We found five-six optimal clusters separating different sub-phases within the IBM phase. These clusters separating different sub-phases have physiological relevance to internal struggle and were labeled as classes for classification using support vector machine (SVM), naive bayes (NB), decision tree (DT), and K-nearest neighbor (K-NN). In comparison with no feature selection and extraction, we found that our phase portrait method of feature selection and extraction had a low computational cost and high robustness of 96–99% accuracy.

Introduction
There are involuntary physiological responses present in the biological systems that are activated when life-threatening situations arise. The prolonged contraction of the diaphragm and external intercostal muscle during BH results in one such response. It is considered that prolonged contraction and hypercap- 5 nia during BH activate chemoreceptors [1] [2]. These receptors on reaching the sensory cortex generate a dyspnea signal, which is crucial for the initiation of breathing. [3] [4].
The question is whether a prolonged contraction has movement patterns, and can sensors capture them? These questions are of importance for abnormal 10 breathing detection, and the answer is in the duration and strength of the struggle phase (SP). It is a phase in which IBMs appear. It starts with physiological break point ((PB), point where first IBM appears) and ends with CB (point where breathing is resumed voluntarily) [5]. The muscle contractions during SP are periodic and very subtle. Thus, invasive methods are used to capture them, 15 but non-invasive methods during strong contractions can be applicable [4], [5], [6]. These contractions are autonomously controlled during SP, supposedly inhibiting the stimulation of receptors formed during diaphragm contraction to extend the BH [4]. Thus, rhythmic varying patterns of IBMs associated with involuntary periodic contractions are present during extended BH. These can 20 be acquired using non-invasive methods.
Previous studies used a simple computational model to detect phases during BH (onset and end of the IBM phase) [7]. However, the IBM phase is more complex than just onset and end. IBMs in SP are rapid continuous movements similar to muscle spasms. Thus, it is hard to detect patterns or sub-phases in 25 a movement of such nature using a simple model. Previous studies had shown complex varying phases of IBMs as the amplitude and frequency increases near the CB [8], [9]. Moreover, the current literature does not present methods to recognize such sub-phases within the IBM phase. Therefore, methods emphasizing detecting patterns or sub-phases within the IBM phase are of importance in 30 2 clinical science and should be developed. We refer to the IBM phase as several events or patterns of IBM during a SP.
In this study, we present a novel scheme (shown in figure 1) that can recognize different sub-phases within the IBM phase during BH, and classify them accordingly. The force sensors were used to capture kinetics associated with 35 IBMs as it comprises mechanical components [4]. We did not use force data directly for the categorization because kinematic patterns of rapid movements can be quantified nicely through jerk and acceleration. Hence, the rapid movements related to involuntary contractions during BH was estimated using acceleration and jerk. These outputs were later used as features for pattern recognition and 40 classification of IBM sub-phases.

Materials and Methods
The IRB review board of Colorado Mesa University has approved this study.
In this study, we recruited eight healthy participants who are recreational divers.
Divers generally have the higher lung capacity and better control of their breath-45 ing due to years of intense training. This makes them a better fit for our study.
The demographic data of participants recruited in our study is listed in table 1. Units:-u years, v centimeters, w kilograms, u years.

Experimental Design
The protocol to collect data from a single participant required a single visit to our lab. On the visit, signed consent was obtained, and data regarding their 50 age, gender, height, weight, and years of experience was recorded as shown in table 1. They were also instructed to avoid eating, drinking, and exercising two hours before the data collection. Two AMTI force plates were used to acquire the data from the participants at a sampling rate of 1Khz. We asked the participant to lay in a supine position on the force plates. The participants 55 were laying with a static posture and covered the force plates surface mostly with their upper and middle back. Moreover, to maintain a comfortable posture during data collection, a cushion and a rolled-up pad were positioned under their head and knee.
We introduced a warm-up session in three steps. In the first step, partici-60 pants breathed normally (relaxation) for five minutes followed by a minute of BH. In the second step, the relaxation period was reduced to two minutes, and the BH phase was incremented by an additional sixty seconds from previous step. In the third step, participants relaxed again, followed by three minutes of BH.

65
Prior to data collection participants were given enough relaxation periods.
For data collection, they were asked to hold their breath as long as they can.
The data was recorded from the participant's first inhalation (starting BH) to voluntary exhalation (ending BH). The participants were asked to lay down after data collection until lightheadedness subsides.

Acceleration and Jerk estimation
The participants had no significant center of mass (COM ) displacement of other body segments due to their static posture. Therefore, we assumed full 90 body COM in the chest, and acceleration ( a) and jerk ( j) of the IBM phase were estimated from the processed vertical ground reaction force (F z ), as shown below.
F a is the dynamic force data vector due to the accelerating center of mass ( a COM ), W is the weight of the participants' chest, m is the mass of the par-95 ticipant's chest. The participants' chest mass is calculated from the thorax percentage contribution to full-body mass [10], and g is the acceleration due to 5 gravity. The superscript displays the dimension of the data vector for a single participant in equations (1) to (4).

Phase portraits 100
A phase portrait ( P ) is a geometric representation of the dynamic trajectories in the phase plane [11]. The a and j data were used to develop phase portraits for feature selection and extraction. The phase portraits yielded phases that overlap between the IBM phase and the NM phase. Thus, developing phase portraits were crucial to remove phases and frequencies, that overlap between 105 the NM phase and IBM phase. This feature selection and extraction through phase portraits may pose useful to enhance the performance (accuracy, computation, robustness) of our pattern recognition and detection scheme.
We first separated the data points overlapping between IBM phase and NM phases by sorting the values of a and j in descending/ascending order from 110 phase portraits. These samples of a and j were then shuffled randomly and independently so that the features will represent unbiased population of the data. Furthermore, peak envelopes, lower ( a RM S(u) , j RM S(u) ) and upper envelopes ( a RM S(l) , j RM S(l) ) were extracted. These features were finally used for unsupervised and supervised learning. 115

Unsupervised learning
We performed unsupervised learning to recognize similarities and/or dissimilarities between the IBM and NM phases, and to recognize specific patterns/subphases/groups within IBM phase. These groups were labeled as different classes for classification.

120
K means cluster analysis was implemented on the extracted features ( a RM S(u) , j RM S(u) , a RM S(l) , j RM S(l) ) as an unsupervised learning algorithm. The K means clustering algorithm groups the data points within these features based on their similarity into clusters. To test the normality, we performed Kolmogorov-Smirnov test. The type (parametric or non-parametric) of test to compare the difference between the IBM phase and NM phase was based on the rejection of the null hypothesis.
We considered a significance value of p < 0.05 to reject the null hypothesis.
Violin and box plots are used to display the distribution, mean and standard 130 deviation of the data.

Supervised learning
A supervised learning approach was implemented to isolate the onset of different sub-phases of IBMs with a classifier. We developed a classification model with training data that makes no assumption about the normality of 135 data. We used SVM, NB, DT, and K-NN as classifiers.
The features that were grouped into different clusters (NM phase and different IBM sub-phases) were labeled as different classes for supervised learning.
Furthermore, these labeled features were split into a ratio of 75:25 for train- a 5000×1 RM S(l) , j 5000×1 RM S(l) ) data. The training data was used to train the classification model, and the test data was used to evaluate the accuracy of the model. We also used K-fold cross (K = 10) validation procedure to validate our training model before evaluating its accuracy with testing data.

IBM phase vs NM phase
We first tested whether the IBMs were present in the vertical ground reaction force, and were detectable through test statistics. We found vertical ground reaction force data was statistically significant (p ≤ 0.05, Wilcoxon sign rank paired test) between the IBM and NM phases. The force amplitude's median 150 value were different between the IBM phase and NM phase as shown in figure   2.
Moreover, statistically significant differences (p ≤ 0.05, Levene's test) in the variance of the ground reaction force between the NM phase and IBM phase were present for all the participants. Figure 2 shows the distribution of the 155 vertical ground reaction force data for all the participants.
The high variance during the IBM phase was due to periodic signal output of varying amplitude. Hence, the IBMs of varying magnitude were present in the force data and were detected using an appropriate statistical measure.

Phase portraits 160
The phase portraits generated from the estimated a and j were then used for feature selection and extraction. We found clear isolation between the IBM phase and NM phase from the phase portraits and their density plot, as shown in figure 3.
The data points for the NM phase were concentrated at a specific region of 165 less magnitude around the baseline signal. The data density was also high in this region as shown in figure 3. Thus, the BH data was mostly composed of baseline noise associated with the NM phase.
The IBM phase concentrically surrounds the NM phase area. These concentric circles represent the regions of low, moderate, and high magnitude of a and 170 j as shown in figure 3. These different regions within the IBM phase suggest different sub-phases or patterns within the IBM phase. In addition, the circular shape of the phase portrait suggests the presence of some rhythmic patterns of IBMs.

175
The goal of clustering is to identify sub-phases within the IBM phase. But before, we first determined the optimal number of clusters (K) that need to be defined in the K means algorithm. We found that based on total withinness, five to six clusters were an optimal choice for the goodness of clustering across most participants. 180 We then tested through clustering whether the IBM phase is a biological phenomenon comprising varying magnitude of a and j. We extracted peak minima and maxima values of the NM phase and IBM phase (sliding window = 25 samples) from phase portraits before feeding them to K means algorithm.
The cluster analysis displays clear isolation between the NM phase and IBM

Classification
Supervised learning classified these complex patterns or sub-phases with a computational model. Our aim of classification was to design a scheme that detects the clustered regions with high accuracy.
We trained different classifiers for the labeled data. The data were labeled 205 based on the clusters. We performed 10-fold cross-validation procedure to validate our classification model (shown in figure 5A). We further evaluated the model accuracy with the testing data. We found that our classification model can predict with an accuracy of 96.5-99.9%.
We also trained models without feature selection and extraction. We found 210 that the accuracy of those models was consistent with most algorithms, except DT as shown in table 2. The DT accuracy was also sensitive to the number of classes when no features were selected and extracted as shown in figure 5B.
The processing speed also slowed down for most algorithms due to the increased computation cost, as shown in table 2.

215
However, the feature selection reduced the computation cost without loss of accuracy (robustness). In addition, we also found feature selection improves DT accuracy, as shown in table 2. Therefore, our scheme provided a low computation cost with feature selection and higher accuracy with different classifiers. Thus, our model can classify or detect different sub-phases during IBM phase 220 with high accuracy using different algorithms, and low computation cost, as shown in table 2. Note:-The categorical values were used to specify processing speed rather than quantitative. It is because the quantitative values representing the processing speed vary depending on the hardware (processor). Moreover, the tuning parameters for classifiers were fixed while comparing feature selection and extraction against no-feature selection.
This removes the analysis bias.

Discussion
We developed a pattern recognition and classification scheme to detect different sub-phases of IBMs during BH. We used force plates and estimated a 225 and j to quantify the movement during contractions. Furthermore, we used phase portraits as a means to select features for our clustering and classification algorithms. We found that our designed scheme has high processing speed, robustness, and accuracy using classifiers such as SVM, NB, DT, and K-NN.
We suggest that our scheme is of significant interest to practitioners working The use of kinetic, kinematics and neural signals (EMG) is not uncommon for movement classification [12], [13], [14]. Based on the application of our study,

Physiological interpretation of clusters
We used clusters to group different phases of IBM phase. These groups displayed changes in a and j data values during a BH. We found from these 255 clusters that there are sub-phases of low, moderate, and high magnitude within the IBM phase. The low values of a and j during IBMs were clustered with NM phase in clusters 1 and 6. The low values of a and j in these clusters represented the onset of IBM. Moreover, the high and moderate data values of IBM phase grouped in clusters 2, 3, 4 and 5 has physiological relevance to extend BH.

260
Our study showed different levels of a and j clustered in groups 2, 3, 4, and 5 were correlated with force and were associated with increasing IBMs or struggle phase during BH. The struggle phase is a phase in which participants can hold their breath beyond PB before reaching CB [9]. A previous study showed an increased amplitude and frequency of pressure signals is an indicator 265 of struggle termination [7], [5], [9]. In our study, we observed similar results for most participants, where force amplitude incremented continuously until the end of the SP, as shown in figure 6B. However, in a few participants, the signal amplitude decayed before the SP ended, as shown in figure 6A. The low force signal amplitude before SP end could be due to high fatigue index and low 270 energy expenditure as the O 2 level in blood during BH was depleting [18], [19].

Accuracy, Speed, and Robustness
We used different classifiers that makes no assumption about normality, such as SVM, DT, NB, and K-NN [20], [21], [22], [23]. Our study also showed that the classification accuracy does not improve radically with feature selection and 275 extraction as most classifiers performed well with the raw data. Moreover, accuracy does not decrease with an increase in the number of classes (clusters, K = 5 to 11). This creates a question about our scheme, whether feature selection using phase portraits was necessary. The answer depends on the criteria considered for detection because we also found that the greedy algorithm such as 280 DT classification accuracy was sensitive to feature selection, extraction, and the number of classes. Therefore, if the goal is to detect different IBM sub-phases with high accuracy and ignore other aspects such as processing time and flexibility of using different algorithms, then feature selection and extraction using phase portraits are not necessary. However, ignoring these criteria will increase 285 the hardware cost as faster processors and built-in computation methods will be needed. Therefore, our detection scheme offers high accuracy, speed, and flexibility with other algorithms without loss of accuracy (robustness).
The other aspect of the classification accuracy of our scheme is parameter optimization. We have mentioned in our results that we kept parameters consis-290 tent while comparing the accuracy with and without feature selection. Hence, the DT model without features was not optimized subjectively to attain higher accuracy. The rationale was to reduce any analysis bias. However, the biased approach of optimizing the parameter for higher accuracy of the DT training model will not change the fact that there will still be a trade-off between accu-295 racy and speed. Therefore, in our study, feature selection and extraction using phase portraits are of significant importance as it reduces the constraints related to processing speed without impact accuracy [24], especially for greedy deterministic algorithms like DT.

Limitations & Future work 300
There are information-based constraints with two-dimensional features ( a, j) in this study. The lack of high-dimensional feature space limits the degree of freedom to process enough information. But additional information can be captured using multichannel sensors with a high sampling rate. The inertial measurement units (IMUs) sensors can be an alternate [15], [25]. The IMUs 305 are embedded with accelerators, gyroscopes, and magnetometers, and they can provide linear and rotational kinematic information about IBM in the vertical direction [25] , [26]. Therefore, a high-dimensional feature space can be acquired using such sensors and will be the scope of our future studies.
In addition, our study does not account for the relationship between physio-310 logical factors [27] (such as motivation level, blood lactic acid level, lung volume, partial levels of O 2 and CO 2 (P O 2 and P CO 2 ) and IBM sub-phases. Therefore, in this study, IBM's physiological role is inferred. These physiological factors are the key factors for SP characteristics determination [5], [9]. Thus, we cannot develop a concrete relationship between IBM sub-phases and SP physiological 315 characteristics. It is something we will be considering studying in the future.

Conclusion
We found in our study that IBMs during BH can be captured through force sensors. In this study, we developed a pattern recognition and classification scheme using estimated a and j from the vertical ground reaction force. The 320 a and j were used to quantify the rapid movements associated with involuntary contractions. Our scheme recognizes and classifies the IBM phase from the NM phase during BH, and also identifies and classifies different sub-phases of varying (low, moderate, and high) magnitude within the IBM phase. We also developed phase portraits using a and j to select and extract specific features 325 so that accuracy, robustness, and computational cost of our scheme can be enhanced. To the best of our knowledge, this is the first study that had developed a pattern recognition and classification scheme to detect IBMs from vertical ground reaction force.

Authors contribution
The study was conceived and designed by RES, GW, JF, NB, BA. JF did the data collection. RES designed the scheme. RES and SG did the data analysis.
RES drafted the manuscript. GW, NB, JF, and BA provided input and feedback on the study. All authors provided critical revision of the manuscript and final 335 approval of the version to be published.

Conflict of interest
The authors do not have any conflict of interest.

Ethical approval
The study was approved by the Colorado Mesa University, IRB. The par-340 ticipants volunteered for the study with a signed consent, and all applicable 14 institutional and governmental regulations concerning the ethical use of human participants were followed during the course of this study.

Data availability
The data is available from Dr.


Application of Validation Obligations to Security Concerns

Our lives become increasingly dependent on safety- and security-critical systems, so formal techniques are advocated for engineering such systems. One of such techniques is validation obligations that enable formalizing requirements early in development to ensure their correctness. Furthermore, validation obligations help hold requirements consistent in an evolving model and create assurances about the model's completeness. Although initially proposed for safety properties, this paper shows how the technique of validation obligations enables us to also reason about security concerns through an example from the medical domain.

Introduction
As software systems become more and more responsible for our daily life experiences, it is natural to discuss how to engineer them to ensure their safety and security. Both safety and security are already mature disciplines, and individual processes for dealing with the domain-specific problems are available such as attack trees [14] for security concerns or model checking [7] for safety properties. However, as pointed out in literature (e.g., see [3,11]), there are a limited amount of cross-cutting techniques available at our disposal, which are capable of being used effectively in both domains.
A validation obligation (VO) [10] is a logical formula associated with the correctness claim of a given validation property. This technique helps formalize and validate software systems, thus ensuring their overall correctness. While initially proposed for the safety domain, we believe VOs are equally beneficial for validating security concerns in software systems. The benefit of applying VOs to assert the correctness of security concerns lies within the unified approach offered by VOs for modeling all sorts of proprieties, e.g., safety, security, and functional properties. In this fashion, we do not need individual correctness assurance approaches for each set of requirements, which can cause problems while keeping the model consistent. VOs, substantiated with multiple formal techniques, offer a property-agnostic approach for checking completeness and conflict freeness of models.
This paper aims to investigate the application of VOs for the correctness assurance of security concerns. The rest of the paper is structured as follows: Section 2 discusses the Event-B method -the formal method we have used in our approach. Next, Section 3 introduces the idea of VOs. Then, we exemplify the application of VOs to security concerns in Section 4 using an example from the medical domain. Next, Section 5 compares the current approaches with VOs for formal modeling of security-critical systems. Finally, we conclude the paper in Section 6 with an outlook on future work.

The Event-B method
State-based formal methods [9] enable modeling systems with a strict formal syntax, thus allowing to establish correctness assurance with techniques like model checking and theorem proving. This establishes the model's consistency and shows that the model does not lead to a faulty state. Furthermore, statebased formal methods follow the correct-by-construction approach meaning that the model is incrementally enriched with behavior while the correctness of each step is ensured.
One of the well-known state-based formal methods is Event-B [1]. In Event-B states are made up of variables that are constrained by invariants. Events define transitions between the states. A model can be refined via the refines keyword. This means the original specification is advanced concerning variables or events. This refinement relationship, however, needs to be proven to ensure that existing model constraints are not violated in the process of refinement. This is done through proofs. contexts define the static elements of a model and are seen by machines defining the dynamic behavior.

Validation obligations
A VO is composed of a model and a validation task (VT) that must be successfully executed on the model. Once a VT is successfully executed, it establishes the presence of the associated requirement. A VT comes in different forms, e.g., (LTL) model-checking, proof obligations (POs), or even manual inspection of the state space. The input parameters of each VT depend upon the associated requirement. For example, for the LTL model checking, the VT gets an LTL formula as an input parameter. It is the judgment of the designer which VT is best suited to the cause and what are its appropriate parameters. Following is the formal definition of a VO: As a VO is assigned to a requirement, an id uniquely identfies it. The assigned VT has also an id to identify it. In the area of VOs context refers not to the context of machines as defined earlier but to the context the VO is applied in, i.e., the model we investigate. Second last is the applied technique, e.g., model checking, and last are the input parameters, e.g., the LTL formula.
The concept of VOs should be seen as a carrier technology. It is not bound to a particular formal method. In fact, VOs can be applied to all state-based formal methods alike. In this paper, however, we use Event-B only to exemplify the proposed approach. Following that, we argue that we can validate everything we explicitly model.
The power of VOs comes from three aspects.

Application of VOs to security concerns
We now show the application of VOs to establish the correctness of security concerns through the example of a hemodialysis machine model. As aforementioned, we believe that security concerns can be treated like other properties when developing models, which is an advantage as there is no need for their special treatment, thus making the overall modeling process simpler.

Illustrative example
Hemodialysis is a medical treatment that uses a device to clean the blood. The hemodialysis device transports the blood from and to the patient, filters waste and salts from the blood, and regulates the fluid level of the blood. Due to the involved complexity of the dialysis process, the resulted medical treatment is monitored by a professional caregiver for treatment compliance and desired output. Traditionally hemodialysis is performed in a standalone mode, i.e., patients come to a medical facility, get connected to the device, and let dialysis be performed. However, this monitoring is also possible via remote access but demands additional security precautions. The basic architecture of hemodialysis machines is depicted in Figure 1. The requirements specification of hemodialysis machines is discussed in detail in the work of Mashkoor [8]. We first design an abstract model and then refine it to show that VOs can be used to gain confidence in the soundness and conflict freeness of the requirements. We then extend the example a second time to show the advantages of creating views on a model. Creating a view means in our context that we keep the abstract model in the background but only focus on one detailed aspect of the model at a time. This helps better understand a model's behavior or debug it by only viewing what is necessary to satisfy a particular requirement.

Formal model
We start the modeling process with a small subset of the hemodialysis machine requirements specification. We also add some additional security concerns as follows.
-SAF1: In order to start the treatment, the parameters must be within the permissible range. -FUN1: There are three types of staff IDs: maintenance, nurses, and doctors.
-SEC1: The staff has to log in to start the treatment.
-SEC2: Only doctors and nurses are allowed to start the treatment. Figure 2a shows the base model of the system. It models the SAF1 safety requirement. In the original specification multiple parameters depending on the patients condition can be set thus ensuring a personalized treatment. SAF1 is an abstract reference to that, we enter a treatment value which serves as a token  for more complex parameters requiered in the original specification. We model SAF1 along with the SEC1 security requirement. There is an Event-B context ctx modeling the login status. The corresponding machine m1 models two events: login and startSystem. Guards, namely @grd1 and @grd2, of the startSystem event prevent the event from being fired prematurely. Figure 2b shows how the model looks after the refinement. It introduces FUN1 and SEC2 requirements into the model. We add roles as data type in the refined context ctx1 and configure the set of roles that are allowed to perform a treatment in @inv4. The login event ensures that the role logged into the system is now tracked via id. @grd3 in startSystem checks that only allowed ids are able to start the treatment. Also note that the startSystem event in m2 extends the event from m1 meaning that @grd1 and @grd2 are still present and active but are hidden form this view of the model to avoid confusion.

Ensuring soundness and conflict freeness of requirements
To each requirement, we associate a VT with which we ensure the presence of this requirement in the model. The VTs can be formulated as: -SAF1/m1/PO: treatmentParameters ∈ 1..5 -SEC1/m1/LTL:G({loggedIn = yes} ⇒ e(startSystem)) 1 PO means we have a proof obligation that treatmentParameters ∈ 1..5 is true in every state of the model, meaning the patients treatment parameters are within the allowed range. For the LTL formula G(lobally) means the condition in the brackets is always true. The term inside states that when loggedIn = yes the event startSystem is enabled. m1 is the model or in our case the machine we apply the VT to as laid out in Section 3.
The proof of the VT for SAF1 is relatively simple. We can use the Event-B tool support, i.e., the Rodin platform [2], to show that @inv1 holds in every state of the model. The proof can be discharged automatically by the tool. The VT regarding SEC1 was established using the LTL formula with the ProB [7] model checker, which we employed to do LTL model checking of the model. With both VOs discharged, we can be confident that the concerning requirements are correctly modeled. Now we tackle FUN1 and SEC2. For those requirements we write the following VTs: We demand proof that the set of roles consists only of doctors, nurses, and maintenance for FUN1. For SEC2 we demand that globally the system can only be started if the logged-in role is registered in the set of roles that are allowed to perform treatment, i.e., doctors or nurses. Again with the help of the corresponding tools, we see that both VTs on the model are executed and satisfied, thus ensuring the soundness of the model m2 regarding the requested requirements, i.e., FUN1 and SEC2. However, if we run our VTs for SEC1 and SAF1 previously established on m1 on m2 to ensure that these requirements are also present in the refinement, we spot a problem. The VT representing SEC1 fails. We can find an instance where we are logged into the system but not allowed to start the treatment. This is the case with the login of the maintenance role.
A failing VT helps us discover a flawed requirement/design in the model, provided that the task was correctly chosen and the parameter was correctly formulated. In our example the requirement encoded as the VT for SAF1 is no longer satisfiable as loggedIn = yes is no longer sufficient for starting the system. We could draw three consequences from this: 1. Requirements SEC1 and SEC2 may be contradicting each other. However, this is is not the case here. 2. We could adjust the model by removing the event guard that controls the startSystem event allowing every loggedIn person to start treatment. However, this will lead SEC2 to fail; hence the formulation of the requirement as a VO prevented us from introducing new bugs when trying to fix the existing ones. 3. Requirements SEC1 and SEC2 are ambiguous in how they are formulated and need to be clarified. In our case, SEC1 is indeed very broad. We can solve the issue by either making SEC1 more explicit by stating that the allowed staff has to be logged in or merging the requirement entirely with SEC2.
In our example, we settle for option (3) to clarify requirements: we refactor the corresponding VO, and the fixed VO passes without any new conflict.

Creating views
Different views help designers spot flaws in the model or help non-technical stakeholders better understand the model. Views are a unique form of refinement that aims not to introduce new behavior but to how the model is perceived. VOs help create views by showing their soundness and conflict freeness. However, views affect the associated VT, i.e., the model changes, and the output which decides VT's success or failure also changes accordingly. While the output of a VT is more concrete on an instantiation view, and thus, depending on the task, it can provide a concrete example of why the VT succeeds or fails. The stakeholders can then easily understand the scenario. The knowledge gained from the view can be feedback on the requirements of the model and may result in new VOs to ensure the presence of the emerging requirements.
Creating a scenario-view in Event-B is achieved by replacing deferred constructs with concrete ones, thus adapting the model for a specific scenario. Let us consider our hemodialysis example again. Figure 3 shows a created view. Instead of reusing a deferred function where we map arbitrary ids to roles as we did in INITIALISATION/@act4 of m2 we use concrete mappings as one can see in @act4 of theINITIALISATION of m2Concrete. We successfully execute the VTs and establish conflict freeness and soundness of the requirements for this view.
The state-space of the model m2Concrete is visualized in Figure 4 2 . It shows two states. One can switch between the states via logging in as a different user or looping back to them by logging in as the same user. From this view, the stakeholder may discover a flaw in the model, for example, the fact that startSystem in the upper state is a loop that always ends in the same state. The consequence of this is that after startSystem was fired by the logged-in role, the logged-in role can be changed. For further treatment, this might be unwanted as the responsible role should not be changed mid-session, nor should the maintenance be able to log in after treatment is started. From this we can formulate a new requirement, for example: -FUN2 Once the treatment is started, the logged in ID can not change.
This requirement can then be encoded into a VO and run against the original model m2. A VT for this requirement could look like this: This formula would check that after the firing of the startSystem event, the login event cannot occur, of course, until the treatment is over.
We can go even a step further. Suppose we find a property of a scenario desirable for all scenarios that are created in the same way, i.e., by using the concrete mapping of ids to roles we used in the example. In that case, we can translate this property into a VO that has to hold for all scenarios that are created, relying on this mapping for initializing the treatmentAllowed variable. Every time we create a scenario view the same way, it must comply with this VO.

Related work
The Tokeneer case study [12] is a well-known example of the application of formal methods to the security domain. Tokeener is an extensive system that provides safety for a network of workstations in a protected room, which can only be entered via verified bio-metric scans and id cards. In addition, the Tokeneer system has the task of verifying access for persons and checking certificates for credentials. The case study was modeled in different languages like the modeling language Z [4]. For this implementation, there exist several approaches to verify the system, e.g., Cristiá and Rossi [5]. The specification was also translated from Z to Event-B by Rivera et al. [13]. While these contributions show that formal methods can be successfully applied to help design and implement securitycritical systems, they only tackle the problem from the verification side. The question of validation is mainly unaddressed.
VOs aim to complement by showing that the actual requirements specified by stakeholders are present in the model. Up to now, many techniques in formal methods have aimed to show the absence of faulty behavior. For example, model checking traverses the state space to check if a state violates a previously defined invariant. Another example is proofs that can show if the access to data structures is well defined. However, verification, i.e., checking whether faulty behavior is absent in the model, is not enough as a designer has no feedback on what the model is capable of. For this, we need validation, and while there exist validation approaches, e.g., proposed by Fitzgerald et al. [6], these are tool and languagespecific. That is where VOs are a valuable addition, as they provide a tool and language-independent formalism. Requirements are formulated as VOs, which evolve with the model while remaining traceable. Additionally, different model views enable other stakeholders to understand the model and reason about the requirements relevant to them without going into unnecessary details.

Conclusion and future work
This paper shows how the VO approach can validate security concerns alongside safety and functional properties in a formal model. A uniform approach for validating different system properties is a big plus, thus making the overall development process more straightforward. Furthermore, the VO approach helps spot conflicting requirements and existing bugs and prevents from introducing the new ones. Additionally, we can create views on a model that facilitate the model's understanding by all stakeholders. From views, we can extract knowledge that we can feedback into the model. Finally, we can formulate VOs that describe the scenario's desired outcomes, thus ensuring compliance for the whole development cycle.
In the future, we want to apply the VO approach to large-scale case studies and show the multitude of VOs in different settings. In such a study, we would like to investigate the following research questions: -What are the limitations of the VO approach? -Is the VO approach equally beneficial to formal methods which are not statebased? -How does the approach scale in large-scale case studies?