BITS: Bi-level Imitation for Traffic Simulation

Simulation is the key to scaling up validation and verification for robotic systems such as autonomous vehicles. Despite advances in high-fidelity physics and sensor simulation, a critical gap remains in simulating realistic behaviors of road users. This is because, unlike simulating physics and graphics, devising first principle models for human-like behaviors is generally infeasible. In this work, we take a data-driven approach and propose a method that can learn to generate traffic behaviors from real-world driving logs. The method achieves high sample efficiency and behavior diversity by exploiting the bi-level hierarchy of driving behaviors by decoupling the traffic simulation problem into high-level intent inference and low-level driving behavior imitation. The method also incorporates a planning module to obtain stable long-horizon behaviors. We empirically validate our method, named Bi-level Imitation for Traffic Simulation (BITS), with scenarios from two large-scale driving datasets and show that BITS achieves balanced traffic simulation performance in realism, diversity, and long-horizon stability. We also explore ways to evaluate behavior realism and introduce a suite of evaluation metrics for traffic simulation. Finally, as part of our core contributions, we develop and open source a software tool that unifies data formats across different driving datasets and converts scenes from existing datasets into interactive simulation environments. For additional information and videos, see https://sites.google.com/view/nvr-bits2022/home

Introduction
Simulation is an integral part of developing effective robotic systems. Simulators allow developers to rapidly verify changes and triage erroneous behaviors before deploying to physical systems. Realistic simulators are especially crucial for autonomous vehicles (AVs), because it is costly and potentially dangerous to test new features and changes directly on the road. Yet, despite advances in physics simulation and high-fidelity sensor simulation, AV developers still primarily rely on large-scale, real-world road testing for validation and verification [1][2][3][4][5][6][7]. One critical reason why is that existing simulation platforms do not generate realistic behaviors for simulated road users, such as cars and pedestrians, which is difficult because, unlike physics and graphics, it is challenging to design models that generate human-like behaviors from first principles.
Today's mainstream driving simulators synthesize agent behaviors by either replaying recorded driving logs or implementing heuristics-based controllers. While log replay allows for scenariospecific triaging, it is difficult to validate new features as replayed agents do not react to counterfactual ego motions. On the other hand, heuristics-based controllers are often equipped with simple driving logic and can thus respond to new ego behaviors in a closed-loop manner [8][9][10]. While these methods can produce plausible traffic flows, synthesizing diverse and complex driving behaviors such as yielding and cutting-in for a large number of real-world scenarios remains challenging.
On the other hand, learning-based approaches can ground reactive behavior generation in real-world driving logs. For example, recent works show that trajectory forecasting models trained from largescale driving logs can accurately infer distributions of future agent trajectories in many challenging scenarios [11][12][13][14]. While these methods excel at predicting realistic trajectories, they are brittle under domain shifts such as new scenes with unseen driving behavior, and the multi-agent nature of traffic simulation may cause a combinatorial explosion in the number of agent states. This challenge is exacerbated when applying prediction approaches to closed-loop behavior simulation over long time horizons, as prediction errors at each step compound over time [15], leading to divergence and irreversible failures such as collisions and driving off-road.
In this work, we aim to develop a learning-based traffic simulation model that can generate diverse, stable, and realistic traffic behaviors. Our key insight is two-fold. First, while learning stable longhorizon driving behaviors requires large amounts of data, the problem has a natural bi-level hierarchy that can be exploited to improve learning efficiency. Specifically, we decouple the learning problem into high-level intent inference and low-level goal-conditioned control. Our model leverages a 2D birds-eye-view structure of urban driving and learns to generate a spatial distribution of intended goal locations (Fig. 1). A low-level controller policy then generates short segments of goal-conditioned behaviors that move agents towards their inferred goals. In addition, the spatial goal distribution aids in disentangling multi-modal behaviors and generates diverse traffic simulations. At the same time, while such a hierarchical policy greatly improves learning efficiency, agents may still encounter unseen situations that the model alone struggles to handle. To stabilize the long-term behaviors of the model, we augment the policy with a prediction-and-planning module. The module samples likely trajectory segments from the hierarchical policy and selects actions with rule-following cost functions as regularization. This way, the overall framework balances between generating human-like behaviors at in-distribution states and preventing divergences at out-of-distribution states.
We name our method BITS (Bi-level Imitation for Traffic Simulation). We evaluate BITS on two popular driving log datasets, Lyft Level 5 [16] and nuScenes [17]. The Lyft dataset contains 1000 hours of driving data collected along a 6.8 mile route in Palo Alto. While densely covered, the trajectory annotations are auto-labeled using a perception stack, resulting in abundant labeling errors [18]. In contrast, the nuScenes dataset contains 5.5 hours of manually-annotated trajectories spanning two cities (Boston and Singapore) with more diverse scenarios. Through these two datasets, we demonstrate the capability of our method under different types of learning challenges. Beyond generating realistic traffic behaviors, we also explore ways to evaluate the generated behaviors through a suite of analytical and learned metrics, since conventional trajectory generation metrics such as ADE and FDE are ill-fitted for evaluating closed-loop traffic simulation. Finally, as part of our core contributions, we develop and open source a software tool that unifies data formats across different AV datasets and allows users to transform scenarios from existing datasets into interactive simulation environments. We hope that our novel traffic behavior simulation method, along with the evaluation protocol and data interface software, can serve as a foundation for further research on this topic.

Related work
Traffic simulation. Approaches for traffic simulation can broadly be categorized into two groups: macroscopic, studying large scale traffic flows without instantiating individual agent states, and microscopic, focusing on individual agents in traffic and simulating their behaviors on the road [19,20]. Since our ultimate goal is to validate autonomous vehicles and their interaction with surrounding agents, we focus our attention on microscopic simulation. Existing systems for microscopic simulation such as SUMO [8], Aimsun [9], and VISSIM [10] use analytical models to control agents in a scene, including cellular automata, the intelligent driver model (IDM), and the optimal velocity model (see [21] for a comprehensive review). These analytic models typically have fixed routes for vehicles to follow and separate longitudinal and lateral motions of agents, with some models omitting lateral motion entirely. Accordingly, these analytical microscopic traffic simulation tools lack sufficient complexity and expressiveness for developing or evaluating autonomous driving features.  Figure 1: BITS framework overview: Decision context c t is a tensor containing the semantic map and rasterized agent history concatenated channel-wise. Given c t as input, (1) the spatial goal network produces a 2D spatial distribution of short-horizon goals, (2) the goal-conditional policy generates a set of actions for each sampled goal, (3) a trajectory forecasting model predicts the future motion of the neighboring agents, and finally (4), based on the predicted future states, the framework selects the set of actions that minimizes a rule-based cost function.
As a result, recent works have started to develop more expressive models for traffic simulation, generally based on neural networks [22][23][24], that learn from real-world trajectory datasets [25,16,17]. Notably, STRIVE [24] proposes to generate near-collision scenarios by searching in the latent space of a trajectory prediction model. TrafficSim [23] adapts a graph-based trajectory prediction model to perform scene-level traffic simulation. Compared to STRIVE, which focuses on generating worstcase scenarios and short open-loop trajectories, we aim to synthesize a broad range of long-horizon closed-loop traffic behaviors. Compared to TrafficSim, which relies on scene-level control to ensure consistent interaction among agents (e.g., coordinated collision avoidance), our method enables each agent to act without coordination with others at the model level. Such an agent-centric setup allows our method to be deployed in more practical simulation usecases, where different types of agents (analytical or learned) are mixed together to interact in a scene. Moreover, we also show that our method outperforms TrafficSim in simulation diversity and stability.
Trajectory prediction. A separate set of research aims to predict the trajectories of agents in traffic, with initial works applying analytical models such as Social Forces [26,27], Hidden Markov Models [28] and IDM [29]. However, limited by their expressiveness, they fail to scale to the level of complexity required by autonomous driving. As a result, various data-driven models have been proposed, which directly learn to predict agent behaviors from a wealth of demonstrations. Several early examples include Social GAN [30], GAIL [31], MFP [32], and DESIRE [33]. More recent trajectory prediction models can be further categorized into agent-centric models, which generate independent predictions for each agent in a scene [34,33,35], and scene-centric models, which generate joint predictions for all (or a subset of) agents in a scene [36,37]. For a comprehensive review of trajectory prediction methods, see [38]. While these methods perform well over short time-horizons (up to 5s), their performance generally degrades over longer time-horizons. To combat this, many recent state-of-the-art methods adopt a multi-stage approach which first predicts agents' goal locations and subsequently links agents' current positions to their inferred goals [39][40][41][42][43][44][45][46][47]. While these methods solely focus on open-loop prediction, we will show in this work that inferring and conditioning on an agent's goal also improves the stability of long-horizon closed-loop simulation.
Imitation learning. Our bi-level imitation learning method is heavily inspired by literature in hierarchical decision making and multimodal imitation learning. A hierarchical policy consists of a high-level planner that sets abstract goals and a low-level policy that learns to achieve the goals [48][49][50][51][52][53][54]. Methods along this vein have favorable properties such as compositionality [53] and interpretability [55], and they have achieved superior performance in long-horizon tasks especially [51][52][53][54]. At the same time, an equally important desideratum of traffic simulation is behavior diversity, which most prior works neglect. Multimodal imitation learning was recently studied in the manipulation domain [56][57][58][59]. Notably, GTI [58] trains a CVAE-based planner to set multi-modal subgoals in observation space for a low-level goal-conditioning controller to achieve. Our method adopts a similar hierarchical structure but instead exploits the domain structure of driving to efficiently represent goal distributions as 2D birds-eye view spatial maps (shown in Fig. 1). We empirically show that our method generates realistic, diverse, and stable long-horizon traffic simulations.

Bi-level imitation for traffic simulation
In this section, we dive into the details of the traffic simulation problem and our primary technical contributions. We propose (1) a hierarchical imitation learning framework that generates diverse and realistic traffic behaviors, (2) a prediction-and-planning module that stabilizes long-horizon simulation, and (3) a suite of analytical and learned evaluation metrics for traffic simulation. Specific implementation details (network architectures, shapes) will be described in Sec. 4.1 and the Appendix.

Traffic simulation as imitation learning
We take an agent-centric approach to traffic simulation, i.e., each agent makes decisions in a decentralized manner without explicit coordination. As mentioned previously, this allows for flexible integration with existing simulation frameworks containing other types of simulated agents and encourages the emergence of new interactive behaviors. We focus on simulating vehicle traffic in this work, but an agent can be any type of road user captured in driving logs (e.g., cyclists, pedestrians). We use s and c to denote the dynamic state and decision-relevant context for an agent, respectively. Specifically, state s includes the position, heading, and velocity of an agent. Context c = (I, S) includes a local semantic map I and the h previous states of an agent and its N neighboring agents S t−h:t = {s Given the decision context information c t and the current state s t , the goal of a traffic simulation model π θ is to generate the next state of the agent s t+1 = T (π θ (c t ), s t ) subject to a dynamics transition function T (·). We use a simple unicycle model with dynamics constraints as T and defer more realistic vehicle dynamics models to future works.
We leverage driving logs captured in the real world [16,17] to train our traffic model. Since log data readily includes semantic maps and the trajectories of all observed agents, we can treat driving logs as a set of multi-agent expert demonstration sequences τ = {c i=0 and formulate traffic simulation as a supervised imitation learning problem. However, the nature of urban driving poses significant technical challenges. First, the decision process is partially-observed as the model does not have access to the underlying intent of the demonstrator and other decision-relevant cues such as the turn signals of other vehicles. Accordingly, action supervisions are inherently ambiguous and are usually modeled with probabilistic distributions [34,33,35]. Although this ambiguity complicates training, effectively modeling action distributions also enables generating diverse counterfactual traffic simulations. Second, since each agent acts without explicit coordination, their joint behaviors create a combinatorial space of possible future states. Such uncertainty makes generating stable traffic simulations extremely challenging. Below, we describe how our approach can generate multimodal simulations with a stochastic hierarchical policy and mitigate uncertainty in state evolution with a prediction-and-planning module.

Bi-level imitation learning for multi-modal behavior generation
The goal of our traffic simulation model is to produce diverse and plausible behaviors by learning from real-world driving logs as demonstrations. Most existing methods in trajectory prediction use deep latent variable models (e.g., VAEs) to capture the behavior distributions. However, findings in the imitation learning literature [60] suggest that learning to generate stable long-horizon behaviors requires a large amount of training data. Our method instead decomposes the learning problem into (1) training a high-level goal network that captures the spatial distribution of possible short-term goals, and (2) training a deterministic goal-conditional policy that learns to reach the predicted goal. The spatial goal network exploits the 2D birds-eye-view structure of driving motion and represents the spatial goal distribution efficiently with a 2D grid. This decomposition additionally moves the burden of modeling multi-modal trajectories to the high-level goal predictor, enabling the low-level goal-conditioned policy to reuse goal-reaching skills to improve sample efficiency.
Spatial goal network. The spatial goal network is trained to predict the distribution of the short-term goal pose (2D position and heading) p(s t+H |c t ) of an agent given its decision context c t . Following prior works [22,61], we first encode the decision context into a rasterized semantic map, which includes the semantic map I and past agent trajectories rasterized as 2D bounding boxes in additional channels. The model takes as input the rasterized semantic map and outputs a 2D grid of goal likelihood as well as residual components to refine the predicted goal location. The output takes the shape of a 4-channel tensor with the same spatial size as the input rasterized map. Channel 0 is the likelihood of the coarse goal location 2D probability map. Each pixel in channel 1 and 2 is the (x, y) scalar residual relative to the grid location. Channel 3 is the heading prediction at each grid location. Once a location is selected based on the probability map in channel 0, the location is corrected by the residual and transformed into a goal poses t+H in the agent local coordinate frame. We treat the 2D location map as a joint distribution and train via cross-entropy loss across locations. The other channels are trained with masked regression losses (e.g., squared error).
Goal-conditional policy. The goal-conditional policy takes the form of a deterministic trajectory generator s t:t+H = π θ (c t ,s t+H ). Although we may further augment the policy with stochastic components, we empirically found it unnecessary as the short-term goal largely reduces the uncertainty in prediction. Inspired by prior works [34,23], instead of directly regressing each state in an agent's trajectory the model predicts controls (velocity, change of heading) at each future time step and forward integrates them through an agent's dynamics model (e.g., extended unicycle dynamics [62] for vehicles). The errors between the predicted and reference trajectory are then backpropagated directly through the dynamics model. Overall, this strategy provides a strong dynamically-grounded learning signal which corrects predictions at earlier steps to reduce errors at later steps.

Prediction and planning for long-horizon stability
So far, we have described a bi-level imitation learning method that can generate plausible traffic simulations from limited data. The policy can synthesize diverse behaviors by sampling from the multi-modal spatial goal predictor. However, the performance of the policy remains bounded by the size and coverage of training data. Driving logs are biased towards nominal behaviors and contain almost no safety-critical situations such as collisions or driving off-road. The objective of generating diverse behaviors further amplifies this challenge, as agents are encouraged to enter previously-unseen regions of the map and create new interactions. As a result, to achieve stable long-horizon simulations, agents must generate reasonable behaviors even at states where guidance from training data is lacking.
To this end, we propose to augment our policy with a prediction-and-planning module to stabilize long-horizon rollouts. The module draws action samples a t from the stochastic bi-level policy π θ described above and selects the action that minimizes a rule-based cost function C given the predicted future states of the environment S t:t+H , that is, arg min at∼π θ C(a t , S t:t+H , c t ). This approach is similar to the motion planning pipeline in a typical modular AV stack, with the important difference that we use our learned policy to generate human-like motion trajectory candidates. The key idea is that the policy π θ can directly follow the data likelihood at in-distribution states, where most action samples are rule-following, and receive corrective guidance at states where the most likely actions may lead to bad consequences. In addition, the sampling module allows for agile adjustment of the simulator (e.g. level of diversity, emphasis on multiple objectives) without retraining. Below we describe the model for future state prediction and the cost function for action selection.
Future state prediction. Since we assume an analytical vehicle dynamics model and known static map, the main task of the model is to predict the future motion trajectories of nearby agents. We follow a typical trajectory prediction pipeline and featurize each agent by its local and global scene context. Specifically, we use RoIAlign [63] to crop the features extracted by the intermediate layer of a deep CNN. The per-agent features are then concatenated with a global scene context feature (extracted by the final layer of the same CNN) to make the final trajectory prediction s (i) t:t+H for each neighboring agent i. The model is illustrated in Fig. 1. We use deterministic prediction in this work and defer more sophisticated probabilistic prediction and planning to future work.
Cost-based trajectory selection. We consider two rule-based costs: collision and road departure. The collision cost is computed based on distances from the corners of two bounding box rectangles using distances between the four corners. To calculate the road departure cost, we first generate a distance map that records the Manhattan distance to the drivable area in pixel space (obtained efficiently via D convolution steps). The resulting distance map assigns zero to points within the drivable area, with values increasing outside the drivable area until saturating at D. We directly use this Manhattan distance value as a penalty. Note that both cost terms are zero for nominal trajectories, i.e., trajectories that do not result in collision and road departure, minimizing their effect on selecting among rule-following action samples.

Evaluation metrics for traffic simulation
Designing metrics for simulation is particularly difficult because of the lack of ground truth. As a result, metrics such as average displacement error (ADE) and final displacement error (FDE), commonly used to evaluate trajectory prediction, do not suit the evaluation of simulation models. To address this evaluation gap, we propose three types of simulation metrics: (i) metrics measuring how much simulated agents violate common traffic rules, such as driving off-road or causing collisions with other agents; (ii) metrics measuring the statistics of simulation rollouts, including resemblance to collected driving logs in terms of driving characteristics such as speed profile, control effort, coverage of the driving area, and behavior diversity between different simulation trials; (iii) data-driven metrics learned from real-world driving logs, such as measuring the likelihood of simulation rollouts under data-driven trajectory forecasting models. Here we describe (ii) and (iii) in details.
Coverage and diversity. To calculate how much of a scene is covered by the agents, we first compute a simulator's trajectory distribution via Density Estimation with a Gaussian kernel over all time steps of the simulator's rollouts, focusing on the 2D spatial distribution of the trajectory. To measure the coverage of the map, we count the number of grid points where the KDE estimate is above a threshold, separating the count between drivable areas and non-drivable areas. To measure the diversity of stochastic policies, we run multiple trials with the same initial condition and collect the density estimates for each trial. Given two different trials, we compute the Wasserstein distance between the two density profiles. In particular, all grid points with non-zero density are flattened and a distance matrix is computed containing the Euclidean distances between every pair of grid points. The density profile is then normalized to sum to 1 and the Wasserstein distance can be computed efficiently as in [64]. For n trials of the same scene, we calculate the Wasserstein distances between the n(n − 1)/2 pairs of density profiles and take the mean as a metric for diversity: where Wass(·, ·) is the Wasserstein distance and ρ i is the density profile for the i-th trial.
Learned metrics. There are many potential ways to learn a metric that evaluates whether simulated behavior is human-like. One such way is to evaluate simulated agent trajectory likelihoods using a prediction model trained from real-world driving logs. To evaluate this possibility, we develop an occupancy-based prediction model that predicts where an agent will be in future time steps. The model uses a similar structure to our spatial goal network and discretizes the position space into bins. The model is trained to minimize a cross-entropy loss function and its predictions are then used to compute the trajectory likelihoods of simulated trajectories. We include more details in the Appendix.

Evaluation
Our experiments seek to validate the primary claims that (1) BITS can generate plausible behaviors by learning from real-world driving logs, (2) compared to other flat policies, our hierarchical policy learning framework achieves better sample efficiency and behavior diversity, (3) our proposed prediction-and-planning module is effective at providing corrective guidance for out-of-distribution states. We conduct evaluations with two large-scale real-world driving datasets, Lyft Level 5 [16] and nuScenes [17]. Since learning-based traffic simulation is a new topic and lacks a standardized benchmark, in this work we also develop and open source a software framework that unifies data formats across different AV datasets (starting with the two used in this work) and can transform scenes from datasets to interactive simulation environments. We use this framework to run closed-loop simulations and report performance based on the metrics described in Sec. 3.4.

Evaluation setup
Datasets. The Lyft dataset [16] contains 1000 hours of driving data collected along a 6.8 mile route in Palo Alto. The dataset contains many repeated trips along each road segment. Since the annotations are auto-generated using a perception stack, there are many labeling errors including inaccurate agent positions, headings, and semantic types [18]. In contrast, nuScenes [17] contains 5.5 hours of accurate manually-labeled trajectories spanning two cities (Boston and Singapore) with more diverse scenarios and denser traffic. Through these datasets, we compare our method to baselines under different learning challenges such as noisy labels and small training sets. For both datasets, we train all models on trajectories from the train split and conduct evaluation on 100 scenes randomly sampled from the validation split. We consider only vehicle simulation in this paper and defer simulating other types of agents (e.g., pedestrians, cyclists) to future works.
Simulation environments. As stated above, we initialize our simulation environments from real driving data, leading to realistic agent placements and dynamic states. Note that all scenes are drawn from the validation split previously unseen to the trained models. Each agent in the scene is independently controlled by replicas of the same model. The simulation runs at a frequency of 10 Hz and results are reported on 20-seconds simulation episodes.  We consider methods from both the traffic simulation and trajectory prediction literature. SimNet [22] is a deterministic behavior-cloning model for traffic simulation. TrafficSim is an agent-centric adaptation of the original traffic simulation method in [23] that features an isotropic Gaussian CVAE. We remove the scene consistency loss in training since we do not assume control over all agents. SocialGAN [30] learns to generate trajectories through adversarial imitation. TPP is adapted from Trajectron++ [34], comprised of a discrete CVAE with Gaussian trajectory decoder for each discrete mode. We also consider variants of these methods augmented with our planningand-control module (marked with "+p"), i.e., selecting future action samples with a cost function. We also evaluate ablations of our method, BITS (max) takes the maximum-likelihood action instead of sampling and BITS (sample) samples actions for rollouts without the prediction-and-planning module described in Sec. 3.3. All methods share the same rasterized input format, ResNet-18 encoder backbone, and MLP-based trajectory decoder networks.
Metrics. As mentioned in Sec. 3.4, designing evaluation metrics for traffic simulation is challenging as there are not single quantities that can summarize the performance of a method, and we cannot easily compare with ground truth dataset trajectories as our goal is to generate new and diverse simulation rollouts. To address this problem, we consider three types of evaluation metrics.
Environment metrics measure rule violations, environment coverage, and trajectory diversity. Both coverage and diversity as described in Sec. 3.4 are calculated from 5 simulation trials with different seeds per scene. We define a critical failure as an agent colliding with other agents or driving off-road Figure 2: Trajectories generated by each stochastic method over 5 trials in the Lyft dataset [16]. Our method (BITS, last row) generates diverse and stable long-horizon simulation rollouts (visualized as colored lines emanating from agents). Other methods suffer from a lack of diversity (e.g., TPP [34] on top) or high collision and off-road rates (e.g., TrafficSim [23] in the second row). Agents are represented with blue bounding boxes and trajectory line color denotes simulation timestep.
for more than 1s. Failure Rate (FR) is the average fraction of agents experiencing a critical failure in a scene. We also report raw collision rate (coll) and road departure rate (offroad).
Dataset metrics compare simulation and ground truth data statistics. They are computed using a normalized Wasserstein distance between the histograms of the driving profiles of the simulated and recorded trajectories. We focus on speed and jerk, commonly used as driver comfort metrics, in the main paper and include others in the Appendix. We also report scene Average Distance Error (sADE) which measures the average position differences between simulated and recorded trajectories. Note that sADE is not suitable for measuring simulation realism and is included only as a reference since it heavily penalizes alternative simulations (e.g., turning left vs. going straight).
Finally, learned metrics as described in Sec. 3.4 measure simulation realism based on a likelihood model trained from real-world driving log. Table 1 and Table 2 show quantitative results of closed-loop simulation on Lyft and nuScenes datasets, respectively. Fig. 2 qualitatively visualizes trajectories generated by selected methods and Fig. 3 shows a more detailed analysis on time-to-failure caused by road departure error. We make the following core observations from these results.

Main results
Recorded driving data is noisy. As stated above, both datasets contain certain levels of labeling noise indicated by non-zero failure rates for ground truth data (labeled "Dataset"), with higher noise in Lyft than nuScenes and a majority of errors stemming from vehicle-vehicle collisions due to imprecise bounding box labels. Our method BITS is able to achieve lower failure rates than even the recorded trajectories in both datasets.
Sample efficiency. nuScenes contains far fewer training samples than Lyft, necessitating high sample efficiency in order for a policy to manage compounding errors over long simulations. As shown in Table 2, even without the prediction-and-planning module, both variants of BITS achieve low failure rates compared to other baselines. For a more direct analysis, we also report the mean time-to-failure in Fig. 3, observing that failure rates in nuScenes increase significantly over the course of simulation for the non-hierarchical policy baselines and remain low for BITS and its ablations.
BITS generates diverse and stable simulations. We observe that the baseline methods exhibit trade-offs between generating diverse rollouts and overfitting to a single mode of behaviors. For example, in Lyft, TPP suffers from mode collapse which yields low failure rates at the cost of low diversity. TrafficSim achieves relatively high diversity and coverage, but also high failure rates. This observation is corroborated by the visualizations shown in Fig. 2, where all simulation trials by TPP are visually identical and resembles the ground truth (on right, titled "Dataset"), and while the trajectories generated by TrafficSim are diverse, some suffer from collisions and road departures. In contrast, BITS simultaneously attains high diversity and coverage with a low failure rate. This contrast is more pronounced in nuScenes where the training set is small. BITS achieves a balanced performance even without the prediction-and-planning module thanks to its high sample efficiency.
Prediction-and-planning is not always effective. The prediction-and-planning module is generally effective in reducing failure rates, with two important exceptions: (1) when action samples are not diverse, and (2) when all action samples lead to failure. Case (1) is exemplified by TPP in the Lyft environment (Fig. 2), where the model's predictions overfit to a single behavior mode. Case (2) is exemplified by SocialGAN in both datasets. While the simulations are relatively diverse, they have high failure rates, entailing poor action sample quality. The prediction-and-planning module has negligible effects on the simulation in both cases.
Quantifying behavioral realism. As discussed above, evaluating simulation realism remains a challenging open problem because there is no single correct answer for traffic simulation. Here we consider both dataset statistics and learned metrics as a proxy for quantifying behavioral realism. For Lyft, we see that all methods achieve comparable speed and jerk statistical distances relative to the recorded trajectories. As expected, SimNet has the lowest sADE due to its behavior cloning objective. In nuScenes, BITS achieves comparable performance to SimNet in dataset metrics, showing that our method does not have to sacrifice behavioral realism for diversity and stability.
Finally, we consider the learned metric described in Sec. 3.4. To show that this occupancy likelihoodbased metric indeed captures meaningful data likelihoods, we roll out ground truth trajectories with different levels of Ornstein-Uhlenbeck noise [65] and measure the predicted likelihood score. As shown in Fig. 3, the likelihood score decreases smoothly as the noise intensity grows, indicating that the learned metric captures the effect of disturbances well. We report the learned likelihood scores for representative baselines in Table. 3. Sensibly, we see that in both datasets the ground truth dataset trajectories have the highest likelihood scores. BITS yields comparable or higher likelihood scores than other baselines, with scores on par with ground truth dataset trajectories on nuScenes data.   Fig. 1 in the main text) BITS framework overview: Decision context c t is a tensor containing the semantic map and rasterized agent history concatenated channel-wise. Given c t as input, (1) the spatial goal network produces a 2D spatial distribution of short-horizon goals, (2) the goal-conditional policy generates a sequence of actions for each sampled goal, (3) a trajectory forecasting model predicts the future motion of the neighboring agents, and finally (4), based on the predicted future states, the framework selects the sequence of actions that minimizes a rule-based cost function.

Implementation details
As shown in Fig. 4, our work includes three learned component: spatial goal network, goal-conditional policy, and the future state predictor. We will describe their shared input, network backbones, and the output heads below. Table 4 includes key hyperparameter selections.
Input. All three networks take rasterized decision context as input. A decision context c t includes (1) a semantic map including information such as lane marks, crosswalks, and lane boundaries and (2) kinematic history states of all agents (vehicles) in the view. For the semantic map, we follow the original rasterization scheme included in Lyft [16] and nuScenes [17] development kit, respectively. Specifically, Lyft semantic map is a 3-channel tensor that includes lane boundary, lane areas, traffic light faces, and crosswalk information. nuScenes's semantic maps are 7-channel tensors including lane, road segment, drivable area, road divider, lane divider, pedestrian crossing, and walkaway. Following prior works [22,61], we use a binary occupancy map same size as the semantic map to represent the kinematic states (i.e., position, heading, and extent) of agents at a given timestep. Each agent in the view is rasterized as filled bounding boxes on the occupancy map. State histories are represented as multi-channel occupancy maps. We concatenate the semantic map and the rasterized agent history map channel-wise and use the tensor as input for all models.
Architecture details. All three networks use an identical ResNet-18 ConvNet backbone to encode an input decision context tensor into a compact feature vector. Here we describe the output heads of each network. Goal-conditional policy. Given the feature and a goal pose, i.e., target position and heading in the agent coordinate frame, the goal-conditional policy uses a MLP-based trajetory decoder to generate a length-h trajectory as actions. As mentioned in the main text, inspired by prior works [34,23], instead of directly regressing each state in an agent's trajectory, the decoder predicts control inputs (velocity, change of heading) at each future time step and forward integrates them through an agent's dynamics model (e.g., extended unicycle dynamics [62] for vehicles) to obtain the trajectory predictions. The model is trained end-to-end with an L2 regression loss against the recorded trajectories in the dataset, similar to a regular behavior cloning objective.
Future state predictor. The network is part of our prediction-and-planning module that selects action based on the predicted future states of the environment. Since we assume an analytical vehicle dynamics model and known static map, the main task of the network is to predict the future motion trajectories of nearby agents. As described in the main text, we follow a typical trajectory prediction pipeline and featurize each agent by its local and global scene context. Specifically, we use RoIAlign [63] to crop the features extracted at an intermediate layer of the ResNet-18 encoder. Specifically, we use a cropping window of 7 × 7 at the end of the second block of the network. The per-agent features are then concatenated with a global scene context feature (extracted by the final fully-connected layer of the ResNet-18) to make the final trajectory prediction s (i) t:t+H for each neighboring agent i. We only make predictions for agents that are visible in the rasterized map. The model is trained with an L2 regression loss against the recorded neighboring agent trajectories in the dataset. The goal-conditional policy and the future state prediction network share encoder network weights and are trained jointly.
Spatial goal network. The spatial goal predictor extends the ConvNet encoder backbone with a U-Net-style decoder network (mirroring the encoder size) to generate the spatial goal prediction. Specifically, the model generates a 2D grid of goal likelihood as well as residual components to refine the predicted goal location. The output takes the shape of a 4-channel tensor with the same spatial size as the input rasterized map. Channel 0 is the likelihood of the coarse goal location 2D probability map. We use a grid size of 0.5 meter per pixel. Each pixel in channel 1 and 2 is the (x, y) scalar residual (in meters) relative to the grid location. Channel 3 is the heading prediction at each grid location in radian. Once a location is selected based on the probability map in channel 0, the location is corrected by the residual and transformed into a goal poses t+H in the agent local coordinate frame. Note that it is possible to further discretize the heading prediction into discrete bins [66], but we empirically found that the heading prediction has negligible impact on the goal-conditional policy. We treat the 2D location map as a joint distribution and train via cross-entropy loss across locations. The other channels are trained with regression losses (e.g., squared error). Training setup. We train all models on shared cloud computing nodes equipped with NVIDIA Tesla V100 GPUs (32gb GPU memory). We train all models for 100k iterations (gradient steps) and choose the checkpoint based on their respective offline validation metrics (e.g., prediction error). On average, training takes between 20-30 hours to complete with single-GPU jobs.
Cost functions for prediction and planning. As mentioned in the main text, we consider two rule-based costs: collision and road departure. We follow prior work [34] and compute the collision cost based on distances from the corners of two bounding box rectangles using distances between the four corners. The minimum distance d min (·) is approximated as: which is further illustrated in Fig. 5. We then add the following collision loss Col = Sigmoid(−αd min − β), where α and β are parameters used to shape the sigmoid loss. In our experiments, α = 1.0, β = 4.0.
To calculate the road departure cost, we first generate a distance map that records the Manhattan distance to the drivable area in pixel space. To be specific, we first pick a maximum distance constant D and set all pixels inside the drivable area to 0, and all pixels outside the drivable area to D. Then we perform the following convolution step D times: x i,j = min{x i,j , x i−1,j + 1, x i+1,j + 1, x i,j−1 + 1, x i,j+1 + 1}, where x ij is the value at the i, j coordinate. The resulting distance map assigns zero to points within the drivable area, with values increasing outside the drivable area until saturating at D. We directly use this Manhattan distance value as a penalty. To account for the size of the vehicle, we perform RoIAlign on the distance map with vehicle patches that take the vehicles' size and orientation into account. Note that both cost terms are close to zero (the collision loss may be nonzero due to the sigmoid function) for nominal trajectories, i.e., trajectories that do not result in collision and road departure, minimizing their effect on selecting among rule-following action samples.
Learned metrics. To compute likelihood of a simulated trajectory, we use an occupancy-based prediction model that predicts where an agent will be in future time steps. The model uses a similar structure to our spatial goal network and discretizes the position space into bins. The model is trained to minimize a cross-entropy loss function and its predictions are then used to compute the trajectory likelihoods of simulated trajectories. During evaluation, the likelihood of an agent's trajectory is calculated in a receding horizon manner. For each time step, we compare its rollout trajectory for the next T = 20 steps to the predicted occupancy and calculate the likelihood and aggregate the whole rollout duration by taking the mean. The likelihood values of multiple agents in the scene are also aggregated by taking the mean.

Baselines
SimNet. SimNet [22] is a deterministic behavior-cloning model for traffic simulation. Prior work also show that its rasterized encoding scheme yields strong performance in trajectory prediction tasks [61]. We use the same rasterization input and CNN-based encoding backbone for all baselines and our BITS model. SocialGAN. SocialGAN is adapted from a trajectory prediction model [30]. The model is trained using conditional generative-adversarial learning, where the condition is the decision context and the generation target comes from the trajectory dataset. To facilitate a fair comparison, the model uses the same encoding backbone and MLP-based trajectory decoder as all other methods.
TPP (Trajectron++). TPP is adapted from the Trajectron++ model [34], which is a CVAE model with a discrete latent space and generates a Gaussian Mixure Model (GMM) as trajectory prediction. We modified the encoder to take rasterized features as input, which shares the same map encoder as the proposed BITS model. The TPP model generates predictions of the control input, which are propagated through a unicycle model to obtain the trajectory prediction. For simplicity, we directly let the decoder predict the variance of predicted trajectory instead of propagating the variance through the dynamics as in [34]. The latent dimension is chosen to be 10 in our experiments.
TrafficSim. TrafficSim is an agent-centric adaptation of the original TrafficSim work [23], meaning that agents in a scene are controlled by independent replica of the same model instead of by a single model that explicitly coordinates their actions. The TrafficSim baseline uses an isotropic Gaussian CVAE to model action distributions and facilitate diverse simulation. We adopt a similar differentiable dynamics trajectory decoder from the original paper but remove the scene consistency loss in training since we do not assume control over all agents.

Complete evaluation metrics
Here we provide the exhaustive list of (non-learned) evaluation metrics we considered in this work and report performances in Table 5 for Lyft and Table 6 for nuScenes.
• FR: The average fraction of failed agent in each scene. The failure can be caused by either collision failure or offroad failure (defined below). • collFR: The average fraction of failed agent in each scene, where the failure is caused by agent colliding with any other agent in the scene at any timestep throughout an episode. • offroadFR: The average fraction of failed agent in each scene, where the failure is caused by agent departing drivable region for more than 1 second (10 timesteps for a simulation frequency of 0.1 second). • coll: The average fraction of agent colliding with another agent. We in addition list the type of collision occurred (rea, front, side). Note that each collision type is independently calculated, i.e., each agent can have multiple type of collision per episode. • offroad: The fraction of timesteps that an agent spend outside of drivable area, averaged across agents per scene. • coverage and diversity: As described in detail in the main text, both metrics measure the diversity of rollouts across multiple simulation trials with the same initial condition. The coverage metric calculates the (discretized) map area covered by multiple simulation trials with the same initial condition. The diversity metric in addition computes the Wasserstein distances among the density profiles of the map coverage attained by different simulation trials. In practice, we accumulate the metrics across 5 trials and compute the coverage density by discretizing the map at a 2m by 2m resolution. • Dataset metrics. Dataset metrics compare the driving profile of simulation trajectories with trajectories recorded in the dataset. We consider speed, lon acc (longitudinal acceleration magnitude), lat acc (latitudinal acceleration magnitude), and jerk. To calculate the difference between the driving profile of the simulation and the dataset, we first collect histograms of these driving profile measurements for both the simulated and the dataset trajectories across all evaluation scenes. We then calculate the distances between the histograms of the simulation and the dataset using Wasserstein distance normalized by a constant. We use 20 bins for all histograms, with a speed range of We also report scene Average Distance Error (sADE) and scene Final Distance Error (sFDE), which measures the average and the final position differences between simulated and recorded trajectories, respectively. Note that sADE and sFDE are not suitable for  measuring simulation realism and is included only as a reference since it heavily penalizes alternative simulations (e.g., turning left vs. going straight).

Planning cost weight ablation
As described above and in the main text, we use two cost terms: collision and road departure in our prediction-and-planning module. We conduct an ablation study on different planning cost weight setting using the Lyft dataset and report the result in Fig. 6. Because there are only two cost terms, we mainly consider the ratio of the cost weights. We set offroad cost weight to either 0.0 and 1.0 and continuously vary the collision cost weight at a log scale. We observe that both terms are effective at reducing rates of their corresponding cause of failure, the reduction saturates when the weight is above a certain value, and the overall failure reduction performance is not sensitive to the cost weight ratio. Figure 6: Failure rates of BITS with different planning cost weights.

Prediction and planning horizon ablation
Here we investigate the impact of the planning horizon H in BITS's planning and control module. We train all three components (i.e., goal-conditional policy, future state predictor, spatial goal network) with the same set of prediction horizons: {10, 20, 50, 80} on the Lyft dataset. We report performances using essential metrics in Table 7. We observe that H = {10, 20} yield similar performances, but the failure rates increase significantly as the model expands its prediction horizon to H = {50, 80}. This is because predicting long-term future is extremely challenging, especially when all agents are controlled by stochastic policies. The model's decisions can be easily misled by wrong future predictions, leading to critical mistakes. At the same time, we also observe that while short-term predictions result in more successful simulation, they lead to less smooth driving behavior caused by more frequent braking and acceleration, as indicated by the dataset metrics (e.g., higher acc and jerk).
Hence an important future direction is to dynamically adjust the prediction and planning horizon by taking into account prediction uncertainties.

Additional qualitative results
We include additional qualitative results for BITS. Rollouts using the nuScenes dataset is shown in Fig. 7 and that of the Lyft dataset is shown in Fig. 8. Each row shows different rollouts from the same initial scene configuration. The recorded trajectories are shown in the rightmost column for reference.


Towards Building Cyberphysical Systems with Agile Architecture

*e current stage of technology development is characterized by an increase in the complexity of the created anthropogenic systems, a constant expansion of the scope of information technologies, an increase in the intelligence level of the created systems, and the appearance of new paradigms for building information-oriented systems such as cyber-physical systems, the Internet of things, and cloud and fog systems. Modern information-oriented systems very often have dynamic structure, implement complex adaptive behavior, and can be considered as systems with agile architecture. *e article discusses one of the possible approaches for building cyberphysical systems with agile architecture on fog platforms. *e idea of the proposed approach is to accumulate knowledge about the current state of the observed cyberphysical systems in the form of knowledge graphs. As a model, it is proposed to use multilevel relatively finite state operating automaton at the upper level and knowledge graphs at the lower level. A distinctive feature of the developed approach is that models that describe the current state of the observed system can be built automatically.

Introduction
e progress of engineering and information technologies makes it possible to create very complex anthropogenic systems. e complexity of such new generation of systems results from the increasing of the number of elements, the number of internal and external links, the complexity of behaviour, and the variability of the structure and behaviour of such systems in time.
Modern anthropogenic systems can be defined as complex, multilevel, heterogeneous, distributed, networkcentric and data-centric, and intelligent systems. e implementation of these qualities requires the presence of an information processing component, including large-volume information storage and powerful computers. Tallows consider modern anthropogenic systems as Software Intensive Systems (SwIS) [1].
Modern systems consist of elements of different nature, such as mechanical and electromechanical elements; it can also be natural entities, biological and human entities.
Nowadays, one can observe appearance of such new classes of systems as cyberphysical systems (CPS) and sociocyber systems [2,3].
In other words, one can say that a significant part of complex anthropogenic systems has the property of changing their structure and behaviour depending on the external or internal contexts. Such systems can be defined as variability-intensive systems (VIS) [4,5]. General trends in the development of anthropogenic systems allow assuming that the systems of next generation will be also VIS.
Solving the problem of variability management for complex, multilevel heterogeneous distributed systems is a nontrivial task, especially when reconfiguration decisions are made autonomously at several levels in different subsystems of a distributed system. is article discusses one of the possible approaches for solving the problem of managing variability at the architectural level by using the agile architecture approach (AAA). e investigation is conducted in relation to CPS built on fog platforms [6], which are actively used in the construction of modern large-scale systems. e proposed article includes 13 sections. Section 2 defines such concepts as variability, agility, agile architecture (AA), and their relationship. Section 3 contains the problem statement of the study. Section 4 discusses the current state of the agile architecture systems (AAS) construction technology. Section 5 describes the suggested approach. Section 6 discusses the problem of building an Agile Architecture CPS (AA CPS). Section 7 discusses the main tasks to be solved to support agility in CPS. In Section 8, it is proposed to use the automata representation to describe the mechanisms of agility. Section 9 describes the Agility Support Models (ASM). In Section 10, it is proposed to define 5 maturity levels when building AACPS. Section 11 discusses possible approaches for implementing AACPS. Section 12 provides an example of using AAA. Section 13 contains conclusions on the article.

Variability, Agility, and Agile Architecture
Nowadays, many different terms are used to describe the variability of the structure and behaviour of the systems. In relation to information system, for SwIS and for CPS, such terms as adaptive, flexible, reconfigurable, dynamic, and agile are widely used [4,5,7]. e term agility in Wikipedia is defined as "the ability to change the body's position efficiently" (https://en. wikipedia.org/wiki/Agility). In the IT sphere, this term is also actively used. In relation to SwIS, the term agility can be defined as the ability of a system to adapt to a certain context [5].
ere are three main variants of the term agility use: (i) the use in a broad sense as a synonym of such terms as flexibility and adaptability [7], (ii) relation to the SwIS design process [8], and (3) relation to SwIS architectures [4,5]. It should be noted that the concepts of Agile Process (AP) [8,9] and AA are correlated to each other, although the mechanisms used are different. AP is a process that is focused on working with constantly changing requirements and belongs to the design stage. AAS are systems which are able to adapt to changes in the runtime context.
Recently, instead of the term agility, the term variability is sometimes used [4,5,9]. However, according to the authors of the article, these are different concepts. Variability can be defined as the ability of a system to change its state, and agility can be defined as the ability of a system to respond to a change in state or context.
It should be noted that majority of modern complex systems implement the mechanisms of agility [10,11]. e subject of consideration in this article is AACPS implemented on fog platforms.

Problem Statement
ere are several reasons why we deal with the variability of structure and behavior, the main of which are the following: (i) increase in performance by means of system scaling, (ii) increase in reliability (availability) indicators due to the implementation of the self-healing procedure, (iii) support for intelligent interfaces, (iv) restructuring in order to optimize the functioning, and (v) modernization of the system. e problem of variability management can be formulated as follows. It is necessary to find mechanisms that allow determining the current state of a CPS, to present information in the required form to stakeholders and (or) to form reaction to a change in the structure, behavior of the observed (managed) system (OMS), or a change in the context. e proposed article considers a possible approach for solving the problem of managing variability at the architectural level for multilevel complex distributed heterogeneous CPS.

The State-of-the-Art Agile Architecture Systems
Publications related to AAS can be divided into two groups: publications related to agile architecture systems themselves and publications related to the study of mechanisms that can be useful in implementing AAS. e first group includes early publications on dynamic computers such as [12], publications related to various aspects of adaptation, early publications related to AA [4], and recent publications on the topic [5,11]. All these works are devoted to the consideration of the general principles of AAS building but contain few information about possible approaches to the agility mechanisms realization.
Despite the fact that the concept of AA itself was introduced quite a long time ago, the authors are not aware of works that offer AA models and frameworks.
is article is a further development and generalization of the results of the authors research in the fields of the synthesis of relatively finite automata, the synthesis of knowledge graphs [19,20], and data acquisition systems in distributed heterogeneous CPS [21].

Suggested Approach
e developed AAA is based on the following principles.
(1) e proposed AAA assumes the construction of a multilevel system of models operating in discrete time in terms of discrete states. Each element of the model is a dynamic model of the corresponding architectural element of the model and can be to answer all the requests of all potential stakeholders, which can include both users and subsystems, for example, the agility management subsystem.
e need to move the OMS to a new architectural state (Ast) may be required in the following cases: internal events can occur, such as changes in the structure because of the failure of individual subsystems, the appearance of new subsystems, or suboptimal functioning of the system, when it is necessary, for example, to perform load balancing.
AACPS can be considered as a Context Aware System (CAS) [23]. At the same time, it is necessary to distinguish between static and dynamic agilities. At the development stage, development time agility is implemented, and at the operation stage run time agility is realized.
It should be noted that the proposed approach is close to the Digital Twins approach [24]. It can also be considered as an implementation of the VIS [5] and evolutionary architecture concepts [25] applied to CPS.

Agile Architecture CPS
In recent years, CPS has found increasing practical usage and CPS can be considered as one of the possible areas of application of the proposed approach. It should be noted that CPS can be considered as a design paradigm that is based on the architectural approach to design [26]. To date, significant parts of the existing and projected SwIS are systems consisting of components of different physical nature, i.e., CPS. At the same time, it should be noted that the increase of the level of intelligence of created CPS and the increase of usage of ambient intelligence systems (AMIS) [27] are observed.
e Generalized Structure of the CPS. In the most general sense, modern CPS can be defined as multilevel (multilayer) systems, which are systems of the surrounding intelligence. It should be noted that modern CPS can include many thousands of elements. e generalized structure of the CPS is shown in Figure 1. In general case, CPS consists of 6 levels (layers): sensor level, fog computing level, cloud computing level, CPS level, CPS systems level, and AmIS. e sensor layer, fog layer, and cloud layer correspond to the levels of the IoTreference model [28]. Many CPS are built on the fog [18] platform, and, for CPS fog, the platform is represented as a set of services. e system can consist of an arbitrary number of CPS that can be integrated at different levels (data, applications, and user interfaces). At each level, the models are described using specialized dictionary. Stakeholders interact with the system through the distributed human machine interface subsystems, which support AMI interface [27]. It is necessary to mention that different groups of stakeholders use different Domain Specific Languages (DSL) [29] to interact with CPS and can have concerns correlated with different levels.
e Main Types of Variability that Occur in CPS. Obviously, when using models in run time mode, it is necessary to track changes in the CPS state. e main types of variability in structure, behavior, and context that may need to be monitored when using the model approach are shown in Figure 2. is variability classification can be conceded as adaptation of [30] to CPS and run time agility. Variability can manifest itself both in the structure and in the behavior of an OMS. Changes in the structure and/or behavior can occur for a number of reasons. When using dynamic architectures, this is a normal process. Changes in the structure and behavior of the OMS can be associated with the implementation of contextual or content adaptation in order to optimize functioning. Changes in the structure may be due to malfunctions or modernization of the system, in particular, switching on or off the equipment. us, we can assume that models can be used to solve tracking changes in the structure of the OMS, behavior changes, and tasks of tracking user activities.

Main Tasks to Be Solved to Support
Agility in CPS e implementation of the proposed approach is reduced for solving the following main tasks: (i) building and maintaining a multilevel model in an adequate state, (ii) building a script that implements the procedure for collecting data about the current state of an OMS, (iii) support DSLs using which different categories of stakeholders can communicate with OMS, and (iv) bringing the OMS in line with the model.

Automata Representation of AACPS
Formally, AACPS can be defined as: AACPS � <OMS, OMSMM, SH, I1, I2, I3, I4>, where OMS is an observable and managed system, OMSMM is agility management module (subsystem), and SH is a set of stakeholder groups (not necessarily people) and 4 interfaces. I1 is the interface for maintaining the relevance of the OMS, I2 is the interface for supporting the correspondence of the architectural state of the model and the OMS, I3 is the interface for querying the state of the OMS, and I4 is the interface for managing the OMS ( Figure 3). e following main special cases can be distinguished: (i) the structure and behavior of the OMS being static, (ii) when it is only necessary to monitor the state of the OMS, and (iii) when there is only one user group. In the case when the variability of the structure and behavior is absent or minimal, there is no need to use dynamic models. In this case, models can be built in statics at the design stage based on Model Driven Architecture (MDA) models.
If stakeholders only want to receive data about the state of the OMS, then we are dealing with a DSH [31]. In this case, the control can be carried out via other channels, for example, in manual mode. If there is only one group of stakeholders with the OMS, then models can be built in terms of the corresponding subject area. If there are several different groups of stakeholders who have different concerns, then the task of presenting the model becomes more complicated.
Formally, the interaction of OMS and OMSMM can be represented as interacting automata. e OMS automaton can be defined as OMSA � <InOMSA, OutOMSA, STOMSA, TROMSA>, where InOMSA is a set of input signals, OutOMSA is a set of output signals, STOMSA is a set of internal states, and TROMSA is a function of transitions and outputs. InOMSA is information about events that initiate the transition to a new state. ese can be internal events, external events, or commands to change the state (reconfiguration). OutOMSA is, for the most part, logs. STOMSA is a set of states of the elements that make up the OMS, and these can be elements of different physical nature. TROMSA are the rules, according to which a decision is made to switch to a new state. e rules define FF and they can be of any complexity. e OMSMM automaton can be described as OMSMM � <InMMA, OutMMA, STMMA, TRMMA>, where INMMA are logs coming from OMSA or commands from users. OutMMA is basically the results that are given out according to user requests. STMMA is a set of internal states of an abstract automaton. TRMMA is a function (table) describing transitions between OMSMM states. is function can be quite complex. e output signals can be associated with both states and transitions. e output signals represent the results on the DSL [29].
In general, OMSMM is a distributed system consisting of a set of interconnected nodes (OMSMMN), each of which can be described as OMSMMN � <SE, AM, I, Srv> where SE is the entity being modeled, AM is a model, I is an interface with the OMS, and Srv are model access services. Interfaces are intended for communication with the underlying layers, and services are intended for communication with the overlying layers.
e simulated entity can have any nature. With rare exceptions, the model is a virtual entity. I is interface for communication with the observed object; Srv are supported services.
e interface I can be defined as I � <RQ, ES>, where RQ is a set of requests for data collection, and ES is event streams. e typical structure of the agility support node is shown in Figure 4. e agility support node is a virtual machine that includes the following main elements: (i) Agility Support Models (AM) repository, (ii) AM processor, (iii) DSL processor, (iv) events processor, (v) log request processor, and (vi) control signals generation processor. e AM repository is designed to store AM files that can be presented in different forms. AM processor is responsible for working with models. Events processor is responsible for processing the flow of events, which are presented in the form of logs (L). Log request processor is responsible for the formation and execution of scripts that implement the collection of data necessary for building an AM. is processor sends requests to receive logs (L). Control signals generation processor is responsible for generating management actions for OMS reconfiguration.
e functioning of the agility support node can be organized in different ways [21]: (i) the user request processing is implemented within a single process; with each request, the AM is built from scratch; (ii) there is a separate process responsible for keeping the AM up to date; for each specific moment of time there is an up-todate AM; (iii) a mixed strategy is used; for example, at the upper level, the AM is built in the background mode, and, at the lower level, the model is completed when a specific request appears. e third option is of real practical interest, since it is quite difficult to store and keep up-to-date complete model of a large CPS. us, two parallel processes are implemented in the agility support node: the process of keeping the model up to date and the process which is responsible to realize stakeholder requests. e algorithm for keeping the model up to date: (1) Start the OMS monitoring procedure (2) Receive logs (3) Clean and sort logs (4) If AM correction is required, then continue; otherwise go to item 2 (5) AM correction and go to item 2 Algorithm for processing user requests: (1) Wait for a request from a user (2) Request acceptance and transformation DSL⟶MQL (Model Query Language, the language of the request to the model) (3) Request to the model using MQL (4) If the response is not received, then continue; otherwise MQL⟶ DS transformation and go to item 1 (5) Script synthesis (6) Script execution (7) Construction of the required AM of OMS (8) Request for a new model (9) MQL⟶ DSL and the transition of the item 1 e task of matching the state of the OMS with AM can be defined as the task of obtaining the OMS with the required ASt according to the model, which is AM. If we take into account that each ASt i corresponds to exactly one AM i , then the automaton describing the dynamic structure and behavior, which will be discussed below, can be conceded as a Digital Twins (DT). In this case, each transition is additionally loaded with the reference (Ctr) to the script that is to be executed for transition to this ASt. If for each architectural element of the OMS a kind of a set function is available that can set the element to any valid state and the order of execution does not matter, then the task becomes trivial.

Agility Support Models
As a system of models that meet the requirements for ASM, a two-level model M � <MA, MSB> can be proposed, MA is a model of the upper (architectural) level, and MSB is a model of the lower (structural-behavioral) level. MA describes the OMS in terms of architectural states and MSB in terms of structure and behavior. An MSB can contain an arbitrary number of nesting levels.
e Top-Level Model. is model describes the behavior of an OMS in terms of the change of ASt under the influence of external or internal events (Ev) and control actions (Ctr), which sets the OMS into this state. Using this model, it is possible to describe the AA [7,8] in terms of changes in architectural states ( Figure 5). e transitions between architectural states can be matched with Fitness Function (FF) [25]. e concept of agility assumes that for one system there are several architectures; in the other words, AA system may Journal of Computer Networks and Communications 5 have several architectural states. e transition between ASt occurs under the influence of both internal and external factors. In the first case, we can talk about the implementation of self * mechanisms (self-testing, self-healing, etc.), and, in the second case, we can talk about CAS [23]. As a model describing the functioning, an AA system it is proposed to use a multilevel relative finite state operational automaton (MLRFSA) [15,16], which describes the implementation of agility mechanisms in terms of transitions between architectural states (In the Figure 5, Ev denotes events and Ctr denotes the reconfiguration control signals). e event can be either internal or external. It should be noted that only in the simplest cases the structure of the considered automaton is fully known; most often there is some incomplete knowledge about the structure of the automaton. In this case, it is necessary to solve the problem of an automaton constructing (synthesis). Algorithms of MLRFSA synthesis are described in sufficient detail in the publications of the authors [15,16]. e automaton according to Figure 5 is a MLRFSA operating in discrete space and discrete time [13,32]. is is a class of automata in which the sets of acceptable parameters are, in general, finite only at the interval of one step of behavior. At the same time, it is possible to change the set of valid input, internal, and output states of the automaton, as well as the set of valid functions of transitions and outputs of the automaton; i.e., completely rebuild the automaton.
A MLRFSA state can be described by 10 parameters where d a r is an input parameter vector; d b r is an internal state parameter vector; d c r is an output parameter vector. Functions F b r are the functions that describe conditions of the changing automaton internal state and F c r are the output functions.
e automaton behavior can be described as follows: States d b r , d c r , and d a r and functions F b r and F c r describe automaton at the r-th moment of time and must satisfy the following conditions: e transition from the automaton MLRFSAr to the automaton MLRFSA r+1 at r + 1 moment can be defined as e functions of the automaton transitions from one state to another can be also represented as If many functions are used in the transitions, then the transitions are described as follows: where z and v define the type of functions used in transitions.
To each of the conditions, their statuses can be assigned: main condition, precondition, and postcondition. For preconditions, numbers of key terms under which they are true can be determined, and postconditions define terms on which they are false. If you set all the elements in the conditions to their corresponding predicates P zv0 (F zv0 (g)) and P zv1 (d zv1 ), . . . , P zva (d zva ), which take the value 1 when the variables are defined (true) and 0 otherwise, taking into account the considered relations, the automaton model of the observed object can be described in the form of a structure: linking d S and d w . For practical implementation, described MLRFSA are required to have relevant information about the current state of the system, the changes taking place in the structure of the system, and the context; on the basis of this information, a decision on the transition of the system to another architectural state is made. e MLRFSA synthesis algorithms are considered in detail in [13,14,32].

Lower-Level Models.
e lower-level models are domainoriented architectural models and can include several points of view [33]. e type of model used is determined by what DIK stakeholders want to get about the OMS. It should be noted that large CPS are characterized by the presence of several groups of stakeholders with different concerns. e functioning of SwIS is usually described in terms of structure and behavior [22]. e Structural-Functional Model (SFM) can be used for this purpose. is model is designed to generate models describing the OMS in terms of the structure and implemented behavior in the form of BP. e proposed SFM is a bipartite colored graph and is defined as where OP is the set of operators, L is the set of arcs, COL is the coloring of arcs, GO are the rules for starting operators, SEMO is the semantics of executing operators, STRM is the strategy mask, tk is the set of tokens, TK0 is the initial markup, CSG is the control signal generator, and RMS is the resource monitoring system. OP operators are operators of any level of complexity; the operators are connected by arcs L, which can be colored in one of 4 colors: arcs, through which Ld data is transmitted, Lc arcs, through which control signals are transmitted that allow the execution of OP, Lz arcs, through which requests for the execution of operators are transmitted, signals, and Lr arcs, through which signals about the availability of resources are distributed.
GO are rules for launching operators: where tki is the presence of a type i token at the operator input and STRMi is the i-th bit of the STRM mask.
us, depending on the mask used, there are 16 different ways to check the readiness of operators for execution. is model can be used as a metamodel. On its basis, it is possible to build various kinds of private models that can describe the OMS in terms of both structure and behavior. A more detailed description of this model and how to build it based on log files is contained in [21]. For automatic construction Journal of Computer Networks and Communications of the SFM model, one can use modified Process Mining algorithms [18]. is model is one of the possible models. It should be noted that different models can be used at different levels.
e top-level models are models in terms of which the agility mechanism is implemented, and the lower-level models are, on the one hand, a way to store information about the current state of the OMS and, on the other hand, a source of information for the formation of SFM.

AAA Maturity Levels
When solving real life problems, the proposed AAA is not necessarily used in full. One can define 4 levels of AAA maturity.
Level 0. Systems with a Fixed Architecture. e OMS has a fixed architecture, which is created at the design stage. Reconfiguration at the architectural level is not planned at the design stage. e transition between ASt can be carried out automatically. However, all the ASt are known at the design stage. An architectural automaton can be built in statics at the design stage.

Level 3. Evolutionary Architectures.
e OMS is designed according to the principle of an evolutionary architecture. e transition between ASts can be carried out automatically. At the design stage, all the ASts are not known. In this case, both ASts themselves and individual elements may be unknown. e architectural automaton (OMSMM) is built in dynamics.

Possible Approaches to Implementation
In general, a large-scale CPS model is a system of model systems (SoM), which can be considered as system of systems [22]. Depending on the type of OMS, models can be organized into the SoM in different ways. Different ways of representing ASM can be used at different levels. Figure 6 shows two edge variants of building a SoM. Figure 6 shows a variant when all interactions between the system elements are implemented through the use of ASM, and Figure 6, b shows a variant when deferred processing is implemented and the OMS and ASM interact through a telemetric communication channel and through a database (DB). In reality, there may be a large number of intermediate variants that are most often used in practice.
In Figure 6, the following designations are adopted: P is a physical entity, Li is the i-th level of the OMS, Mi is the i-th level model, and DSL is the interpreter of a domain-oriented language.
One can define three main approaches to the implementation of AM: (i) the implementation of the model, i.e., using JAVA in the form of an object model, (ii) based on using ontologies [34], and (iii) based on using knowledge graphs [19,20,35,36]. Using JAVA to build models allows get minimal delays, but it is very expensive in terms of programming. e use of ontologies and knowledge graphs gives slower solutions but allows using existing tools, such as SPARQL [37]. Since the models are quite complex, it is advisable to implement them as cloud services. Individual domain-oriented model fragments can be placed in the fog layer.
Nowadays, it is preferable to build ASM in terms of knowledge, in particular, using ontologies and knowledge graphs. e issues of automatic ASM construction in terms of knowledge graphs are considered in [19,20].

Case Study
As an example, which illustrates the possibilities of using the proposed AAA in practice, one can consider the crane complexes, which is an element of a flexible production system of an automated assembly and welding site.
is system, in turn, is a part of a manufacturing enterprise with a high level of automation, planning to implement the concept of Industry 4.0 [38][39][40][41].
e existing system is designed to collect and process data from crane systems and is designed for use on production sites, each of which consists of several shop-floors, in which bridge and semicrane cranes work. e cranes are equipped with analog and digital sensors. Analog sensors are used to measure distances between cranes, between trolleys (for double-bed cranes), and between the trolley and the edge of the crane, weight of the load, ambient temperature, motors, temperature of frequency converters, voltage, and current of the power supply network. e discrete sensors include sensors that determine the extreme upper and lower positions of the hook, the condition of the repair gates, the position of the hatch in the cabin, and all fuses in the switch cabinet. Data is collected from all sensors at 250 ms intervals. e total number of measured parameters on each tap is about 560. e total amount of data received from each tap per month is approximately 2 TB.
Based on the data received from the sensors, the tasks of assessing the condition of the cranes and ensuring their operability are solved. In operating mode, different data such as data on the state of the fuses must be collected at a frequency of at least 2 GHz, from the load mass sensors of at least 2 Hz, and from the engine temperature sensors of at least 1 Hz. In the idle mode of the crane, it is necessary to collect information only about the state of the power supply and the ambient temperature. When collecting data, it is necessary to ensure that a delay in data transmission from the cranes to the workplace of the chief mechanic is no more than 5 seconds. e structure of a preexisting CPS is shown in Figure 7. It is a distributed system that includes equipment installed on cranes, workshop equipment, and main (plant) server. e equipment installed on the cranes includes sensors and actuators (S&A). For data acquisition (DA) subsystem 8 Journal of Computer Networks and Communications (DASS), a local controller (LC) and a local database (LDB) are used. e data acquisition process is divided into two stages: data collection from sensors, which is carried out on the crane, and data transfer from the crane to the central server. e list of data to be collected is fixed. Data are collected from the sensors in real time. MIT-SUBISHI MELSEC-Q series industrial controllers installed on the crane are used, providing control of the crane, as well as being a buffer for data collected from sensors. e internal memory of the controllers is accessed via the Seamless Message Protocol. e crane also hosts a single-board ODROID computer for data transfer from LC to ODROID; an Ethernet communication channel with a bandwidth of 100 Mbit/s is used. is channel also transmits commands to the control mechanisms of the crane. e channel provides high quality communication but has limited bandwidth. Overloading the channel can lead to the loss of control commands and cause failures in the operation of the crane. e data are collected by theODROIDC3 single-board computer. ODROID C3 has sufficient computing power to poll the registers at the required frequency. e collected data are placed in the LDB installed on the crane.
Data transfer from the LDB database to the central server can be carried out via a Wi-Fi communication channel or by an operator. e use of the Wi-Fi channel is problematic due to the high level of electromagnetic interference from welding and AC sources installed in the shop-floor. On average, the ratio of the number of lost and transmitted packets is 1 : 5. Data collection by the operator involves the use of the operator's tablet. e software installed on the tablet allows receiving data from the LDB over the radio channel and caching it on the tablet. e range of the radio channels does not exceed 20 meters, which requires the operator to be located near the crane.
is system can be attributed to the maturity level 0, since it is actually a system with a fixed architecture.

Problems.
is CPS was in trial operation for some time, according to the results of which the owners formulated the following problems: (i) insufficient amount of collected data, (ii) long delays in obtaining the data by decision-makers, (iii) low data quality, and (iv) high total cost of ownership (TCO). e enterprise owners have plans to implement a system that meets the industry 4.0 standards, including usage Digital Twins (DT) technologies. But, nowadays, the list of features which are to be realized is absent. e analysis showed that in order to realize Industry 4.0 ideas it is necessary to achieve at least agility level 3. In addition, the owners of the system have plans for building and using the corporate knowledge graph, which is also needed to be taken into account when working on the modernization of the system.
A more detailed analysis of these problems showed the following.
e insufficient amount of data is collected due to the following reasons: (i) user requirements for the composition and quality of the data collected are constantly changing; (ii) users cannot say with certainty what data they need in the future.
e reasons for the three remaining problems are the very high level of electromagnetic interference caused by the presence of powerful welding equipment, which does not allow realizing an effective data exchange between the mobile crane and the workshop server. In the existing system, an operator with a tablet goes to the crane and takes data via Wi-Fi. Using this mode leads to unpleasant consequences: (i) there will be large delays in receiving data, while some of the data may become outdated, and (ii) a part of data may be lost due to buffer overflows if the operator with the tablet does not have time to pick up them.
If you want to increase the amount of data being captured, it is necessary to increase the buffer size. e requirement to have a human operator responsible for data collection increases the TCO. e analyses of these problems have shown that the root cause of the problems is a lack of agility.
us, the solution of the problem of improving the efficiency of the system operation was reduced to solving 2 problems: (i) the problem of an agile system organization and (ii) the problem of implementing effective interaction between the crane controller and the workshop server.
To solve these problems, the following main decisions were made: (i) to increase the level of agility, it is proposed to use the model approach described above, (ii) it is proposed to modify the structure of this system and use the concept of edge (fog) computing, and (iii) to solve the problem of excluding the human operator from the data transfer process with a tablet, it is proposed to use clustering mechanisms.

Suggested Solution.
For DA on the crane, an additional intermediate link (fog node) "ARM CPU" was developed, which provides data collection management (Figure 7). e crane additionally houses a controller based on the APM processor, which performs the functions of a fog node. In the repair shop, controllers are also placed, which perform the functions of fog nodes. In order to reduce the volume of collected and stored data, fog nodes use a priori information from the low-level model (LLM) about the object from which the information is collected, the types of sensors are installed on it, and the dynamics of changes in the measured values, i.e., models of the observed objects, are stored. We can assume that the controllers placed on the cranes form the mist level and the controllers placed in the repair areas form the fog level. e initial scheme assumed measurement of parameter values with a certain sampling step, representation of data in binary form, and their storage in the database. To reduce the data stored in the database, the new scheme detects changes in values, identifies ascending and descending edges, and records the time when these events occur. When collecting data using analog sensors, dynamic changes in the sampling rate are provided depending on the state of the crane. e sampling rate is reduced after the system analyses the rate of change of the measured values over a fixed period of time.

A Cluster-Oriented Model for Collecting Data from
Cranes. To solve the problem of reducing the response time, it is proposed to use the cluster model [42]. e traditional data collection scheme involves direct data transmission using Wi-Fi. e Wi-Fi communication channel in the workshops has low reliability due to the high level of interference. e data transfer rate is inversely proportional to the distance to the access point. Data transfer to the server starts when the threshold value of the accumulated data is reached. With this scheme, the data transfer rate is calculated as c (l) � k/l, where k is the gaintransfer rate. e structure of fog DCS is shown in Figure 8. To ensure fast and reliable DA without the involvement of an operator, a cluster scheme for collecting data from cranes was developed. In each shop-floor, Wi-Fi access points are installed in the repair areas, which act as relay nodes. When a certain amount of collected data is reached on the tap (250 MB), data is transferred from the tap to the head element of the cluster (the crane having the minimum distance to the access point). If there is no possibility of communication with the head element of the cluster (no line of sight), data transmission is carried out through nodes (cranes) located in the visibility zone. Access points are connected to the in-shop network via Ethernet. When the crane is moved to the repair area, the crane data is transmitted to the relay node (Wi-Fi access point). High-level models are realized as distributed knowledge graphs and are located on repair areas, on workshop servers, and on the main server.
As a result of the use of the model approach and the transition to the fog structure, the following problems were solved: (i) providing the ability of expanding the list of collected parameters through the use of high-level model and (ii) the use of low-level model which has halved the amount of data collected. e use of clustering mechanisms allows solving two problems: (i) reducing the response time for data collection from tens of minutes to units of seconds and (ii) excluding a human operator from the DA process. e use of these solutions allowed achieving level 2 of the maturity model because only a limited number of models are available.
In the described above example, the benefits obtained are the reduced data access time (reduced response time) and the reduced TCO, since the transition to new equipment is simplified and opportunities for the transition to systems corresponding to the ideas of Industry 4.0 open up. e further direction of development is associated with the transition to DT technologies [40,41], while it is planned to actively use the created models. It should be noted that the joint use of Industrial Internet of ings paradigms leads to the need to create dynamic DT, but it is a subject of separate investigations.

Conclusion
Using the proposed AAA to CPS implementation allows solving a number of important problems, such as reaching a new level of complexity of the created anthropogenic systems and solving problems of increasing the level of service availability. It should be noted that the paper considers the application of AAA to the construction of the large-scale VIS CPS, while this class of systems is considered as an example of complex heterogeneous systems. e idea of the proposed approach is that the system has several architectures (architectural states) and it is possible to switch between architectural states. is is a core idea of all agile architecture approaches. Our contribution is that we show how it can be done.
Our approach can also be considered as one of the possible implementations of the DevOps approach to design. us, there is an opportunity to get benefits in the TCO terms. Performance gains can be obtained if we consider agility as an approach to reconfiguration. In this case, the gain can be obtained through contextual and (or) content adaptation. e possibility of implementing adaptation mechanisms makes it possible to increase the level of reliability.
A necessary and sufficient condition for using the proposed model approach is a sufficiently high complexity of the structure and behaviour of OMS that have high structural dynamics and adaptive behaviour.
Under these conditions, the use of the model approach reduces the response time to a request for the status of the OMS. In addition, the presence of model knowledge about the past states of the system allows determining the root causes of events and predicting future states of the system to implement proactive management. In addition, the availability of knowledge about past states allows using learning mechanisms.
As the main subject of our future R&D, we see the problem of agile architecture cyberphysical systems security. Nowadays, it is not possible to state unequivocally that agile architecture systems provide a higher level of security compared to traditional architectures. In principle, it is possible to consider the possibility of transition to an architectural state that provides a higher level of security, but this requires assessing the level of security in terms of architecture and building a security model for each architectural state. e authors failed to find adequate approaches in the available sources. We plan to investigate the agile architecture system security and develop architectural tactics for improving security.
Data Availability e data are gathered from crane complexes, which are an element of a flexible production system of an automated assembly and welding site. Provided data files contain volume of data in GB produced by each of the cranes. e data used to support the findings of this study have been deposited in the repository (https://zenodo.org/record/ 5109526#.YPfeXXUzY5k).

Conflicts of Interest
e authors declare that they have no conflicts of interest.