A Method For Automated Drone Viewpoints to Support Remote Robot Manipulation

Drones can provide a minimally-constrained adapting camera view to support robot telemanipulation. Furthermore, the drone view can be automated to reduce the burden on the operator during teleoperation. However, existing approaches do not focus on two important aspects of using a drone as an automated view provider. The first is how the drone should select from a range of quality viewpoints within the workspace (e.g., opposite sides of an object). The second is how to compensate for unavoidable drone pose uncertainty in determining the viewpoint. In this paper, we provide a nonlinear optimization method that yields effective and adaptive drone viewpoints for telemanipulation with an articulated manipulator. Our first key idea is to use sparse human-in-the-loop input to toggle between multiple automatically-generated drone viewpoints. Our second key idea is to introduce optimization objectives that maintain a view of the manipulator while considering drone uncertainty and the impact on viewpoint occlusion and environment collisions. We provide an instantiation of our drone viewpoint method within a drone-manipulator remote teleoperation system. Finally, we provide an initial validation of our method in tasks where we complete common household and industrial manipulations.

I. INTRODUCTION
Remotely teleoperated robots can assist in many unstructured and inaccessible settings. Effective telemanipulation in these environments requires the remote operator to achieve sufficient situational awareness for manipulation. One of the major factors affecting awareness is visual information, which benefits from quality live views during manipulation. The requirements for a quality view can vary dramatically over the course of the task, requiring a variety of positions and orientations. One solution to meet these requirements is to provide an adapting view from a camera on a drone. To reduce the cognitive load and effort associated with manual control of the drone camera, the drone viewpoint can be automated based on available environment and task information. However, while an automated drone has the This work was supported in part by a NASA University Leadership Initiative (ULI) grant awarded to the UW-Madison and The Boeing Company (Cooperative Agreement # 80NSSC19M0124). 1  [esenft|pragathi|gleicher|bilge]@cs.wisc.edu Top: A potential application of our method where an automated drone view supports a user teleoperating a robotic manipulator to press an estop button. Our method suggests alternate drone viewpoints (red) to which the operator can reposition the drone for additional perspective. Bottom: The operator can use the adaptive drone view and additional visual information, such as a view from the end-effector camera, to complete the task.
potential to assist operators with the range of views required to support manipulation tasks, using a drone also introduces challenges. One challenge introduced by the mobility of the drone is how to select a viewpoint when there are multiple quality viewpoints spread across the workspace. Another challenge is how the viewpoint selection should compensate for drone flight characteristics, such as stability of the viewpoint and occlusions or collisions with objects in the environment. To address these challenges, we propose a method that combines human-toggling of automaticallygenerated drone viewpoints with an otherwise automated drone view that considers occlusions, collisions, and drone uncertainty. In our implementation, an operator controls a manipulator through a remote interface and the drone automatically tracks the end-effector to provide an appropriate task view. Many existing works, summarized in Section II, have used drones to provide a dynamic view for applications such as search and rescue or cinematography. However, these existing methods have not focused on providing an automated drone view that is designed for cluttered manipulation environments. Manipulations often benefit from the perspective of multiple viewpoints of the task, for example to see specific parts of the manipulated object. However, particularly in cluttered environments, these viewpoints can require drastically different camera positions. For example, when pressing a button on a panel, as shown in Figure 1, two good viewpoints may be on opposite sides of the workspace. Our method leverages two simultaneous optimizations to allow operators to toggle between multiple viewpoints during manipulation tasks. The first optimization locally commands the drone pose. The second optimization globally queries the workspace, aggregating alternate viewpoints where the operators can reposition the drone, for example to get a different perspective of the manipulation task. When providing viewpoints in cluttered environments, the challenge of uncertainty in the drone's command tracking (i.e., small perturbations and oscillations from the desired pose) must also be addressed. Failing to account for uncertainty could lead to unexpected collisions or occluded views.
In this work, we present a method to enable an automated drone viewpoint for telemanipulation tasks. Our method combines a set of nonlinear optimization objectives, which are defined based on environment modeling from a sensor mounted on the manipulator and empirically measured drone pose uncertainty. These objectives are used within a localglobal optimization that both controls the drone and searches for and proposes alternate viewpoints to the operator. We provide an open-source implementation of our method which provides multiple live camera views from an RGB-D camera mounted to the manipulator and the drone on-board camera. Finally, we provide an initial validation of our method in an indoor, cluttered environment by performing manipulations of household and industrial objects.

II. RELATED WORK A. Methods for Camera Control
Automatic camera control has been studied extensively in computer graphics to provide users with an effective view of a virtual environment (see [1] and [2] for surveys). Many of these existing methods rely on a fully unconstrained camera and a complete understanding of the virtual scene. In real-world scenarios, automatic camera control can be achieved using a robot (such as a drone) that dynamically and autonomously react to visual changes in the environment. In robotics, visual servoing [3] and active perception [4], [5] are control methods that are used to move a robot based on a stream of vision data. In visual servoing, features extracted from vision data are used to define a target pose for the robot and to determine how the robot should move [6]. This technique has been used previously for autonomous drone control (e.g., in [7] and [8]). In active perception, a robot (with an on-board camera) autonomously adapts its viewpoint to gain more information to complete tasks such as object recognition, manipulation, or reconstruction.
We draw from the optimization-based techniques for camera control in computer graphics where visual properties are encoded as objectives, but include real-world drone considerations. Additionally, to address the specific case of telemanipulation, our method is designed to work with an approximate geometric model of the environment captured on site by sensors mounted on the manipulator. Our work shares parallels with 3D visual servoing, but unlike active perception where the viewpoint is adapted for a robot to autonomously complete a task, our method adapts the viewpoint to support human users completing manipulation tasks.

B. Vision for Remote Robot Teleoperation
During teleoperation, effective visual feedback from the remote environment is critical for the operator's situational awareness. A single static view of the remote environment is unlikely to provide all the visual information necessary for adequate awareness [9]. While multiple static cameras can sometimes provide such information (e.g., in [10], [11]), the placement of the cameras is often strategically chosen for the scenario at hand, and slight variations in the task or the environment can make it difficult to complete the task. Additionally, at the time of robot deployment, environments may be inaccessible for setting up multiple static cameras. Hence, one or more dynamic cameras (i.e., cameras that can provide different views) can be valuable in the presence of varying task requirements. Manually controlling a dynamic camera can be effective in certain situations, such as first-person teleoperation of a humanoid robot (e.g., in [12], [13] using head-tracking for camera control). However, in many telerobotics applications, manually controlling a dynamic camera introduces additional workload on the operator and reduces teleoperation performance. Prior work in laparoscopic surgery [14], [15], telemanipulation [16], [17], [18], [19], and search and rescue [20], [21] has shown the effectiveness of employing autonomous dynamic cameras.

C. Drone as Viewpoint Provider
A dynamic camera can be moved using several mechanisms such as a gantry system [22], an articulated head [23], a stationary robot arm [19], a mobile robot arm [24], or a drone [21]. These mechanisms provide different types of viewpoints to the user based on their range of motion and degrees of freedom. For example, the camera on an articulated head in [23] provides first-person pan-tilt views, whereas the camera on a drone in [21] provides thirdperson, six degree-of-freedom views. Each mechanism for moving a dynamic camera has trade-offs. For example, a camera on a drone has high mobility, but uncertainty in its pose information due to flight dynamics that can result in unsafe, unstable, or occluded viewpoints. Drones have been used to provide views of teleoperated robots such as another drone [25], a construction machine [26], or a ground robot [21], [27]. However, these works have not considered cluttered manipulation environments where small errors in pose information can lead to collisions or occluded views.
Drones are also widely used for autonomous aerial cinematography. Our work shares similar goals with drone cinematography such as visual objectives that define the quality of a viewpoint [28], [29] and on-the-fly avoidance of occlusions and collisions [30]. However, visual objectives for telemanipulation need to be tailored to provide a quality viewpoint for manipulation tasks rather than to produce cinematic shots. Similarly, the occlusion and collision objectives can leverage the specific geometries present in manipulation scenarios. In many cinematography applications, events are scripted and the user is able to initially provide the system with specifications about desired viewpoints (for example, through keyframes in [31], [32]). In contrast, the required views or regions of interest are not known beforehand in a number of scenarios where robots are deployed for telemanipulation. This prevents the user from providing any information ahead of time. Thus, telemanipulation scenarios also necessitate fast, online point-to-point optimization and preclude the use of offline trajectory optimization methods.

D. Summary
Previous work has highlighted the opportunities provided by drones to provide viewpoints for robot operators. However, there remain a number of questions that need to be addressed when considering the specific scenario of telemanipulation in unknown cluttered environments.
Similar to [18], [19], [33], our work adapts viewpoints to support human users completing manipulation tasks. Our work differs from previous work in drone camera control in three ways. Our work does not require a semantic understanding of the task or full representation of the workspace; instead we use a simplified geometric understanding of the manipulator, environment, and task (similar to [34]) for selecting viewpoints. Second, we dynamically integrate user preferences by calculating and presenting the user with alternate viewpoints that can be used to reposition the drone. Finally, we draw from the work on drone pose uncertainty (see [35] for a survey) and integrate the empirically measured drone pose error into our nonlinear optimization formulation for viewpoint selection.

III. METHOD
In our method, a drone is autonomously controlled to provide an operator with a dynamic camera view during manipulation tasks. In this section, we describe geometric cost functions that define a quality viewpoint while accounting for uncertainty in the drone pose and avoiding collisions with the environment. We also describe our solution to provide the operator with alternate drone viewpoints.

A. Drone Characterization
Viewpoint characteristics (e.g., occlusion) can be sensitive to small changes in position. One challenge with using drones for dynamic viewpoints is compensating for errors in the drone pose caused by flight dynamics and disturbances. To address this challenge, we geometrically characterize the drone pose uncertainty and use this geometric characterization to inform our nonlinear optimization objectives.
Our premise is to empirically measure the maximum expected deviation of the drone pose. To do so, we use a series of command-tracking tasks (e.g., prolonged stationary pose tracking) to measure the discrepancies between the commanded and actual drone pose. The largest deviations in the drone state variables (x, y, z, θ) are used to represent the uncertainty. Practically, drones can often control yaw (i.e., angle or θ) with a relatively high degree of precision compared to Cartesian directions and we observed this effect on our platform too. Thus, we incorporate the uncertainty from the three Cartesian directions into the objectives involving geometric collisions and occlusions. Depending on the characteristics of a particular drone platform, it is also possible to add additional uncertain parameters as part of the geometric viewpoint objectives.

B. Environment modeling
To geometrically assess collisions and occlusions, the representation of the environment model needs to support rapid distance checking between shapes. In this work, we use planes as a simple geometric primitive to approximate the environment. Planar surfaces are pervasive in human-made environments (e.g., walls and tables), and there are many existing methods to recognize them from unstructured point cloud data. We use these planes to evaluate both collisions and occlusions. Further, we assume that the manipulator will interact with objects on the planes. Thus, we approximate a region of interest for manipulation by calculating the point on the plane closest to the current position of the manipulator's end-effector. This approximation allows us to formulate objectives for viewing angle and distance that effectively support manipulation.
We model the drone and manipulator by enveloping their geometries with simple primitives to obtain convex hulls that are used to approximate collision, viewing, and occlusion regions. The drone collision hull is modeled as a cuboid encapsulating the drone geometry. Each dimension of the cuboid is extended by the empirically-measured drone uncertainty. The drone viewing hull is modeled as a convex hull connecting a cuboid representing the drone camera's position and the manipulator's end effector position. The cuboid is centered on the drone camera position and has the dimensions of the drone pose uncertainty (see Figure 2). The manipulator collision hull is represented by a polyline connecting each of the joints from the base to the endeffector with a constant width based on the width of the robot. Finally, the manipulator occlusion hull is represented with the same structure as the collision hull but without the end effector as part of the polyline.

C. Drone pose optimization
The commanded drone pose is calculated online using a weighted-sum nonlinear optimization: where c i are weighted objective terms (introduced below) with corresponding weights w i and x contains the drone state variables (e.g., x, y, z, θ). The weighted objectives can be divided into three categories: collision objectives, view objectives, and motion-limiting objectives. To improve performance, the cost function at each iteration is optimized over Angle between the drone camera direction and the vector from the drone to the end effector θ β (x) Angle between the vector from the end effector to the closest plane and the vector from the end effector to the drone d dr→plane k (x) Distance between the drone collision hull and the k th plane (negative distance when intersecting) d dr→mc (x) Distance between the drone collision hull and the manipulator's collision hull d dr→ee (x) Distance between the drone and the manipulator's end effector dee→cp(x) Distance between the end-effector and the closest point on a plane d vh→pl k (x) Distance between the viewing hull and the k th plane d vh→mo (x) Distance between the viewing hull and the manipulator's occlusion hull.

dc→new(x)
Distance between the current drone position and the proposed position g P σ (.) The higher-order Gaussian with order P and a standard deviation of σ: g P σ (x) = e −(x 2 /(2σ 2 )) P

Fig. 2. Illustration of geometric quantities used in the objectives. The objective function that uses the geometric quantity is denoted in parantheses.
a small rectangular search space from the last commanded pose. Symbols used in the objectives are defined in Table I. 1) Collision objectives: These objectives steer the drone away from collisions with the environment and the manipulator.
• Environment collision: The drone should stay away from planes.
• Manipulator collision: The drone should stay away from the manipulator.
2) View objectives: These objectives provide a view that supports the operator in doing the current task. The view cost balances viewing the manipulator end effector (the element interacting with the environment) and providing enough context to solve the task. This formulation does not rely on object recognition nor inferring the task goal. We instead use geometric relations to generalize to different tasks and environments.
• Visual target: The camera should be centered on the manipulator end effector.
• Distance to visual target: When the manipulator is close to a plane, it might indicate the operator's intention to interact with an object on this plane. In this case, the drone should move closer to the manipulator.
• Perspective angle: The drone should maintain a specified angle with the plane to provide perspective of any objects on the plane.
• Environment occlusion: The drone's viewing hull should avoid intersection with environment planes.
• Manipulator occlusion: The drone's viewing hull should avoid intersection with the manipulator's occlusion hull (a polyline that excludes the end effector).
3) Motion-limiting objectives: This objective ensures that the current commanded position is within reach of the drone.

D. Alternate Viewpoint Generation and Selection
There are a variety of situations where the operator may desire a different view than what the drone is currently providing. For example, a view from a different angle in the workspace may provide additional perspective or help to gather additional task-specific information (e.g., reading a gauge on one side of a valve). There may also be instances where the online drone viewpoint optimization converges to a local minimum that is insufficient to complete the task.
In these cases, we provide the operator with a small number of quality alternate views for the drone camera. This is accomplished through a second viewpoint optimization that searches globally for alternative views that minimize the drone cost function. Specifically, this optimization uses only the previously defined view and collision objectives (i.e., no motion-limiting objectives). This optimization is seeded using a random starting drone pose and searches the full workspace at each iteration. Additionally, we add one objective to this optimization which penalizes viewpoints that are similar to the current drone viewpoint: The viewpoints from this optimization are clustered and a predefined number of top candidates are stored to maintain a diversity of viewpoints. Stored alternate viewpoints expire after a short amount of time to encourage new viewpoints as the manipulator changes pose. During operation, the user is able to cycle through the alternate viewpoints using a simple toggle. When the user identifies a preferential viewpoint, they can signal the drone to transition to the alternate viewpoint.

IV. IMPLEMENTATION
To test our method for automated drone viewpoint assistance, we developed a manipulator-drone system with a multiple-view remote interface. In this section, we describe our apparatus, optimization details, and interface design. All system-specific implementation quantities (e.g., drone uncertainty, optimization weights) appear in Table II. Our code is available online. 1 1 https://github.com/emmanuel-senft/drone-panda Optimization constants Alternate viewpoint constants 0.1 w 9 1

A. Apparatus
The setup consists of a Franka Emika robot controlled in joint position. The desired joint positions are calculated by a nonlinear inverse kinematics optimization that balances matching the desired end-effector pose, limiting joint velocity, and avoiding robot joint limits. The robot is equipped with a gripper, ATI force-torque sensor, and Kinect Azure RGB-D camera mounted to the distal link with a known transform. The drone camera is provided by a Ryze Tello drone. The drone position is captured used a Optitrack motion capture system and a proportional-derivative (PD) controller is used to command the drone velocity through a python wrapper based on the official Tello SDK. 2 The different components of the system communicate using ROS [36]. The environment is mapped from the Kinect's point cloud using RTAB-Map [37] and is acquired by running an automated routine that controls the robot-mounted camera to create an environment map. Planes are extracted from the resulting point cloud using Point Cloud Library's [38] plane segmentation. The resulting plane locations and inlier points are further processed to determine plane boundaries (i.e., edges) and non-rectangular surfaces are subdivided into smaller planes as necessary using a recursive routine.

B. Optimization
Our implementation consists of three optimization routines that determine the drone pose, manipulator pose, and alternate drone viewpoints. Each optimization was written in RUST and is solved using the PANOC method of OpEn library [39]. The search space, relative to the current drone pose, was bounded to ±1 cm for each dimension of the drone position and ±0.1 rad for the drone orientation, and θ ref and w were set empirically. A C++ wrapper connects the RUST code to ROS. The drone and manipulator pose optimizations run at 100 Hz whereas the alternate viewpoint optimization runs at 5 Hz. All of the geometric collisions and occlusions are computed using the ncollide3d 3 library. When the operator selects an alternative viewpoint, we freeze the manipulator control and use a simple path planning procedure where the drone ascends to the maximum height, moves to above the selected alternative viewpoint, and descends.

C. Interface and Controls
For telemanipulation, we developed a multi-view interface within RViz (as shown in Figure 3) which combines the drone camera view, end-effector camera view, and a simulated view. The simulated view gives a global perspective that consists of the environment point cloud, manipulator, current drone position, and the alternate proposed drone position. The interface also contains a series of widgets at the bottom of the interface, such as toggling the control frame, toggling the alternate view, commanding the drone to move to an alternate view, and the drone battery level. The available control frames are the drone camera frame, the end-effector frame, the manipulator base frame, and the simulated view camera frame. The simulated view also contains visualization of the end effector's applied torque to assist in tasks involving physical interaction, such as turning valves.
The operator uses a gamepad controller to control the 6D pose of the manipulator and to provide high-level commands to the drone such as takeoff, land, and move to alternate view.

V. PRELIMINARY STUDY
To demonstrate our method, we designed a series of common household and industrial manipulation tasks that involved operating an e-stop button, switches, circuit breakers, and valves. The environment was setup as shown in Figure  4. To assess the behavior of our system in these tasks, we carried out a preliminary validation study, involving a single expert operator. In the future, we plan to conduct user studies for a more comprehensive and multifaceted evaluation.

A. Tasks
We designed three tasks with multiple manipulations spread around the workspace. In each task, the manipulator was controlled by an expert operator located in a different room than the manipulator and drone setup. Our primary goal of this preliminary validation study was to show examples of drone viewpoints and alternate views that our system proposes over the course of manipulation tasks in cluttered environments. A secondary goal was to show that our proposed nonlinear optimization was able to avoid occlusions and collisions in live tests. Periods of occlusion (i.e., when there was a partial or no view of the end effector) and collisions for the drone were coded by two of the authors. Transitions between viewpoints (i.e., alternate viewpoints) were not considered in determining occlusions. Figure 5 shows the trajectory of the drone for each task. Example drone viewpoints are also featured in the supplemental video. 4 We also provide additional detail for Task 1 as a case study, including example views (middle of Figure 5), the evolution of the view and collision cost function (bottom of Figure 5), and a snapshot of the cost map (view and collision) in a state similar to Figure 5.D (see Figure 6). No collisions were observed during the execution of the tasks. Notable observations during each task are described below.

B. Results and Observations
Task 1 had a duration of 4:36 (four minutes and 36 seconds). No periods of occlusion were noted during this task. A notable observation was that, as the manipulator approached the stop valve, the drone view was distant and appeared to be in a local minimum. To obtain a more informative viewpoint, the operator repositioned the drone to an alternate side view of the valve (viewpoint E).
Task 2 had a duration of 4:58. There was a one second occlusion as the end effector rotated to approach the circuit breakers, which prompted the operator to reposition the drone to a side view. There was a three second occlusion of the end effector when approaching the light switches that was automatically resolved by the drone controller.
Task 3 had a duration of 3:53. There was an occlusion for five seconds as the manipulator approached the valve. The drone view was higher than desirable and appeared to be in a local minimum. The operator repositioned the drone for a closer, level view of the end effector. There was another occlusion of the gripper for less than one second that the drone resolved automatically. When the manipulator interacted with the toggle switch, the drone maintained a viewpoint that was nearly orthogonal to avoid collisions, which caused the switch to become briefly occluded for less than one second while the end effector remained visible.

VI. DISCUSSION
Our method was able to generally provide an unoccluded view of the manipulator. Observed cases of occlusion were brief and were resolved either by the autonomous viewpoint adaption or, in cases of local minima, by the operator switching to an alternate viewpoint. We believe that these preliminary results demonstrate the promise of our proposed contributions. The local-global optimization is able to generate a range of unoccluded alternate viewpoints, as shown in Figure 6, that can offer additional visual task information. The lack of collisions and minimal occlusion observed in our validation study illustrate the value of the geometric objectives conditioned on the drone uncertainty.
Limitations-Our method has some inherent limitations. First, our method relies on a static map of the environment and does not consider new or dynamic obstacles. Our map also comes from a single scan from the manipulator robot, which does not model all possible obstacles in the environment. In the future, we would like to supplement the robot map with online mapping from the drone that can both update changing parts of the map and model areas missed by the initial robot scan. Our method assumes environment features can be modeled as planes. While many engineered environments are flat, there are certain environments (e.g., curved surfaces, free-standing objects) that may not work well with our objective structure. All drone viewpoint heuristics were modeled as costs, rather than constraints. While we did not note any collisions during our preliminary evaluation, a constraint-based formulation may be appropriate in other circumstances. We geometrically bounded the drone uncertainty, however in some environments, this may be too conservative and a probabilistic model of uncertainty may be more appropriate. In some cases, the user may desire a specific viewpoint based on task semantics (e.g., reading a gauge). In the future, we would like to supplement our method with additional user controls.
Our implementation and preliminary validation had limitations we plan to address in future work. First, our drone had only four degrees of freedom, which limited the viewpoints that the optimization could consider. In the future, we would  Figure 5.E corresponds to a relocation to alternate #3. like to explore drones with a pan-tilt camera, which can provide additional views, such as overhead. Our implementation also relied on external drone state estimation via motion capture. Future work could explore how our method works with other techniques for localization. Finally, our method and system should also be evaluated in a user study with a wider range of tasks.
Conclusion-We presented a method for automated drone viewpoints during robot telemanipulation. We developed an implementation and showed through a preliminary validation that it can provide collision-free unoccluded views during common manipulation tasks.


A Human-Centric Take on Model Monitoring

Predictive models are increasingly used to make various consequential decisions in high-stakes domains such as healthcare, finance, and policy. It becomes critical to ensure that these models make accurate predictions, are robust to shifts in the data, do not rely on spurious features, and do not unduly discriminate against minority groups. To this end, several approaches spanning various areas such as explainability, fairness, and robustness have been proposed in recent literature. Such approaches need to be human-centered as they cater to the understanding of the models to their users. However, there is a research gap in understanding the human-centric needs and challenges of monitoring machine learning (ML) models once they are deployed. To fill this gap, we conducted an interview study with 13 practitioners who have experience at the intersection of deploying ML models and engaging with customers spanning domains such as financial services, healthcare, hiring, online retail, computational advertising, and conversational assistants. We identified various human-centric challenges and requirements for model monitoring in real-world applications. Specifically, we found the need and the challenge for the model monitoring systems to clarify the impact of the monitoring observations on outcomes. Further, such insights must be actionable, robust, customizable for domain-specific use cases, and cognitively considerate to avoid information overload.

Introduction
Machine learning (ML) is increasingly playing an integral role in our day-to-day experiences. Increasingly, the applications of ML are no longer limited to search and recommendation systems, such as web search and movie and product recommendations, but ML is also being used in decisions and processes that are critical for individuals, businesses, and society. With ML based solutions and pipelines in highstakes applications such as hiring, lending, criminal justice, healthcare, and education, the resulting personal and professional implications of ML are far-reaching. Consequently, it becomes critical to ensure that the underlying ML models are making accurate predictions, are robust to shifts in the data, are not relying on spurious features, and are not unduly discriminating against minority groups. This emerging field, called model monitoring, can be viewed as part Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
of a broader ML model governance (Kurshan, Shen, and Chen 2020) and responsible ML framework (Arrieta et al. 2020), and is at an inflexion point, as evidenced by legal/regulatory requirements, requirements from the perspective of web-scale ML applications, and adoption of practical and scalable approaches. Model monitoring is receiving greater attention in light of regulations such as EU GDPR, CCPA, and the EU Trustworthy AI 1 initiative and several ML deployment failures in practice.
From an operational angle, large-scale ML systems require maintenance not only by the virtue of possessing software code but also because of the nuances of ML as a domain itself (Sculley et al. 2015). ML-specific nuances include dependency on data whose distributions can shift during production from when the model was designed and the dependency of a model on another model's output which can cascade issues from the other model onto the dependent model (Sculley et al. 2015). Such nuances imply that maintenance of ML systems requires monitoring its various aspects, such as data and models. Such monitoring is essential to ensure that the model does not become stale or degrade in performance due to changing real-world conditions or changes in the data collection process and data processing pipelines. In Figure 1, we illustrate the various activities in the lifecycle of an ML model and highlight the focus of this work in the monitoring activity. More broadly, the emerging field of model monitoring pertains to practices for deploying and maintaining ML models in production reliably and efficiently (Mäkinen et al. 2021). Monitoring of deployed ML models is needed to determine how often the model needs to be retrained and handle the following issues: (1) Data drift: The distribution of features may change over time, causing the quality of predictions made by the model to gradually degrade (Breck et al. 2019).
(2) Changes in the relationships between input and target variables: The changes in real-world conditions may alter the relationships between input and target variables (often referred to as "concept drift" (Gama et al. 2014;Tsymbal 2004)), and thereby result in degraded model performance.
(3) Data integrity and operational challenges: Compared to traditional software, ML models are more tolerant to unintended or unexpected changes in their inputs, and hence may continue to make predictions, even when the inputs may have been corrupted. As a result, the predictions may be erroneous or of poor quality. Thus, it is essential to ensure data integrity and detect any undesirable changes in the data pipeline (e.g., changing the measurement unit of a feature from feet to yards). (4) Reduced performance for subgroups of users: Although the model performance may not change as a whole, it is possible for the model to exhibit poor performance for certain subgroups of users (for whom, say, the relationship between input and target variables may have changed). Hence, in addition to monitoring overall accuracy and other performance measures, it is important to ensure that the model does not develop bias, and instead performs well across various subgroups of users.
We note that the above issues are of interest not only for data scientists but also for ML engineers, product managers, business decision makers, policy, compliance, and legal teams, internal and external auditors, and other stakeholders. In other words, given the importance of model monitoring as part of a broader AI model governance framework, model monitoring need to be human-centered not just in terms of usability by humans but also accounting for human behavior (Shneiderman 2021;Wing 2021). In this work, our goal is to understand the needs and challenges in monitoring deployed ML models from a human-centric perspective. This perspective is absolutely critical and is missing from existing literature. Key Contributions: The goal of our study is to unearth practical and real-world challenges that are often encountered in real-world settings employing machine learning models. We conducted semi-structured interviews with 13 practitioners who are experienced with deploying ML models and engaging with customers spanning domains such as financial services, healthcare, hiring, online retail, computational advertising, and conversational assistants. We identi-fied various human-centric challenges and requirements for model monitoring in real-world applications. Specifically, we found that relevant stakeholders would want model monitoring systems to provide clear, unambiguous, and easy-tounderstand insights that are readily actionable. Furthermore, our study also revealed that stakeholders desire customization of model monitoring systems to cater to domain-specific use cases.

Related Work
This work lies at the intersection of several emerging areas of machine learning research, namely, detecting dataset shifts, monitoring model behavior via model understanding and explanations, monitoring fairness and robustness of machine learning models, and human-centered studies and open source tools focused on the aforementioned aspects. Below, we discuss some of the key works across each of the aforementioned areas.
Dataset Shifts There is a rich literature on techniques for detecting shifts in the data (e.g., see Breck et al. (2019); Cormode et al. (2021); Gama et al. (2014); Karnin, Lang, and Liberty (2016); Tsymbal (2004); Webb et al. (2016); Zliobaitė, Pechenizkiy, and Gama (2016) and the references therein). Both verifying the validity of model inputs and detecting changes in the features or model outputs are important challenges encountered in practical ML applications. The former is often addressed by including user-defined tests such as tests to check if a feature value is within a specified range (Schelter et al. 2018). For the latter, statistical hypothesis testing and confidence interval based approaches have been proposed. Statistical hypothesis testing involves checking if two given sets of samples are drawn from the same distribution by using a test statistic. Student's t-test and Kolmogorov-Smirnov test are examples of commonly used tests (Wasserman 2004;Murphy 2012). More advanced tests such as Maximum Mean Discrepancy can also be used for higher dimensional data (Gretton et al. 2012). As these tests require sufficient fine-tuning (e.g., selecting the kernel and its hyperparameters), confidence interval based approaches (Efron and Tibshirani 1994) are often employed for detecting drifts in practice (Nigenda et al. 2022). In addition to the above approaches which are model-agnostic, specialized methods that leverage the model internals and the training data have also been proposed to determine the extent of drift and take remedial steps (Garg et al. 2020;Lipton, Wang, and Smola 2018;Reddi, Poczos, and Smola 2015;Wu et al. 2019).
Interpretability As argued by several recent works (Doshi-Velez and Kim 2017), model understanding is absolutely critical to ensure that ML models are relying on appropriate features when making predictions. To this end, model interpretations and explanations are widely being used to monitor model behavior. Many approaches have been proposed to directly learn interpretable models for various tasks including classification (Letham et al. 2015;Wang and Rudin 2015;Lakkaraju, Bach, and Leskovec 2016;Lou, Caruana, and Gehrke 2012;Bien and Tibshirani 2009) and clustering (Kim, Rudin, and Shah 2014;. To this end, various classes of models such as decision trees, decision lists (Letham et al. 2015), decision sets (Lakkaraju, Bach, and Leskovec 2016), prototype (case) based models (Bien and Tibshirani 2009), and generalized additive models (Lou, Caruana, and Gehrke 2012;Caruana et al. 2015) were proposed. However, complex models such as deep neural networks and random forests are often shown to achieve higher accuracy than simpler interpretable models (Ribeiro, Singh, and Guestrin 2016); thus, there has been a lot of interest in constructing post hoc explanations to understand their behavior.
A variety of post hoc explanation techniques have been proposed, which differ in their access to the complex model (i.e., black box vs. access to internals), scope of approximation (e.g., global vs. local), search technique (e.g., perturbation-based vs. gradient-based), explanation families (e.g., linear vs. non-linear), etc. For instance, LIME (Ribeiro, Singh, and Guestrin 2016) and SHAP (Lundberg and Lee 2017), are model-agnostic, local explanation approaches that explain individual predictions of any black box model by training a linear model locally around each prediction. These approaches rely on input perturbations to learn these interpretable local approximations. Several other local explanation methods have been proposed that compute saliency maps which capture importance of each feature for an individual prediction by computing the gradient with respect to the input (Simonyan, Vedaldi, and Zisserman 2014;Sundararajan, Taly, and Yan 2017;Selvaraju et al. 2017;Smilkov et al. 2017). A number of other local explanation methods (Koh and Liang 2017;Ribeiro, Singh, and Guestrin 2018) have also been proposed in the literature. An alternate approach is to provide a global explanation summarizing the black box as a whole (Lakkaraju et al. 2019;Bastani, Kim, and Bastani 2017), typically using an interpretable model. Some recent work has shed light on the downsides of post hoc explanation techniques. Rudin (2019) argues that post hoc explanations are not reliable, as these explanations are not necessarily faithful to the underlying models and present correlations rather than information about the original computation. There has also been recent work on exploring vulnerabilities of black box explanations (Adebayo et al. 2018;Slack et al. 2020;Lakkaraju and Bastani 2020a;Rudin 2019;Dombrowski et al. 2019)-e.g., Ghorbani, Abid, and Zou (2019) demonstrated that post hoc explanations can be unstable, changing drastically even with small perturbations to inputs.
Fairness It is crucial to ensure that ML models deployed in real-world applications do not unduly discriminate against minority subgroups. To this end, monitoring the fairness of ML models has become common place in recent times (Dwork et al. 2012;Hardt, Price, and Srebro 2016). The initial literature on fairness in machine learning emphasized heavily on outlining the precise definitions of statistical fairness (Hardt, Price, and Srebro 2016). Several competing and contrasting notions of fairness emerged during this phase which can be broadly categorized into: 1) group fairness which emphasizes that protected groups should receive similar treatment as that of advantaged groups (Berk et al. 2018;Hardt, Price, and Srebro 2016) 2) individual fairness which requires that similar individuals to be treated similarly (Dwork et al. 2012), and 3) counterfactual fairness which captures the intuition that a decision pertaining to an individual is fair if it is the same in the actual world and a counterfactual world where the individual belonged to a different demographic group (Kusner et al. 2017). Furthermore, various metrics have been proposed to realize each of the aforementioned notions of fairness. For example, statistical (demographic) parity, equalized odds, equality of opportunity, and predictive parity are metrics proposed to enforce group fairness.
There are pros and cons to each of the aforementioned notions and metrics of fairness. For example, Dwork et al. (2012) argue that the group fairness notion of statistical parity leads to highly undesirable outcomes e.g., one might end up incarcerating women who pose no safety risk to ensure the same proportions of men and women are released. On the other hand, Kim, Reingold, and Rothblum (2018) highlight that assessing individual fairness is often hard in practice because it is hard to determine what is an appropriate metric function to measure the similarity of two individuals. Similarly, realizing counterfactual fairness in practice is also non-trivial because we do not have access to the counterfactuals of real world decisions i.e., there is no ground truth to determine if someone would have been incarcerated if that individual was a male instead of being a female and vice versa. Furthermore, prior research has also established that certain notions of fairness (calibration and balance conditions) are fundamentally incompatible and cannot be simultaneously optimized (Kleinberg, Mullainathan, and Raghavan 2017;Chouldechova 2017 (Hermann and Balso 2017). In contrast to these tools and techniques, the focus of this work is on understanding the needs, requirements, and challenges associated with monitoring deployed models from the perspective of relevant stakeholders.
Human-Centric Perspectives and User Studies There have been several user studies and interviews to understand the desiderata for model explanations and fairness (Doshi-Velez and Kim 2017; Cheng et al. 2021). For instance, (Bhatt et al. 2020) conducted interviews with data scientists to understand the use cases and accompanying desiderata for explaining models. On the other hand, Lakkaraju and Bastani (2020b) carried out a user study to understand if misleading explanations can fool domain experts into deploying racially biased models, while Kaur et al. (2020) found that explanations are often over-trusted and misused. Similarly, Poursabzi-Sangdeh et al. (2021) found that supposedlyinterpretable models can lead to a decreased ability to detect and correct model mistakes, possibly due to information overload. Lage et al. (2019) used insights from rigorous human-subject experiments to inform the design of explanation algorithms. While the above works touch upon humancentric perspectives on explainability and fairness, they do not focus on human-centric perspectives on model monitoring which is the key goal of our work.

Study Design
We collected desiderata data for model monitoring from 13 practitioners who are experienced with deploying ML models and engaging with customers spanning domains such as financial services, healthcare, hiring, online retail, computational advertising, and conversational assistants. We did so by conducting semi-structured and one-on-one interviews virtually and analyzed the results from each interviewee. All the interviewees had an understanding about ML model monitoring and had interacted with various model monitoring tools.
In the following, we discuss the experimental design choices, the data collection process and the data analysis approach for the study.

Data Collection
We collected desiderata from thirteen (n=13) ML practitioners with expertise in the model monitoring space. We recruited these participants by reaching out to ML practitioners who are either working on developing model monitoring tools or needing such tools for their application use cases.
Based on an average of 2.7 years of professional experience of the interviewees within the MLOps space, we considered four to be experts, four to be beginners, and five to be

Data Analysis
We analyzed the responses to all the questions using inductive content analysis (Krippendorff 1980). The interview notes were analyzed in two ways. First, each transcript was individually annotated for model monitoring application areas/use cases, requirements, and challenges described by each of the interviewees. Second, the responses to each question mentioned in Table 1 across all interviews were pooled and analyzed to inductively generate common themes and categories for ML monitoring application areas, design requirements, and challenges. Responses to IQ1 were identified either as a domain or a use case. For example, phrases and words such as "financial services", "insurance", "banking", and "adtech" were labeled as domains. Whereas, phrases such as "fraud detection", "credit lending", and "speech recognition" were labeled as a use case. Responses to IQ2, IQ3, and IQ4 were analyzed to identify human-centric desiderata for model monitoring. Sentences that contained phrases such as "need to", "should have", "be able to", and "it would be great if" were labeled as requirements. Phrases such as "difficulty", "challenges", "not possible", "hard", and "risky" were analyzed to identify challenges faced by the interviewees. Further, Responses to IQ3 were also analyzed to discover interviewees' interpretation of model monitoring and what it entails. To do so, authors' domain knowledge of model monitoring was leveraged to label the aspect of model monitoring discussed by an interviewee. For example, if interviewees discussed data drift as a part of the response to IQ3, data drift was labeled as an aspect of model monitoring for that interviewee. The IQ3 labels across all interviewees were pooled to characterize various aspects of model monitoring as discussed by them.

Application Areas, Design Considerations and Challenges for Model Monitoring: Practitioners' Perspectives
In this section, we discuss the use cases, requirements, and challenges discussed by the interviewees. We begin with the results of IQ3 analysis, where we describe various aspects of model monitoring. Then, we discuss the results of IQ1 analysis, namely, the application areas mentioned by the interviewees. Then, we discuss the interviewees' desiderata and challenges for model monitoring.

What is Model Monitoring? Practitioners' Perspective
We analyzed the responses to IQ3 contextually to identify the key aspects of model monitoring stated by the interviewees. All the interviewees discussed model performance monitoring and data drift monitoring as a part of model monitoring. One interviewee discussed, "we are interested in prediction drift as a proxy for model performance." Another interviewee mentioned, "Data scientists care more about model performance, deploying more models, etc.". Similarly, one interviewee highlights their need for model monitoring by mentioning, "we need tools for continuous model monitoring to assess end-to-end impact of these [design] changes and ensure model performance." Five (out of thirteen) interviewees emphasized monitoring model fairness, bias, and model versions as a part of model monitoring. One interviewee said, "A subset of our models needs fairness analysis to ensure compliance." Another interviewee mentioned their interaction with a banking client as follows: "If you are a risk manager for banks, then you probably care about fairness/bias." One interviewee discussed fairness monitoring as an aspect of model monitoring and mentioned, "Monitoring fairness could be another interesting addition for my clients who are at the receiving end of regulatory constraints, fines, etc." With respect to model versioning, an interviewee stated, "At [retracted organization name], we have a champion model and

Domain
Use Case AdTech Ads personalization and ads pricing. Consumer Technology Wake-word detection, automatic speech recognition, natural language understanding and interpretation, entity resolution, and text-to-speech generation. Financial Services Fraud detection, credit lending, and churn prediction. Insurance Risk prediction Retail Recommendation models and traffic monitoring. We also noticed interviewees categorizing or grading the relative importance of various aspects of model monitoring as a part of the response to IQ3. Interviewees largely identify monitoring data integrity (Boritz 2005), that is, the accuracy, completeness, and consistency of data for inputs and outputs of a model as a basic requirement. One interviewee remarked, "I think, at the very least, [we need] a system that monitors the exhaust of the model; the outputs and also the inputs of the model." In addition to this, identification of data drift, performance drift, and outlier detection were mentioned as important but intermediate requirements for model monitoring. Another interviewee stated, "On top of basic input output monitoring, another aspect is model performance from accuracy and precision point of view." Finally, monitoring model fairness and bias were expressed as regulation driven, "good to have", and are thus labeled as advanced requirements for model monitoring.

Application Areas and Use Cases
Analyzing responses to IQ1 enabled us to identify specific domains and use cases of interest to the interviewees as shown in Table 2. Some example excerpts include: "of course, classic are fraud, churn use cases, and recommendation for anything [in] retail." Another interviewee states, "our primary market is financial services and fintech, we have some retail, but insurance is clearly a new area where we are seeing new interest." Regarding the financial services domain, one interviewee specifically mentioned that "they're [financial services domain] the most advanced with the use of machine learning or they have the highest bar of regulatory scrutiny." In the context of monitoring for online advertisement, one interviewee said, "if by mistake we advertise to the wrong user, and by mistake, we bid with the wrong price, that will affect the end results, the revenues or the profit." While there are numerous other domains and use cases for ML models, the results here are intended to contextualize the responses of the interviewees to the desiderata and the challenges discussed below.

Human-centric Requirements for Model Monitoring
The following themes for model monitoring requirements emerged based on interviewees' responses to IQ2, IQ3, and IQ4.

Domain-Specific Debugging & Root Cause Analysis:
Interviewees discussed the need for a model monitoring system to discover slices or sub-populations of data where unexpected model behavior and outcomes occur. This would help one gain insight on model errors, when to retrain a model, and domain-specific nuances. One interviewee mentioned, "If it can provide me with some early warnings or signs where things go wrong and give me ways to resolve what's going on." Further, the system should allow customizable levels of abstractions such as feature-level monitoring, prediction-level monitoring, and performance-level monitoring based on the use case. One interviewee stated, "the thing that would really help is customizing monitoring by allowing different overlays of time scale, business metrics, model metrics, etc.". Risk Management, Model Governance, and Privacy: Interviewees would like model monitoring systems to help them manage risk and ensure regulatory compliance. Interviewees emphasized the need for a monitoring system to enable centralization of model governance in an organization rather than have dependencies on an individual or a team that created the model. One of the interviewees cited the work of (Kurshan, Shen, and Chen 2020) in discussing the challenges of model governance and mentioned the need for frameworks with self-regulatory capabilities. Such a system was also discussed to require to reduce human dependency and automate the process of error detection in ML pipelines. One of the interviewees remarked, "In the past, people would manually go [look for errors] and document [the errors] but now we are trying to automate by monitoring." Interviewees discussed the risk associated with retraining the model without monitoring. One interviewee said, "[without monitoring] model refresh can be too risky. How do we ensure the new model is working as intended?" In certain settings, it would also be desirable for the monitoring system to ensure that privacy and confidentiality of various assets such as protected user information in training data and intellectual property associated with the models are protected. One of the interviewees mentioned, "that's [privacy] keeping people from not sending the data to a hosted environment; they want to keep it in their own private VPC and that adds a lot of challenges for monitoring [as a service]." Human-Centered Design: Interviewees discussed the need to preserve human autonomy and decision-making. In other words, the monitoring system should not trigger actions such as automated retraining, but instead provide actionable insights and suggestions to enable humans to make better decisions. Further, the monitoring system should provide relevant and meaningful alerts without cognitive overload. One interviewee discussed, "For individual data scientists, I think the process [of root cause analysis] becomes cumbersome or if they are trying to share information, it becomes untenable when the number of models grows." In the context of emotions, one interviewee stated, "you know if somebody wakes me up at 12 o'clock in the night saying 'Oh, there is a drift and take a look,' and if there is no drift, I am going to be mad." Another interviewee said, "An ideal monitoring system should not create alert fatigue." To avoid this, the model monitoring system needs to be aware of aspects such as the types of alerts, how often they are fired, and how they are presented. The challenge, however, as discussed by one interviewee is that, "we may not have the right metrics that measure human [centered] things such as individual preferences and fatigue."

Challenges for Model Monitoring: Temporal Categorization
We temporally categorized challenges as (1) design challenges before deploying a monitoring system, (2) challenges during monitoring an ML system, and (3) challenges post deployment with respect to the usefulness of monitoring outcomes.
Interviewees highlighted several practical challenges in designing and deploying an ML model monitoring system. These challenges included design questions such as what should be monitored, and how should the monitoring system interact with the model. One interviewee discussed, "sometimes you have models that translate from one layer to another. So monitoring a feature XYZ may not make sense to the customer." Another interviewee discussed that for monitoring systems as a service, "when it comes to actually doing things, there are a lot of other constraints that become more important like people [clients] might not talk about [their use case] or might not explicitly tell you so." These point to the requirement of a monitoring system having the ability to provide some guidelines or in-built options for what to monitor in an ML system.
Interviewees also discussed whether the monitoring system would also need the technical dependencies that a model requires, such as packages and modules, to ingest a model and execute successfully. One interviewee discussed, "let's say an organization uses a package but there I'm not quite sure what the business model is, can anybody go and download that and set that up and run that on their end? I'm not sure." Thus, monitoring systems are discussed as systems that have the ability to take an ML system as an input and provide monitoring insights as an output. In that context, interviewees raise the challenge for an ML monitoring system to possess a super-set of the technical capabilities of different ML systems. We note that this perceived challenge by the interviewees in practice is only a concern if the ML model is required to run specific predictions on the monitoring platform. We discuss this finding in Section 5.
Some interviewees had prior experience with model monitoring systems, and discussed the challenges of protecting privacy of users in training data and confidential information / intellectual property associated with their models. One interviewee said, "For third party monitoring tools offered as SaaS [Software-as-a-Service], the clients have to send the data [to the third party server] -that becomes a privacy challenge." All interviewees emphasized the challenges of adapting a monitoring system to the domain-specific needs of the ML system and the lack of solutions that cater to their specific needs. For example, the data volume experienced by an ML system is highly sensitive to its application context, and could become a key design consideration for an ML monitoring system to be able to handle. Such decisions are currently made manually, and there is a lack of a framework that helps automate the design process for an ML monitoring system. Hence, interviewees highlighted the need for a fully managed monitoring service for their ML systems.
We also observed that the interviewees discussed challenges that may occur when an ML monitoring system is deployed. They discussed the lack of existing reliable solutions in assessing whether an observed drift in data or model performance is a cause for concern. One interviewee mentioned "products have some sort of seasonality or recurring drift that is probably not very interesting for the developers to get alerted for. For example, when I was at [organization name] we would see usage peak on weekends vs. weekdays because most people would open the app on the weekends. If [organization name] were to use a [model] monitoring system, how do we identify drift that is anomalous vs. simply seasonal [or periodic]?" Such a lack of reliability can result in cognitive fatigue that may desensitize practitioners from gaining meaningful insights from the monitoring system. Interviewees also discussed latency challenges, that is, how quickly can a monitoring system detect issues and suggest remedial actions. Here, interviewees discuss the need for the monitoring system to inform the human user about the computation time of various aspects of monitoring that may not necessarily be computed on similar timescale. One interviewee stated, "I can do an F1 score in real time and that's a simple thing to be able to do, given a particular classification threshold but if I want to do AUC which is independent of the classification threshold, all of a sudden that's a very different computation. So it's very interesting that people know the formula, but they expect the same behavior for both which is not theoretically possible." Systems that are slow to identify issues may not be useful in certain contexts, such as autonomous driving, where high-stakes decisions often need to be made in real-time by ML models. Further, due to privacy, intellectual property, and security/compliance considerations, interviewees highlighted that the monitoring tools and systems may need to be present "onpremise" or in-situ rather than a third party housing such a system. Consequently, debugging or maintaining the monitoring system itself may be challenging, especially if the services are being sought from a third party.
Interviewees also discussed the "so what" or value-based challenges with monitoring systems. These discussions focused on the value of the insights gained from such systems, and pertained to aspects such as whether/how the monitoring insights could enable the stakeholders to take concrete actions to improve business outcomes and whether non-experts would be able to understand these insights. As an example, one interviewee quoted, "developers need to know how the model affects business metrics, and monitoring is important for that." 4.5 Challenges for Model Monitoring: Feature-specific Categorization As noted earlier, data drift, outlier detection, data integrity violation, model performance, and bias/fairness are the key dimensions of model monitoring highlighted by the interviewees. Next, we discuss the specific desiderata themes within some of these dimensions. For data drift, we found that all the interviewees consider input and output data distribution monitoring as a necessity for model monitoring. Also, such drift monitoring is treated as an indicator for model retraining. Further, data integrity violation and outlier detection are aspects that are discussed as a part of the data drift monitoring functionality as well. One interviewee said, "customers want simple stuff like data drift, performance monitoring, and some sort of anomaly detection. That is the first set of things they want." Another interviewee mentioned, "customers are very worried about data integrity issues such as a breakdown in their pipeline they want to know immediately. A sudden change in data." We note here that the "customers" referred to by the interviewees are stakeholders interacting with these interviewees in professional settings who have expressed interest in using model monitoring systems as a service.
For bias/fairness monitoring, customer requirements are currently driven through policies and regulations. One interviewee stated, "Fairness is rare so far in my experience but with big banks, regulation and compliance are driving it." Another interviewee said, "There is a smaller subset of cases for bias and fairness monitoring." Words such as "policy","regulatory constraints","fines", and "regulatory push" were used to describe the need for fairness monitoring. Practitioners mentioned that compliance and risk management teams are concerned about bias/fairness also due to its impact on the trust and the reputation of a company. By leveraging ML model monitoring, an organization could proactively detect and mitigate any biases observed in its deployed models instead of having to react when such issues are discovered by external entities.

Conclusion and Discussion
Motivated by the need for understanding the humancentered requirements and challenges in designing monitoring frameworks for ML systems, we performed an interview study with ML practitioners with experience spanning several application domains. We presented findings and insights on real-world use cases, desiderata, and challenges for ML model monitoring in practice based on these interviews. Interviewees discussed both feature-specific and processspecific aspects of model monitoring. Feature-specific aspects include monitoring data drift, model performance, and bias/fairness and ensuring that the alerts are relevant without cognitive overload. Process-specific aspects include the temporal considerations before, during, and after the deployment of the model monitoring system and the ability of a monitoring system to cater to different needs across the lifecycle of an ML system.
Based on the requirements and challenges reported in this work, we discuss potential pathways to address concerns on themes of human-centric design of model monitoring systems. We believe that human autonomy and agency can be preserved using a human-AI decision making framework, wherein the model monitoring system is used in conjunction with a human decision maker. This implies enabling the system to report data integrity violations or different types of drift, and then allowing human users to pursue corrective pathways such as correction of labels or features, data re-sampling, and model retraining. To avoid cognitive load, monitoring systems could include threshold knobs and preference logging features that enable humans to embed domain-specific knowledge for their specific use cases. For centralization of model governance, the model monitoring system could enable automatic report generation and use of natural language to describe the state of an ML system. This would enable non-experts to also have an understanding of the state of an ML system. To address latency concerns, while there may be technological advancements to improve computation speed as well as efficient designs of a system, we also highlight the need for the system to educate the human user about the time estimates for certain computations as well as a comparative view of the difference in compute time for different aspects of a system. Such knowledge can potentially improve user experience while leveraging model monitoring systems.
We highlight a perceived challenge regarding the technical requirements of a model monitoring system. Interviewees described the challenge for an ML monitoring system to possess a super-set of the technical capabilities of different ML systems it monitors. However, we note that to monitor inputs, outputs, and other characteristics associated with a deployed model, a monitoring system does not need to execute the model and can instead take the production logs associated with the model as input. Thus, the model monitoring system may not need the technical infrastructure to run the model itself, and can instead focus on the tools and techniques for prediction of distribution shifts. This understanding influences the design decisions for monitoring systems and we thus highlight that model monitoring systems may be designed in a model-agnostic manner. This also points to the caution required in analyzing human-centric desiderata where perceived challenges by practitioners, who may not necessarily be experts in ML, may stem from misconceptions about the system functionalities.
We also note that inferring a user's cognition is currently an active area of research (Akula et al. 2022;Shergadwala, Panchal, and Bilionis 2022;Wu et al. 2022). While some human-centric requirements may point to the user's desire to be "understood" by a system in real-time, it is currently not practically feasible to so. Further, several studies have highlighted the limits of human oversight and the challenges that arise when attempting to build tools that enable people to monitor technological systems (e.g., see Perrow (1999) and Green (2022)). Thus, we need to be aware of the limits in the ability to build model monitoring tools and interfaces that actually satisfy the human-centric requirements stated by the interviewees.
We acknowledge that our analysis is limited based on the inputs of only thirteen practitioners. However, the intervie-wees are ML practitioners with deep knowledge of ML systems in their respective domains and extensive experience of ML model monitoring systems. Hence, we were able to obtain and analyze human-centric requirements and challenges from a practical viewpoint based on our interviews. More broadly, we encourage MLOps practices to formalize design frameworks for ML monitoring systems that are cautiously informed by human-centered desiderata.


extfieldsofstudy:
Computer Science
Engineering

text:
BVI-DVC: A Training Database for Deep Video Compression

Deep learning methods are increasingly being applied in the optimisation of video compression algorithms and can achieve significantly enhanced coding gains, compared to conventional approaches. Such approaches often employ Convolutional Neural Networks (CNNs) which are trained on databases with relatively limited content coverage. In this paper, a new extensive and representative video database, BVI-DVC, is presented for training CNN-based video compression systems, with specific emphasis on machine learning tools that enhance conventional coding architectures, including spatial resolution and bit depth up-sampling, post-processing and in-loop filtering. BVI-DVC contains 800 sequences at various spatial resolutions from 270p to 2160p and has been evaluated on ten existing network architectures for four different coding tools. Experimental results show that this database produces significant improvements in terms of coding gains over three existing (commonly used) image/video training databases under the same training and evaluation configurations. The overall additional coding improvements by using the proposed database for all tested coding modules and CNN architectures are up to 10.3% based on the assessment of PSNR and 8.1% based on VMAF.

I. INTRODUCTION
F ROM the introduction of the first international standard in 1984, video compression has played an essential role in the application and uptake of video technologies across film, television, terrestrial and satellite transmission, surveillance and particularly Internet video [1]. Inspired by recent breakthroughs in AI technology, deep learning methods such as Convolutional Neural Networks (CNNs) have been increasingly exploited into video coding algorithms [2] to provide significant coding gains compared to conventional approaches based on classic signal/image processing theory.
It is noted that these learning-based compression approaches demand volumes of training material much greater than typically used for conventional compression or existing machine learning methods. These should include diverse content covering different formats and video texture types. As far as we are aware, no such public data sets for this purpose currently exist, and most learning-based coding methods [3] have been currently trained on image or video databases, which were mainly designed for computer vision applications, e.g. superresolution. Most of these databases do not provide sufficient D. Ma, F. Zhang and D. R. Bull are with the Department of Electrical and Electronic Engineering, University of Bristol, Bristol, BS8 1UB, UK (e-mail: di.ma@bristol.ac.uk; fan.zhang@bristol.ac.uk; dave.bull@bristol.ac.uk). 1 The BVI-DVC database can be downloaded from: https://vilab.blogs. bristol.ac.uk/?p=2375. content coverage and diversity. As a result, the generalisation of networks cannot be ensured in the context of video coding, and the optimum performance of employed CNN networks has not been achieved when trained on these databases.
In this paper, a new video database, referred as BVI-DVC, is proposed for training CNN-based video coding algorithms, in particular those tools that enhance the performance of conventional compression algorithms. These include spatial resolution re-sampling, bit depth re-sampling, post-processing and in-loop filtering, all of which have achieved significant coding gains compared to other deep learning enhancements. BVI-DVC contains 800 progressive-scanned video clips at a wide range of spatial resolutions from 270p to 2160p, with diverse and representative content. To demonstrate its training effectiveness, compared to three commonly used training databases [4][5][6], BVI-DVC has been utilised to train ten popular CNN architectures [7][8][9][10][11][12][13][14][15][16] using four video coding tools: post-processing (PP), in-loop filtering (ILF), spatial resolution adaptation (SRA) and effective bit depth adaptation (EBDA). The resulting CNN-based coding tools were then integrated into the Test Model (HM 16.20) of the current High Efficiency Video Coding (HEVC) standard [17], and evaluated based on the Joint Video Experts Team (JVET) Common Test Conditions (CTC) [18].
The primary contributions of this paper are: 1) A publicly available video database (BVI-DVC) specifically developed for training deep video coding compression algorithms. 2) An analysis of the performance of BVI-DVC in the context of four compression tools compared with three existing (commonly used) databases. 3) A performance comparison of ten popular CNN architectures based on identical training materials and evaluation configurations. The remainder of the paper is structured as follows. Section II summarises the related work in deep learning-based video compression and commonly used image/video training databases. Section III introduces the proposed (BVI-DVC) video training database, while Section IV presents the experimental configurations employed for evaluating the effectiveness of the proposed database. The results and discussions are provided in Section V, and Section VI concludes the paper and presents some suggestions for future work.
II. BACKGROUND With increasing spatial resolutions, higher frame rates, greater dynamic range and the requirement for multiple viewpoints, accompanied by dramatic increases in user numbers, video content has become the primary driver for increased internet bandwidth. How we represent and communicate video (via compression) is key in ensuring that the content is delivered at an appropriate quality, while maintaining compatibility with the transmission bandwidth.
To address this issue, ISO (MPEG) and ITU initiated the development of a new coding standard, Versatile Video Coding (VVC) [19] in 2018, targeting increased coding gain (by 30-50%) compared to the current HEVC standard. Concurrently, the Alliance for Open Media (AOM -formed in 2015) finalised an open-source, royalty-free media delivery solution (AV1) [20] in 2018, offering performance competitive with HEVC [21].

A. Machine learning based compression
Image and video compression based on deep neural networks has become a popular research topic offering evident enhancements over conventional coding tools for: intra prediction [22,23], motion estimation [24,25], transforms [26,27], quantisation [28], entropy coding [29,30], post-processing [31,32] and loop filtering [33,34]. New coding tools such as format adaptation [35][36][37] and virtual reference frame optimisation [25] have also been reported. Other work has implemented a complete coding framework based on neural networks using end-to-end training and optimisation [38][39][40][41][42]. It is noted that, despite their performance benefits, few of these approaches are currently being adopted by the latest VVC video coding standard. This is due to the high computational complexity and the large GPU memory requirements associated with CNN computation.

B. Training Databases
Training databases are a critical component for optimising the performance of machine learning based algorithms. A well designed training database can ensure good model generalisation and avoid potential over-fitting problems [43,44]. As far as we are aware, there is no publicly available database which is specifically designed for learning-based video coding. Researchers, to date, have typically employed training databases developed for other purposes (such as super-resolution, frame interpolation and classification) for training. Notable publicly available image and video training databases are summarised below.
• ImageNet [45] is a large image database primarily designed for visual object recognition. It contains more than 14 million RGB images at various spatial resolutions (up to 2848p) covering a wide range of natural content. It has also been used as a training database for single image super-resolution [12]. • DIV2K [4] contains 1000 RGB source images with a variety of content types, which was firstly developed for super-resolution. It has currently been employed as training material by several JVET proposals [46,47] and many other CNN-based coding algorithms [48,49]. • BSDS [50] is an image database originally developed for image segmentation. It contains 500 RGB images, and has been used to train CNN-based loop filters [33] for video coding. Comparing to DIV2K, BSDS has fewer source images and lower spatial resolution (481×321). • Vimeo [51] is a video database originally developed for training CNN-based optical flow and temporal superresolution approaches. It contains 89,800 sequences at spatial resolutions up to 448×256. A constraint is imposed on motion vector magnitudes between any two adjacent frames and content with dynamic textures has not been included in this database. Vimeo has not been frequently employed for deep learning based coding approaches and in particular has not been used for those approaches that exhibit superior improvements over standard video codecs (e.g. HEVC and VVC) [3]. • CD (Combined Database) in [6] is a video database combining source content from the LIVE Video Quality Assessment Database [52], MCL-V Database [53] and TUM 1080p Database [54] and has been employed to train CNN-based super-resolution approaches [6]. It contains 29 sequences at two different spatial resolutions, 1920×1080 and 768×432. • REDS [5] is a video database developed for training video super-resolution algorithms [55], which contains 300 video clips with spatial resolution 1280×720. • UCF101 [56] is a large video training database initially designed for human action recognition, and has been frequently used for training CNN-based temporal frame interpolation and motion prediction approaches [57][58][59]. It contains 13320 videos collected from YouTube, which consist of 101 types of human actions. All the sequences in UCF-101 have a relatively low spatial resolution of 320×240. Modern video coding algorithms are required to process content with diverse texture types at high spatial resolutions and bit depths. For example, the standard test sequences included in the JVET Common Test Conditions (CTC) dataset include video clips at UHD resolution (2160p) at a bit depth of 10 bits, with various static and dynamic textures 2 . However none of the training databases mentioned above contain image or video content with high spatial resolution and bit depth, and most do not include any dynamic texture content.  [72], 3 from MCL-V [53], 2 from MCL-JCV [73], 2 from Netflix Chimera [74], 1 from the TUM HD databases [75], and 1 from the Ultra  Video Group-Tampere University database [76]. These sequences contain natural scenes and objects [5], e.g. mountains, oceans, animals, grass, trees, countryside, city streets, towns, buildings, institutes, facilities, parks, marketplaces, historical places, vehicles and colorful textured fabrics. Different texture types such as static texture, dynamic texture 2 , structure content and luminance-plain content are also included.

III. THE BVI-DVC VIDEO TRAINING DATABASE
All these sequences are progressive-scanned at a spatial resolution of 3840×2160, with frame rates ranging from 24 fps to 120 fps, a bit depth of 10 bit, and in YCbCr 4:2:0 format. All are truncated to 64 frames without scene cuts, using the segmentation method described in [77]. To further increase data diversity and provide data augmentation, the 200 video clips were spatially down-sample to 1920×1080, 960×540 and 480×270 using a Lanczos filter of order 3. This results in 800 sequences at four different resolutions. Fig. 1 shows the sample frames of twenty example sequences. The primary features of this database are summarised in Table I  in Fig. 2. The definitions of these feature can be found in [67,78]. It can be noted that the BVI-DVC database has a relatively wide coverage for these three video features, which indicates the diversity of the proposed database.

IV. EXPERIMENTS
In order to evaluate the training effectiveness of the proposed BVI-DVC database in the context of video compression, ten network architectures [7][8][9][10][11][12][13][14][15][16]79] were employed in conjunction with four CNN-based coding modules: post processing (PP), in-loop filtering (ILF), spatial resolution adaptation (SRA) and effective bit depth adaptation (EBDA). These four coding modules were selected since they have been demonstrated to offer significant coding gains over standardised video codecs compared to other tools (e.g. inter prediction, entropy coding, etc.) and also outperform existing end-to-end solutions. Compared to end-to-end image coding architectures, they are also more amenable to integration into standard codecs for practical applications.
In terms of benchmarking databases, based on the limited time and resource available, we have chosen one image database (DIV2K) and two video databases (REDS and CD) to compare with BVI-DVC. The DIV2K is selected because it has been used for training CNN models in multiple JVET contributions and many other CNN-based coding algorithms. Comparing to other video databases such as BSDS, Vimeo and UCF101, REDS and CD contain relatively higher resolution content and have been employed for training successful CNNbased super-resolution approaches.
A. Coding Modules 1) Coding Module 1 (Post Processing -PP): The coding workflow for post processing (PP) is illustrated in Fig. 3. PP is commonly applied at the decoder, on the reconstructed video frames, to reduce compression artefacts and enhance video quality. When a CNN-based approach is employed, the network takes each decoded frame as input and outputs the final reconstructed frame with the same format. Notable examples of employing CNN-based post processing for video coding can be found in [31,32]. 2) Coding Module 2 (In-loop Filtering -ILF): In-loop filtering applies processing at both the encoder and the decoder on the reconstructed frames, and the output can be used as reference for further encoding/decoding. An encoder architecture with a CNN-based ILF module is shown in Fig. 4. The input and the output of the CNN-based ILF are the same as those for PP [33,80].
3) Coding Module 3 (Spatial Resolution Adaptation -SRA): CNN-based spatial resolution adaptation (SRA) down-samples the spatial resolution of the original video frames for encoding, and reconstructs the full resolution during decoding through CNN-based super-resolution. This approach can be applied at Coding Tree Unit level [35] or to the whole frame. Here we only implemented frame-level SRA [81], as shown in Fig.  5. In this case, the original video frames are spatially downsampled by a fixed factor of 2, using the Lanczos3 filter. The CNN-based super-resolution module processes the compressed and down-sampled video frames at the decoder to generate full resolution reconstructed frames. It is noted that a nearest neighbour filter is firstly applied to the reconstructed downsampled video frame before CNN operation [36].

4) Coding Module 4 (Effective Bit Depth Adaptation -EBDA):
Similar to the case for spatial resolution, bit depth can also be adapted during encoding in order to achieve improved coding efficiency. Here Effective Bit Depth (EBD) is defined as the actual bit depth used to represent the video content, which may be different from the Coding Bit Depth (CBD) that represents the pixel bit depth, e.g. InternalBitDepth in HEVC reference encoders. This process is demonstrated in Fig. 6. In this paper, we have fixed CBD at 10 bits, the same as in the Main10 profile of HEVC, and only down-sampled the original frames by 1 bit through bit-shifting. At the decoder, the bit depth of decoded frames is up-sampled to 10 bits using a CNN-based approach. More information about EBDA can be found in [82].

B. Employed CNN Models
For these four coding modules, ten popular network architectures have been implemented for evaluation. These include five with residual blocks, two with residual dense blocks, This network structure has been employed in several CNN-based coding algorithms and has been reported to offer significant coding gains [36].
In this experiment, we have employed identical architectures for these ten networks, as reported in their original publications, and only modify the input and output interfaces in order to process content in the appropriate format. The input of all CNNs employed is a 96×96 YCbCr 4:4:4 colour image, while the output targets the corresponding original image block with the same size. The input block can be either compressed (for PP and ILF), compressed and EBD downsampled (for EBDA) or compressed and spatial resolution re-sampled (for SRA -a nearest neighbour filter is applied before CNN processing). The same loss functions have been used as in the corresponding literature. All these ten networks have been re-implemented using the TensorFlow framework (version 1.8.0).

C. Training Data
Three existing image and video databases are selected to benchmark the training effectiveness of BVI-DVC, including DIV2K [4], REDS [5] and CD [6]. All the original images or videos in each database were first spatially down-sampled by a factor of 2 using a Lanczos3 filter or down-sampled by 1 bit through bit-shifting. The original content (for training PP and ILF CNNs), together with spatially down-sampled clips (for training SRA CNNs) and bit depth reduced sequences (for training EBDA CNNs) were then compressed by the HEVC Test Model (HM 16.20) based on the JVET Common Test Conditions (CTC) [18] using the Random Access configuration (Main10 profile) with four base QP (quantisation parameter) values: 22, 27, 32 and 37 3 (a fixed QP offset of -6 is applied for both spatially and bit depth down-sampled cases as in [85]). This results in three training input content groups for every database, each of which contains four QP sub-groups. For the input content group with reduced spatial resolution, a nearest neighbour filter was applied to obtain video frames with the same size as the original content.
For each input group and QP sub-group, the video frames of all reconstructed sequences and their original counterparts were randomly selected (with the same spatial and temporal sampling rates) and split into 96×96 image blocks, which were then converted to YCbCr 4:4:4 format. Block rotation was also applied here for data augmentation.

D. Network Training and Evaluation
The training process was conducted using the following parameters: Adam optimisation [86] with the following hyperparameters: β 1 =0.9 and β 2 =0.999; batch size of 16; 200 training epochs; learning rate (0.0001); weight decay of 0.1 for every 100 epochs. This generates 480 CNN models for 4 training databases, 3 input content groups (PP and ILF use the same CNN models), 4 QP sub-groups and 10 tested network architectures.
During the evaluation stage, for a specific coding module, the decoded video frames (already up-sampled to the same spatial resolution if needed) are firstly segmented into 96×96 overlapping blocks with an overlap size of 4 pixels as CNN input (YCbCr 4:4:4 conversion). The output blocks are then aggregated following the same pattern and then converted to YCbCr 4:2:0 format to form the final reconstructed frame.

V. RESULTS AND DISCUSSIONS
Four different coding modules have been integrated into the HEVC (HM 16.20) reference software, and have been fully tested under JVET CTC [18] using the Random Access configuration (Main10 profile). Nineteen JVET-CTC SDR (standard dynamic range) video sequences from resolution classes A1, A2, B, C and D were employed as test content, none of which were included in any of the three training databases. It is noted that only class A1 and A2 (2160p) were used to evaluate SRA coding module, as it has been previously reported [36] that for lower resolutions SRA may provide limited and inconsistent coding gains.
The rate quality performance (coding performance) is benchmarked against the original HEVC HM 16.20, using Bjøntegaard Delta measurement (BD-rate) [87] based on two quality metrics, Peak Signal-to-Noise Ratio (PSNR, Yluminance channel only) and Video Multimethod Assessment Fusion (VMAF, version 0.6.1) [88]. PSNR is the most commonly used assessment method for video compression, while VMAF is a machine learning based quality metric commercially used in streaming. VMAF combines multiple quality metrics and video features using a Support Vector Machine (SVM) regressor. It has been reported to offer better correlation with subjective opinions [89]. BD-rate statistics indicate the overall bitrate savings across the tested QP range achieved by the test algorithm for the same video quality compared to the anchor approach. The average BD-rate values are reported in Table II-V for each evaluated training database, network architecture and coding module.

A. Comparison of Databases
It can be observed from Table II-V and Fig. 7 that, for all tested network architectures and coding modules, the coding gains (in terms of average BD-rates for all tested sequences) achieved after training on the proposed BVI-DVC database are significantly greater than for the other three benchmark databases (DIV2K, REDS and CD) for both PSNR and VMAF quality metrics. This is reinforced by considering the mean (among ten networks and four coding modules) of all the average BD-rates for each database; Fig. 8 shows in excess of 4.2% and 5.4% additional bitrate savings obtained by using BVI-DVC compared to the other three databases based on the PSNR and VMAF quality metrics respectively. CD offers the worse overall performance, especially for results based on the assessment of PSNR.

B. Comparison of Networks
We can also compare the ten evaluated network architectures under fair configurations (identical training and evaluation databases). The results in Table II-V are summarised, by taking the mean (among four training databases and four coding modules) of average BD-rate values for each network architecture, in Fig. 9. It can be observed that RCAN, RDN, ESRResNet and MSRResNet offer better coding performance (for both PSNR and VMAF) than the other six evaluated network architectures. This is likely to be because of the residual block structure employed. The coding gains for VDSR, FSRCNN and SRCNN are relatively low comparing to other networks, exhibiting coding loss when PSNR is used to assess video quality especially when they are trained on the CD database (refer to Fig. 7). This may be due to their simple network architecture (FSRCNN and SRCNN) and the large number convolutional layers without residual learning structure (VDSR), which lead to less stable training and evaluation [14,15].

VI. CONCLUSION
This paper presents a new database (BVI-DVC) specifically designed for training CNN-based video compression algorithms. With carefully selected sequences including diverse content, this database offers significantly improved training effectiveness compared to other commonly used image and video training databases. It has been evaluated for four different coding modules with ten typical CNN architectures. The BVI-DVC database reported is available online 1 for public testing. Its content diversity makes it a reliable training database, not just for CNN-based compression approaches, but also for other computer vision and image processing related tasks, such as image/video de-noising, video frame interpolation and super-resolution. Future work should focus on developing large training databases with more immersive video formats, including higher dynamic range, higher frame rates and higher spatial resolutions.