extfieldsofstudy:
Computer Science
Mathematics
Engineering

s2fieldsofstudy:
Computer Science
Engineering

text:
Safe Optimization of an Industrial Refrigeration Process Using an Adaptive and Explorative Framework

Many industrial applications rely on real-time optimization to improve key performance indicators. In the case of unknown process characteristics, real-time optimization becomes challenging, particularly for the satisfaction of safety constraints. In this paper, we demonstrate the application of an adaptive and explorative real-time optimization framework to an industrial refrigeration process, where we learn the process characteristics through changes in process control targets and through exploration to satisfy safety constraints. We quantify the uncertainty in unknown compressor characteristics of the refrigeration plant by using Gaussian processes and incorporate this uncertainty into the objective function of the real-time optimization problem as a weighted cost term. We adaptively control the weight of this term to drive exploration. The results of our simulation experiments indicate the proposed approach can help to increase the energy efficiency of the considered refrigeration process, closely approximating the performance of a solution that has complete information about the compressor performance characteristics.

INTRODUCTION
Many industrial applications use real-time optimization based on characteristics of the underlying processes to improve key performance indicators, such as energy consumption. However, the characteristics may be unknown, or may change with time, for instance due to wear and tear. Thus, model-based real-time optimization is challenging. A common approach is to first estimate the unknown characteristics by collecting observations from multiple operating points. The operating points are explored, for example, by doing plant step tests. These observations generally bring noise to the optimization problem, and a real-time optimization algorithm needs to sequentially estimate and optimize the utility function by utilizing these noisy observations to reveal the decisions leading to the lowest cost.
Considering the cost of exploration such as not delivering the expected demand from the system during exploration, there is a trade-off between exploring more and having more accurate estimations or exploiting collected observations and optimizing the obtained function. This trade-off is well-studied in the literature and multi-armed bandit approaches are suggested as a solution for optimally balancing exploration and exploitation (Bubeck et al., 2012). The first appearance of using upper confidence bounds (UCB) is dated back to Lai and Robbins (1985). After that, many UCB algorithms are developed by researchers for stochastic bandits with many arms (Lattimore and Szepesvári, 2020). Most efficient bandit algorithms are characterized by having a utility function under certain regularity conditions (Dani et al., 2008). Srinivas et al. (2010) divided the stochastic optimization problem into two objectives as unknown function estimations from noisy observations and optimization of the estimated function over the decision set. To satisfy regularity conditions, they leverage the kernel methods and Gaussian process (GP) to model utility function through the regularity assumptions of kernels (Rasmussen, 2004).
If we want to approximate the unknown characteristics and optimize a process in a safety-critical operation, we need to utilize safe exploration algorithms which allow only the exploration of feasible decision points by enforcing safety constraints. In this work, we define a feasible decision point in the given decision set as satisfying all the constraints of the given problem. Bandit setting and Markov Decision Processes (MDPs) are used to formalize the safe exploration problem in the literature (Schreiter et al., 2015;Turchetta et al., 2019). Sui et al. (2015) have introduced the SafeOpt algorithm which optimizes an unknown function over a safe decision set notion under certain assumptions by utilizing the confidence bound construction method of Srinivas et al. (2010). Following a similar approach we recently proposed the ARTEO algorithm (Korkmaz et al., 2022) to cast multi-armed bandits as a mathematical programming problem subject to safety constraints for the optimization of safety-critical systems.
In ARTEO, we model the unknown characteristics of the system using Gaussian processes (GP) and utilize the covariance function of GPs to quantify uncertainty to ensure satisfaction of the safety constraints with high probability. We establish the safety of ARTEO by constructing confidence bounds as Srinivas et al. (2010). We incorporate the uncertainty into the utility function as a contribution to encourage the exploration at points with high uncertainty while continuing to satisfy the optimization goals of the system. The size of this contribution is adaptively controlled by a hyperparameter in accordance with the requirements of the environment.
In this paper, we demonstrate the implementation of AR-TEO for the power management problem of an industrial refrigeration process. The results show that we are able to successfully control three key performance indicators of the process by applying ARTEO. We further develop different demand scenarios to investigate the potential and limitations of ARTEO such as the case when the safety condition is violated, how hyperparameter selection affects the performance of the algorithm, and how ARTEO performs compared to a benchmark optimization problem with fully known system characteristics.

PROBLEM STATEMENT AND BACKGROUND
We want to minimize a utility function f that models the cost of decisions and the uncertainty in the system subject to K safety constraints g k , k = 1, . . . , K. The cost of a decision and the constraints at the chosen point x t ∈ D at each solution iteration t depends on the unknown system characteristics p t ∈ P , and D i ⊂ R defines the decision set of arm i . We want to find a sequence of decisions, x 1 , x 2 , . . . , x T which minimizes f subject to g while revealing the unknown characteristics. After making a decision x t , we obtain a noisy measurement, y t = f (x t , p(x t )) + where f : D × P → R and is R-sub-Gaussian noise for a fixed constant R ≥ 0 (Agrawal and Goyal, 2013). At every iteration, the decision x t must satisfy the constraints g k (x t , p(x t )) + h k ≤ 0, where k = 1, 2, . . . , K with K denoting the number of constraints. The value of h k ≥ 0 is called a safety threshold. Thus, we can formalise our optimisation problem at time t as: subject to g k (X, p(X)) + h k ≤ 0, ∀k = 1, . . . , K (1b) where X = [x i , ..., x N ] for arms i = 1, . . . , N . Since the characteristics p(.) are unknown and need to be estimated, at every solution instance t, we need to first solve an estimation problem to find p, then solve the optimization problem (1). In ARTEO, we utilize GPs to learn p(.) and its covariance matrix to establish confidence bounds that include the true value of safety function with 1 − δ probability where δ ∈ (0, 1) and a coefficient β t is chosen to satisfy explained conditions in Korkmaz et al. (2022). This guarantee can be formalized as next: In Korkmaz et al. (2022), we established the theoretical guarantee of the confidence bounds by choosing β t based on Theorem 3 of Srinivas et al. (2010), Theorem 2 of Ray Chowdhury and Gopalan (2017), and Theorem 1 of Sui et al. (2015) under the regularity assumptions of GPs. Further details regarding β parameter and regularity assumptions can be found in Korkmaz et al. (2022). The implementation details of the algorithm are given in the next section.

Gaussian Processes
Gaussian processes (GPs) are non-parametric models which can be used for regression. GPs are fully specified by a mean function µ(x) and a kernel k(x, x ) which is a covariance function and decides the shape of prior and posterior in GP regression (Rasmussen, 2004). The goal is to predict the value of p including unknown components at the decision point x * by using GPs and then compute f . Assuming having a zero mean prior, the posterior where t, j ∈ {1, .., N } and denotes the index of observa- x j ) and y t is the noisy feedback of tth observation.

ARTEO ALGORITHM
ARTEO is developed for safety-critical systems with high exploration costs. It does not require a separate training phase and learns during normal operation. The algorithm is executed periodically and the optimal solutions for (1) can change due to a change in optimization targets, a change in system characteristics, or can be driven by the exploratory contribution in the cost function. At each solution instance ARTEO updates the posterior distributions of GPs with previous noisy observations and provides the optimal solution for the desired outcome based on how GPs model the unknown components. In a setup with multiple-arms, the decision set D i is defined for each arm i as satisfying the introduced assumptions in Korkmaz et al. (2022). For each arm, a GP prior and initial safe seed set is introduced to the algorithm. The safe seed set S 0 includes at least one safe decision point with the true value of the safety function at that point. Safe seed points are given to accurately assess the safety of chosen operating points in later iterations as in previously published safe exploration algorithms (Sui et al., 2015(Sui et al., , 2018Turchetta et al., 2019). At the first solution instance, each GP corresponding to arm i is fit by a safe seed set and given GP prior to computing a posterior distribution. At later instances, the safe set is expanded by noisy observations as collected after the execution of decided operating points from the realtime optimizer. These observations are used to compute posteriors to use in the optimization of the utility function, which includes the cost of decision and uncertainty.
The uncertainty in the utility function f t is quantified as: where σ(x it ) is the standard deviation of GP i at the point x it for the arm i in the iteration t and n is the number of arms. It is incorporated into the utility function f by multiplying by an adjustable parameter z as next: where C t (X t , p(X t )) represents the cost of decision at the evaluated points. The uncertainty weight z is adapted using present uncertainty in the environment. The impact of adjusting z on the performance of ARTEO will be shown in Section 4.4.
The optimization phase occurs after updating the posteriors of each arm. The real-time optimizer incorporates the posterior of the GP into decision-making by modelling the unknown parameters for each arm by using the mean and standard deviation of GPs. The utility function is the objective function in the real-time optimization formulation and the safety thresholds are constraints. In the utility function, the mean of the GP posterior of each arm is used to evaluate the cost of the decision and the standard deviation of GPs is used to measure uncertainty. In the safety function, the standard deviation of the GP posterior of each arm is used to construct confidence bounds and then these bounds are used to assess the feasibility of evaluated points. The optimizer solves the minimization problem under safety constraints within the defined decision set of each arm. Any optimization algorithm that could solve the given problem can be used in this phase.
We have a constrained nonlinear case study, and we choose an interior-point algorithm to solve our problem.
Since we apply the ARTEO algorithm to a dynamical system simulation, we build a steady-state detector and evaluate costs and constraints in a steady-state. For the steady-state detection algorithm, we use a statistical testbased detector as common in many real-time optimization tools (Câmara et al., 2016). The pseudocode of ARTEO is given in Korkmaz et al. (2022).

CASE STUDY
We apply the ARTEO approach to a food freezing process plant. This case study is introduced first in Widell and Eikevik (2010), which we develop further into a dynamic simulation. We develop the simulation and algorithm with Fig. 1. Simulation of the refrigeration system with ARTEO Fig. 2. Schematic of the case study refrigeration system MATLAB/Simulink 2022a, and conduct experiments on an M1 Pro chip with 16 GB memory. The general structure of the industrial refrigeration control system with the ARTEO layer used in the simulations is given in Fig. 1.

The industrial refrigeration system
The industrial refrigeration system is formed by two subsystems with different pressure levels. The two subsystems consist of 11 compressors: three reciprocating, and eight screw compressors used to freeze stored fish within the refrigeration system. We focus on the screw compressors because they have varying ranges of cooling capacities and different power consumption characteristics. The first subsystem has 5 screw compressors and the second subsystem has 3 screw compressors. The system with five screw compressors is chosen to apply our approach. Fig. 2 illustrates a simplified process layout of this subsystem.
For the dynamic modelling of the screw compressors, 200 seconds is considered to be the time needed to move from one steady-state to a new one based on the dynamic experiments from Fu et al. (2003). Therefore, ARTEO is triggered by 250 seconds time intervals to decouple the optimization actions from the compressor controllers. For representing the performance of screw compressors, the characteristics capturing the cooling load and power consumption data is obtained from Manske et al. (2000) and embedded into simulations but not directly revealed to the optimization algorithm. The system consists of one smallsized, one medium-sized and three large-sized compressors. The minimum and maximum cooling capacities for each compressor type are given in Table 1. The coefficient of  performance for cooling for all compressors is assumed to be 1.6 based on data from Widell and Eikevik (2010).
The cooling demand is subject to change based on the stored amount of fish and environmental factors. Fig. 3 demonstrates the production load during fishing season when the plant is needed to be cooled at full capacity for most days but also with considerable variations. This period is chosen to highlight that large production rates require higher energy consumption, which can be a safety constraint in some cases due to a shortage of electricity in the power grid or similarly a critical cost concern when regulations or power purchase contracts demand enforcing a peak electricity consumption limit.
More specifically, we demonstrate our approach over the period Saturday, Sunday and Monday in Fig. 3 because those days have significant fluctuations in the process cooling demand, which drive the plant to work at different operating points. Operating at unknown points may cause the violation of constraints due to unknown performance characteristics at those points.

Safe learning and optimization with ARTEO
For this case study, we choose the Squared Exponential kernel, which is a positive definite kernel function, as the GP prior for each compressor based on the suggestion of the GP hyperparameter optimization tool under the Statistics and Machine Learning Toolbox in Matlab (Rasmussen, 2004). The minimum and maximum cooling capacities in Table 1 are introduced as decision-set boundaries to the optimization: x i min ≤ x it ≤ x i max i = 1, . . . , 5, t = 1, . . . , 42000 (6) Then, we create the safe seed set including three safe decision points for each compressor to obtain the posterior of GPs before making any decisions. We define the utility function f as follows: where µ Pi(xit) represents the mean prediction of power consumption at production load x and σ Pi(xit) represents the standard deviation of prediction to represent uncertainty in predictions at load x it . M t denotes the desired production load at time t based on the introduced cooling scenario. We develop the following two conditions and assign a positive value z = 1000 when these conditions hold: (1) the cooling demand has stayed the same in the two previous sampling instances, (2) there is at least one feasible decision point that satisfies the cooling demand with a tolerance α = 10 kW. The safety threshold for total power consumption is set as 80% of the maximum power consumption possible for all compressors, which corresponds to 1580 kW, to emulate an external command given to limit the power consumption of the process. The safety constraint g is defined next as: Fig. 6. Target, achieved cooling demands and total power consumption over time where β t is chosen as explained in Korkmaz et al. (2022). The experiment is simulated with the given cooling load trajectory in Fig. 2. Fig. 3 demonstrates the first solution instance of the algorithm, where the uncertainty is high and it can be recognized from the wide confidence bounds. At each instance, GPs are updated with new observations, and the algorithm takes new decisions by the updated GPs. Fig. 4 shows the final results for performance curve estimations at the end of the simulated cooling load scenario. It can be seen that the ARTEO algorithm is able to successfully approximate the power consumption curves very close to their real shape. Fig. 6 illustrates the expected and achieved production loads. The expected production load is satisfied unless: (1) it is higher than what compressors can achieve without crossing the safety threshold, (2) it requires compressors to operate lower than their minimum operating points. Fig. 6 demonstrates that when the total power consumption is on the limit of the maximum power threshold, the expected load could not be satisfied. At those production loads, our approach finds safe operating points which minimize the stated objective function. Fig. 4 shows that at the first solution instance of the algorithm, not all true power consumption curves (in green) for the compressors -particularly for unexplored high load regions -lie inside the confidence bounds constructed by GP regression (blue areas). Since the cooling load scenario in Fig. 6 does not move rapidly from a low load to a high load, the algorithm gets the chance to update the GPs and the corresponding bounds before a high load is requested. We now investigate a counterexample where the system receives a high cooling load demand earlier, without exploring intermediate loads. In this case, ARTEO moves the compressors to an operating point past the allowed maximum power consumption to track the requested cooling load based on the satisfaction of the UCB-based constraint but steps back from the target load to satisfy Fig. 7. Safety threshold is exceeded when the system moves far into a previously unexplored space the actual safety limit once the steady-state detection algorithm registers the new data points and updates the GPs and the UCBs. Fig. 7 illustrates this counterexample case. A basic approach to avoid the safety violation in this example can be to constrain the search space to the vicinity of the previously explored operating points using a distance measure, which would act analogous to input rate-of-change constraints in optimal control applications.

Hyperparameter selection
The performance of ARTEO is sensitive to two factors: (i) the estimation of GP models and (ii) the value of z used for exploration. Gaussian process regression is a technique driven by its own hyperparameters such as kernel selection and noise levels. In this case studybased on hyperparameter optimization -the squaredexponential kernel is chosen as the kernel function for all compressors. We can visualize the total uncertainty in an environment by the standard deviation of GP models summed over a discretization of the operating envelope. In a static environment, we expect the total uncertainty to decrease over time due to exploration. This decrease is led by the size of z, with greater z corresponding to a quicker decrease. Fig. 8 shows the trend in the decrease of the total uncertainty in a case study for two different Here, the total cooling demand is kept at a fixed value and individual compressor loads are moved by the ARTEO algorithm to decrease the uncertainty while maintaining the safety threshold. By adjusting the value of z, it is possible to shift the emphasis in the utility function to minimisation of the uncertainty, if the term z is greater relative to the power consumption term. Conversely, setting a small value of z enables focusing on the minimisation of power consumption, at the expense of leaving a larger uncertainty. It is possible to adapt z in a dynamic environment with changing characteristics e.g. when the compressor performances deteriorate over time. This can be achieved by varying the value of z based on the observed prediction errors for the compressor power consumption values. Higher errors will indicate increasing plant-model mismatch and will increase z for more exploration. Forgetting older data points will also increase uncertainty and will drive exploration without changing the value of z.

Comparison with exact knowledge
Lastly, we compare ARTEO with an optimization, which has access to exact knowledge of the environment. For this comparison the cooling scenario in Fig. 3 is used again and the optimization problem is solved with the same polynomial equations of the power consumption characteristics used in the simulations. The comparison of the results for power consumption and cooling load demand satisfaction is given in Fig. 6. Here we can see that ARTEO performs close to the optimization with exact knowledge even though the characteristics are initially unknown to ARTEO and are estimated through noisy observations.

CONCLUSION
In this work, we introduced and studied an industrial refrigeration case study as an example for the implementation of the ARTEO algorithm for the adaptive and explorative real-time optimization of a safety-critical system under uncertainty. The ARTEO algorithm successfully minimizes the power consumption while tracking the desired cooling load within the operational and safety constraints considered in the main case study. Even though the system starts with very limited information about the process, online learning provides comparable performance to a solution with exact knowledge of the environment.


Retrieval Augmentation for T5 Re-ranker using External Sources

Retrieval augmentation has shown promising improvements in different tasks. However, whether such augmentation can assist a large language model based re-ranker remains unclear. We investigate how to augment T5-based re-rankers using high-quality information retrieved from two external corpora -- a commercial web search engine and Wikipedia. We empirically demonstrate how retrieval augmentation can substantially improve the effectiveness of T5-based re-rankers for both in-domain and zero-shot out-of-domain re-ranking tasks.

Introduction
Large language models (LLMs) have demonstrated strong performance for both retrieval and reranking tasks (Qiao et al., 2019;Nogueira et al., 2020;Tay et al., 2022). Meanwhile, recent research has enhanced LLMs by augmenting them with retrieval capabilities. These retrieval augmentation strategies have demonstrated promising results on tasks such as question answering (Guu et al., 2020;Borgeaud et al., 2021) and knowledge-grounded conversation (Shuster et al., 2021;Cohen et al., 2022). However, it remains an open question as to whether retrieval augmentation can improve the effectiveness of LLM-based re-rankers.
Inspired by the uses of the external expansions (Diaz and Metzler, 2006), we explore how query representations can be augmented using information retrieved from high-quality corpora to improve re-ranking quality. Similar to RAG  and RETRO (Borgeaud et al., 2021), we augment a given query with information retrieved from external resources before performing inference (i.e., re-ranking in this case). We consider external resources from two corpora - * Corresponding Author a commercial web search engine and Wikipedia. We consider two approaches to augment the original query with retrieved information. The first approach augments the query with a sequence of terms while the second augments the query with a bag of words. Finally, we investigate the efficacy of query augmentation strategies for in-domain and zero-shot out-of-domain settings. We evaluate these conditions on three datasets: open-domain Natural Questions (Kwiatkowski et al., 2019), MS MARCO passage re-ranking (Nguyen et al., 2016), and TREC DL Track 2019 (Craswell et al., 2020) and 2020 (Craswell et al., 2021). This paper makes three contributions: (1) To the best of our knowledge, this is the first investigation of retrieval augmentation for LLM-based rerankers; (2) Retrieval augmentation is empirically analyzed using four T5-based reranker variants for in-domain and zero-shot out-of-domain tasks; (3) A comprehensive experimental study is presented using two retrieval corpora and two different query augmentation strategies.

Related Work
Retrieval augmentation for NLP tasks. Due to the opaque nature of knowledge stored in the parameters of LLMs, retrieval augmentation has been introduced for a variety of different NLP tasks. For example, on question answering tasks, REALM (Guu et al., 2020), RAG  augment inputs with a document corpus, enriching the representation using top-ranked retrieved items via Maximum Inner Product Search (MIPS). Meanwhile, RETRO (Borgeaud et al., 2021) uses retrieval to augment at the granularity of small chunks of tokens. It has also been shown that retrieval augmentation can help provide bettergrounded text in dialogue systems (Shuster et al., 2021;Cohen et al., 2022) and in the evaluation of hallucination (Honovich et al., 2021). Inspired by these successes, our work investigates retrieval augmentation for re-ranking using a fixed retrieval component. Query expansion and pseudo-relevance feedback (PRF). In early work, Diaz and Metzler (2006) showed it is effective to incorporate information from an external corpus into a non-neural language modeling framework. We exploit such information when using a pre-trained language model for re-ranking by directly augmenting the original query with the top-ranked results from an external corpus. An orthogonal research direction is to improve re-ranking models by incorporating pseudorelevance feedback (PRF) signals as in (Li et al., 2018;Padaki et al., 2020;Zheng et al., 2020;Yu et al., 2021;Naseri et al., 2021). One essential component therein identifies the relevant information from the pseudo relevance, avoiding the topic shift. Besides, these methods are involved with expensive multiple iterations to collect the PRF and use that for re-ranking. In contrast, our model consumes high-quality external augmentation text and requires one single iteration.

Method
We adopt Nogueira et al.'s method for re-ranking with LLMs (Nogueira et al., 2019). Let q be the query string, d be the document string, and y be a string that represents the binary relevance of a document, e.g., "True" or "False". We construct a (string) instance x as, x = "Query:q Document:d Relevant:y" (1) The model is trained to generate the final token (i.e. y) based on the ground-truth relevance of the querydocument pair. To score a new query-document pair, the normalised score of the final token is used for re-ranking.
We are interested in augmenting x with information from an external corpus. We assume that access to the external corpus is mediated through a retrieval service f such that f (q) = [σ 1 , . . . , σ m ], where σ i is a retrieved passage (e.g. web search snippet, indexed passage). It is important to note that the retrieval service can only retrieve items from a given external corpus and cannot re-rank or re-score documents in the target corpus.
We represent the information f (q) as an augmenting stringq. We can directly concatenate the m passages to constructq; we refer to this as natural language expansion. Although we expect the natural language expansion to be more compatible with LLMs, the fixed capacity of LLM modeling can result in situations where informative text is obscured by 'linguistic glue' often discarded as stop words (Tay et al., 2020). Alternatively, we can extract the most salient topical terms from f (q) as in (Dang and Croft, 2013). Specifically, we select terms using the KL2 method (Carpineto et al., 2001;Amati, 2003). In this method, we select k terms from all of the terms in f (q) using each individual words' contribution in the KL-divergence between the language model in f (q) (denoted as A) and the corpus (denoted as C).
We estimate the corpus language model using the target retrieval dataset. We refer to this as topical term expansion. In both expansion methods, we truncate the concatenated snippets, paragraphs, or ordered set of topical words (according to Eq. 2) to a maximum sequence length.
To incorporate retrieved information, represented asq (the expansion terms), we add the text as a new subsequence ("Description") in x, x ="Query:q Description:q Document:d Relevant:y" Because we are representing instances as strings with a terminal relevance label, we can easily adopt the same re-ranking method as Nogueira et al. (2019).

Experiments
Training data. We use two training datasets, namely, Natural Questions (NQ) originally proposed in (Kwiatkowski et al., 2019), and, the MS MARCO (Nguyen et al., 2016) passage re-ranking dataset. The NQ dataset includes 79k user queries from the Google search engine. The subset of NQ derived in  are used. The data has the form (question, passage, label), where only the queries with short answers are included. The task is to retrieve and re-rank the chunked paragraphs from Wikipedia with up to 100 words for the queries. Meanwhile, we use the MS MARCO triplet training dataset (Nguyen et al., 2016), which includes 550k positive query-passage pairs. For validation purposes, we measure Success@20 (also called Hits@20) on the 8757 questions in the NQ   (Macdonald et al., 2012).

Augmentation details and model variants.
We fine-tune T5-small, base, large, and, xl models on NQ and MS MARCO, with and without augmentation. Specifically, we investigate the individual expansions described in Section 3. For the search engine snippets, we collect up to five organic search result snippets per query. using a commercial search engine. The direct answers that appear in the top of the search results, if any, for answer-seeking queries are skipped to avoid the leaking of the answers in the augmentation. For the Wikipedia paragraphs, we use non-overlapping text chunks with up to 100 words from DPR  for expansion. The hybrid method  from Pyserini 2 (alpha = 1.3) is used to gather the initial top-1k Wiki paragraphs. Thereafter, a T5-xl re-ranker is fine-tuned on the NQ dataset to identify the most relevant paragraphs. Under the natural language setting, we truncate the augmentation text to 64 words, which is about two web search snippets or one Wikipedia passage. Under the topical term setting, we use the top-64 topical terms from the top-5 web search snippets or Wikipedia passages. Statistical significance relative to the non-augmented T5-based rerankers is reported using a two-tailed paired t-test. For the models trained on NQ, we only report the augmentation results using web search snippets. This is because we exploit the Wikipedia corpus pre-processed from the DPR paper , leading to the same corpus as in our NQ benchmark, making the augmentation non-external and is not something that we are exploring here.
Training. We employ Mesh Tensorflow (Shazeer et al., 2018) for training and evaluation. All of the T5 models are initialised with pre-trained checkpoints. The models are fine-tuned and inferred as in (Nogueira et al., 2020). Instead of selecting models from a fixed step, all models are trained up to 150k steps and are selected according to their validation set performance. The query expansion model is trained by augmenting inputs as described in Section 3. During training, a constant learning rate of 1e-3 is used for all models.

Results
We summarize results for the NQ and MS MARCO models in Table 1 and 2, respectively, with validation performance in the second column. We also include the performance of the initial ranking for reference.
For the in-domain experiments, we observe improvements both over the initial ranking and the non-augmented model on both NQ and MS MARCO when augmenting queries with external information. In the NQ development and test sets (Table 1), expanding using search results yields consistent boosts (more than 4% relative improvement in S@1 for T5-base, T5-large, and T5-xl models; more than 15% improvements for T5small). We observe similar behavior for the MS MARCO development set (Table 2), where the use of search engine result augmentation yields consistent boosts. Furthermore, despite the small number of test queries in TREC DL Track 2019 and 2020, augmentation provides consistently significant improvements across different T5 variants.
For the out-of-domain experiments, we measure the zero-shot effectiveness of models trained on a different dataset. When training on the NQ dataset (Table 1), the models, with or without augmentation, fall short on TREC DL tracks. This could be due to the difference in tasks (QA in NQ versus IR in TREC DL) or the relatively small amount of training data available in NQ (∼50k unique queries). However, significant improvements are still observed for T5-small on TREC DL Track 2019 and 2020. By comparison, when training with MS MARCO (∼550k unique queries), the zero-shot performance on NQ is consistently improved over S@1, S@5, and S@10 across different model variants. Noticeably, no non-augmented T5 variant outperformed the initial DPR ranking in terms of S@1, while augmented T5-large and T5xl outperformed the initial ranking. These findings highlight the potential value retrieval augmentation has to improve re-ranking quality in out-of-domain settings.
Augmentation for different T5 model variants. From Table 1 and 2, it can be seen that augmentation yield larger gains for smaller models. For T5-small and base, augmentation tends to provide more consistent improvements for both in-domain and out-of-domain settings; whereas for T5-large and T5-xl, augmentation helps in the in-domain setting, but does not always help in the out-of-domain setting. For example, augmentation barely helps on TREC DL when being trained on NQ, and diminishing improvements are observed in terms of S@20 on NQ-test for models trained on MS MARCO.
With different sources and strategies for augmentation, using the MS MARCO development set (Table 2) as an example, augmentation from both sources improved performance across different T5 variants. Information from Wikipedia resulted in smaller improvements compared to search engine information, suggesting that either corpus size (or comprehensiveness) or representativeness (e.g., domain difference) may be important when selecting an augmentation source. Furthermore, across experimental conditions, improvements from natural language and topical words were comparable, without strong evidence of one being superior.
Relationship with pseudo-relevance feedback (PRF). As mentioned in Section 2, the use of PRF is orthogonal, thus we do not include PRF models for comparisons. Therein, a carefully-designed component to avoid topic shift is crucial for PRF model to perform, which is not something that we are investigating here. Actually, Padaki et al. (2020) demonstrated that PRF on top of BERTbased re-ranker reduced ranking quality by more than 10%, e.g., nDCG@10 decreases from 0.498 to 0.448 after using PRF expansion (namely, "Clas-sicQEConcepts") as reported in Table 3; and the direct use of suggested questions from Google as query expansion can barely help, getting 0.502 of nDCG@10 (namely, "GoogleQuestions"); unless these questions are manually filtered, achieving nDCG@10=0.526 using "FilteredGoogleQuestions". Herein, the use of "FilteredGoogleQuestions" gets close to our settings, but with different resources and involving manual filtering in extra.

Conclusion
We investigated the use of augmentation from highquality external sources for re-ranking tasks, using two external sources, namely, snippets from a commercial search engine and Wikipedia passages. We also investigated representing the external information as natural language sequences and as bags of words. We demonstrated the effectiveness of retrieval augmentation for T5 re-rankers for both in-domain and zero-shot out-of-domain settings.

Limitations
As a showcase of retrieval augmentation for T5-based re-rankers, this work did not comprehensively investigate models beyond T5, like duoT5 , and did not explore more advanced modeling of the external sources, like co-training the augmentation components together with the re-ranker as in REALM (Guu et al., 2020), RAG  and RETRO (Borgeaud et al., 2021). Applying established methods on top of the retrieval augmented re-ranker to further advance the state-of-the-art could be another research direction. For example, we could have applied pseudo relevance feedback (PRF) on top of the retrieval augmentation reranker to see if there exists further improvements. Finally, an orthogonal but highly-related direction is to use retrieval augmentation to improve retrieval, which is not something that we explored here. Instead, this work assumes a fixed retriever that does not use augmentation is used.


Revealing Hidden Context Bias in Segmentation and Object Detection through Concept-specific Explanations

Applying traditional post-hoc attribution methods to segmentation or object detection predictors offers only limited insights, as the obtained feature attribution maps at input level typically resemble the models' predicted segmentation mask or bounding box. In this work, we address the need for more informative explanations for these predictors by proposing the post-hoc eXplainable Artificial Intelligence method L-CRP to generate explanations that automatically identify and visualize relevant concepts learned, recognized and used by the model during inference as well as precisely locate them in input space. Our method therefore goes beyond singular input-level attribution maps and, as an approach based on the recently published Concept Relevance Propagation technique, is efficiently applicable to state-of-the-art black-box architectures in segmentation and object detection, such as DeepLabV3+ and YOLOv6, among others. We verify the faithfulness of our proposed technique by quantitatively comparing different concept attribution methods, and discuss the effect on explanation complexity on popular datasets such as CityScapes, Pascal VOC and MS COCO 2017. The ability to precisely locate and communicate concepts is used to reveal and verify the use of background features, thereby highlighting possible biases of the model.

Introduction
Deep Neural Networks (DNNs) have proven to be successful in providing accurate predictions in several critical object localization tasks, including autonomous driving [6] or medical screening [42]. However, the reasoning of these highly complex and non-linear models is generally not transparent [50,52], and as such, their decisions may be biased towards unintended or undesired features [5, 37,58]. In particular, object localization models sometimes base their decisions on features that lie outside of the segmentation mask or bounding box [28,47], as shown in Figure 1, where the context of a hurdle is used for the segmentation of a horse. Understanding such contextual usage by DNNs is crucial to meet the requirements set in governmental regulatory frameworks and guidelines [14,23].
In order to increase our understanding of DNN predictions, the field of eXplainable Artificial Intelligence (XAI) has proposed several techniques that can be characterized as local, global or a combination of both.
In literature predominantly addressed are local XAI methods for explaining single predictions. These methods compute attribution scores of input features, which can be visualized in form of heatmaps highlighting important input regions. However, in object localization or segmentation tasks, such attribution maps often resemble the prediction, e.g. the segmentation mask itself, and hence add little value for gaining new insights into the decision process of a model, e.g. when no context is used (see Figure 2 (right)).
Alternatively, global XAI aims to visualize which features or concepts have been learned by a model or play an important role in a model's reasoning in general. Nonetheless, it is not clear which features were actually used for a particular prediction or how they interact. For individual samples, latent activation maps of features can be visualized, that however are of low resolution and give no indication whether the feature was actually used or merely present in the input, as illustrated in Figure 1 (right).
Glocal concept-based explanations, on the other hand, offer a new dimension of model understanding compared to traditional heatmap-based approaches by not only indicating where the model pays attention to, but also informing about what it sees in the relevant input regions [1, 46,66]. As far as we know, we are hereby the first work to address glocal concept-based explantions for object localization tasks.
Our glocal methodology is hereby based on Concept Relevance Propagation (CRP) [1], an extension to local Figure 1. Our concept-based eXplainable Artificial Intelligence (XAI) approach (center) goes beyond traditional local heatmaps (left) or latent activation analysis (right), by communicating which latent concepts contributed to a particular detection, to what degree in terms of relevance (R) and where concepts are precisely located in input space. Whereas the shown heatmap from Layer-wise Relevance Propagation (LRP) only indicates the relevance of pixels overall, our method disentangles the latent space and makes it possible to understand individual concepts forming the prediction outcome. Especially in the multi-object context, it is crucial to attain object-class-specific explanations, which is not possible by the analysis of latent activations: Concepts with the highest activation values can refer to any class that is present in the image, such as the horse's rider, or none at all, since activations do not indicate whether a feature is actually used for inference. attribution methods based on the (modified) model gradient, such as Layer-wise Relevance Propagation (LRP) [7]. Specifically, CRP allows to disentangle local attributions by computing relevance scores and heatmaps for individual latent features, thereby making it possible to localize concepts precisely in input space. Concept localization allows us in Section 4.2 to identify context biases and the use of background features, such as the "vertical bar" concept targeting the hurdle in Figure 1.
Contributions In this work, we address the limited explainability insight of heatmap-based local XAI methods for object localization tasks. We therefore present a novel method called CRP for Localization Models (L-CRP) for glocal concept-based explanations based on CRP for state-of-the-art architectures for semantic segmentation and object detection tasks, including UNet [20], DeepLabV3+ [13], YoloV5 [17] and YoloV6 [38] trained on the public datasets of CityScapes [15], Pascal VOC 2012 [19] and MS COCO 2017 [39]. Concretely, 1. We demonstrate how to explain state-of-the-art architectures for segmentation as well as object detection with LRP locally and by adapting CRP glocally on a concept level.
2. We evaluate our method in terms of faithfulness and explanation complexity using different concept attribution methods, including activation, gradient, as well as several parameterizations of LRP-attributions.
3. Being able to localize concepts in input space via CRP heatmaps and having object masks or bounding boxes available, we compute context scores for concepts indicating to which degree a concept is used for encoding background or object features and how the model makes use of the respective information during inference. We show how these insights can be used to detect biases in the model and verify our findings by interacting with the model.

Related Work
In the following, the current landscape of XAI methods for segmentation and object detection models is presented, followed by a description of concept-based XAI techniques.

Explainable AI for Segmentation
The literature of XAI for segmentation predominantly focuses on local techniques. While some works apply backpropagation-based methods such as Grad-CAM [18, 41,61] or LRP [2] to compute attribution heatmaps, others propose to apply perturbation-based methods [28,62]. Contrary to classification tasks, local attributions are not computed w.r.t. a single output class neuron, but w.r.t. the whole output feature map that forms the segmentation mask representing the class. Several such local XAI techniques have been compiled into the Neuroscope toolbox [54].
Alternatively, Losch et al. inspect latent features of segmentation models and introduce Semantic BottleNecks [40] modules to increase the latent space's human interpretability. They visualize and investigate intermediate filter activation maps, which, however, are of limited insight for understanding particular predictions due to not being class-and outcome-specific. Further, the fidelity of activation maps is limited to a convolutional channel's spatial resolution, as shown in Figure 1, where the attributions from L-CRP offer high resolution, as well as object-and outcome-specificity.
Alternatively, another group of works investigates inherently interpretable architectures, such as U-Noise [36], Seg-NBDT [62] or MSGA-Net [32]. However, a large number of models applied in industry and research are not designed to be human-interpretable in the first place and thus require post-hoc methods for interpretation, such as ours.

Explainable AI for Object Detection
Similar to image segmentation, several local XAI methods have been presented for object detection. These techniques can be grouped into methods based on the (modified) gradient such as Gradient-SHAP [33], LRP [31,59], Spatial Sensitive Grad-CAM [63] and EX2 [24], or inputperturbation techniques such as LIME [24] and masking [47,53,64]. Methods based on the gradient hereby explain the output class logit of a chosen bounding box analogously to the classification case.
It is to note, that perturbation-based attribution methods require a high number of prediction re-evaluations by the model, resulting in run times in the order of minutes per data point, as e.g. for [47]. Our method is based on the (modified) gradient and, therefore, can be computed in the order of seconds [1].

Concept-based Explanations
In recent years, a multitude of methods emerged to visualize in a human-interpretable way concepts in the latent space learned by a model. A line of work [9, 11] assumes that individual neurons encode distinct concepts, others view concepts as directions described by a superposition of neurons [34,60]. Similar to contemporary literature [1, 9, 11], we treat each neuron or convolutional feature map as an independent concept to achieve the highest granularity in explanations, while the method presented in this paper can be, in principle, also extended trivially to concept directions in the latent space.
For models operating in the image domain, contempo-rary work relies on activation maximization for visualizing concepts [22,45,48,65], where in its simplest form, input images are sought that give rise to the highest activation value of a specific concept unit. However, high activation does not necessitate that corresponding input features are representative of a neuron's function, as adversarial examples illustrate. In this work, we make use of Relevance Maximization (RelMax) [1] that mitigates the aforementioned issues by choosing reference images from the original training distribution based on maximal relevance instead of activation. Glocal XAI methods try to bridge the gap between the visualization of concepts on a global scale and attribution of their role during per-sample model inference. Shrouff et al. [56] combine TCAV [34] with Integrated Gradients to enable local attributions of neuron vectors, however, without offering localization of latent features in input space. Achtibat et al. [1] propose the idea of CRP, an extension of latent feature attribution methods based on the (modified) gradient, where a concept-specific heatmap can be computed by restricting the backward propagation of attributions through the network. This allows to attain precise concept-conditional heatmaps in the input space, which we will use in Section 4.2 to investigate the use of background features by the model.

Methods
Our glocal concept-based method L-CRP is based on the principle of CRP, with LRP as the feature attribution method of our choice. Therefore, we first introduce LRP and CRP to attain concept-based explanations for individual predictions. Thereafter, we describe how concept-based explanations with L-CRP for segmentation and object detection can be obtained.

Layerwise-Relevance-Propagation
Layer-wise Relevance Propagation [7] is an attribution method based on the conservation of flows and proportional decomposition. For a model f (x) = f n • · · · • f 1 (x) with n layers, LRP first calculates all activations during the forward pass starting with f 1 until the output layer f n is reached. Thereafter, the prediction score f (x) of any chosen model output class is redistributed as an initial quantity of relevance R n back towards the input layer after layer.
Given a layer's output neuron j, the distribution of its assigned relevance score R j towards its lower layer input neurons i can be, in general, achieved by applying the basic decomposition rule with z ij describing the contribution of neuron i to the activation of neuron j. The aggregated pre-activations z ij at output neuron j are represented by z j with z j = i z ij . The relevance of a neuron i is then simply an aggregation of all incoming relevance quantities (2) In order to ensure robust decompositions and stable heatmaps, several purposed LRP rules have been introduced in literature [43,52]. During the experiments in Section 4, we compare the rules of LRP-ε, LRP-γ and LRP-z + , leading to different explanations in terms of faithfulness and complexity as shown in Section 4.1. Please refer to the Appendix A.1 for a detailed description of used LRP-rules and how they are applied to the models.

Concept-Relevance-Propagation
With CRP, the authors of [1] combine global concept visualization techniques with the local feature attribution method of LRP. A first step to the unification of both local and global XAI is the realization that during the LRP backward pass, intermediate relevance scores are readily available, as computed in Equation (2). However, in order to also achieve concept-conditional heatmaps in input space, CRP firstly proposes to restrict the relevance propagation process of LRP via conditions.
Concretely, a condition c l can be specified for one or multiple neurons j corresponding to concepts of interest in a layer l. Multiple such conditions are combined to a condition set θ. To disentangle attributions for latent representations, the relevance decomposition formula in Equation (2) is therefore extended with a "filtering" functionality: where δ jc l "selects" the relevance quantity R l j of layer l and neuron j for further propagation, if j meets the (identity) condition(s) c l tied to concepts we are interested in. For layers without conditions, no filtering is applied. A conceptconditional heatmap can thus be computed by conditioning the modified backward pass of LRP via such a condition c l for the concept's corresponding neuron or filter.
For the visualization of concepts, we adhere to the proposition of [1] and collect reference input images for which a latent neuron is most relevant, i.e., useful during inference. Thereafter, by computing a conditional heatmap for the neuron of interest and reference sample, the relevant input part is further cropped out and masked to increase the focus of the given explanation on the core features encoded by the investigated neurons, as detailed in [1].

Extending Attributions to Object Localization
In order to obtain L-CRP for the generation of local explanations for segmentation and object detection models Figure 2. Local explanations of object detection (left) and semantic segmentation (right) with the LRP-z + rule. In the object detection case, we can explain each individual predicted object. Similarly, for semantic segmentation, the output segmentation map of an object of choice can be explained. Here, in principle, also parts of the segmentation can be investigated. Concept-based explanation variants with L-CRP can be seen for both object detection and segmentation in Appendix A.2. with CRP, the task-specific output vectors and maps have to be handled accordingly.
Semantic Segmentation Image segmentation models follow an encoder-decoder architecture in which the output mirrors (a scaled version of) the input dimensions in width and height. In contrast to the 1-dimensional output in classification tasks, the segmentation output consists of a 2d-map for each learned object category.
For simplicity, we will assume a decision function f for binary segmentation with a one dimensional input x ∈ R n and output f (x) ∈ R n of length n in the following, e.g., f : R n → R n . This is not necessarily a restriction to the input and output size, as any input can be flattened to a single dimension, e.g., n = whc for images of size w × h × c.
Here, any input feature x i (e.g. pixel) might contribute to all of the output values f j (x). Therefore, we can assign to feature x i a relevance score for each output j, that we summarize in a relevance vector R i (x|θ) ∈ R n . In fact, explaining a segmentation prediction can also be viewed as performing an explanation for each output pixel separately, and eventually adding the resulting attribution scores via, e.g., a weighted sum, as The weights can be used to control the meaning and outcome of the explanation. Here, a specific Region of Interest (ROI) can be selected to be explained by setting w i = 0 for pixels i outside the ROI. Further, the relevance values from different output pixels can be weighted differently. Choices include to weight scores according to their model confidence or, equally, by propagation of ones. For modified backpropagation-based attribution methods such as LRP, the output tensor used as the starting point for relevance propagation can be adapted directly in order to control the meaning of an explanation. An example of a local explanation using two kinds of ROIs are shown in Figure 2 (right), where both the whole segmentation output of a bus or the bus wheels are explained via binary masking.
For the initialization of relevance propagation at the output, we adhere to the in-literature often practiced approach by focusing on output scores corresponding to the highest class prediction. In order to weight relevance scores uniformly, the relevance propagation for explaining class y is initialized in the last output layer L as with an indicator function The tensor indices p and q refer here to the spatial dimension w×h and c describing the channel dimension, i.e. class confidence. Thus, not whole channel output maps are explained, but only the output values, which correspond to the selected class.
Object Detection Object detection networks often consist of a decoder part and a prediction module. The output per bounding box then includes class scores and, in addition to segmentation models, information about the position of the bounding box. Local attributions are then similarly computed to the classification task w.r.t. the class score of a chosen bounding box. Regarding a bounding box predictor f : R n → R N ×(nc+4) with an output of N bounding boxes for n c object classes with four coordinates, the relevance R (k) i (x|θ) of feature x i for bounding box k of class y is then given by initializing the relevance propagation at the output as with b representing the bounding box axis. Thus, for each bounding box, an individual explanation can be computed, as is shown in Figure 2 (left).

Experiments
The experimental section is divided into two parts, beginning with the evaluation of our concept-based XAI method using CRP and different feature attribution methods, thereby showing the superiority of computing latent relevance scores instead of activations -especially in the multi-object case. In the second part, we leverage the fact that we can localize the object through ground truth masks or bounding boxes and thus are able to measure how much the background instead of the actual object is used. We show how this can be used to detect possible biases in the model corresponding to specific feature encodings.
Concretely, we apply our method to PyTorch implementations of a UNet on the CityScapes dataset, a DeepLabV3+ on the Pascal VOC 2012 dataset, a YOLOv5 and YOLOv6 model on the MS COCO 2017 dataset. Please refer to Appendix A.1 for more details on the models.

Evaluation of Concept-based Explanations
In recent years, several methods have been proposed to evaluate local explanations, that are readily applicable through unifying frameworks. Following the authors of [12, 27], we evaluate our presented method w.r.t. to faithfulness and complexity. Faithfulness measures whether an attribution truly represents features utilized by the model during inference, while complexity measures how concise explanations are, which is of interest in context of e.g. human interpretation. Since the literature for evaluating local concept-based explanations is limited, we propose two experiments to test for faithfulness and complexity.

Faithfulness of Concept Relevances
In order to assess the faithfulness of our concept-based explanations, we measure the impact on the decision outcome if a set of concepts is perturbed. This idea is analogous to the pixel flipping experiment in [7], only to use latent concepts instead of input features.
Concretely, we begin by computing the relevance scores of all concepts in a layer for a given object prediction. Please note, that since we conceptualize each convolutional channel to correspond to a distinct concept, the concept relevance of a channel is acquired via spatially sumaggregation of intermediate relevance scores. Then, we successively deactivate the most relevant channels first in descending order by setting their activations to zero, reevaluate the model output and measure output differences. Whereas for object detection models the change in the class logit of a predicted bounding box is computed, for segmentation models the mean change of the output mask's logits is calculated.
Alternatively, we also perform concept flipping backwards, by initializing all filters with zero activation and successively "unflipping" the most relevant concepts in descending order, i.e., restoring their original activation values. This technique is designated as "concept insertion".
A faithful explanation is hereby characterized by a strong decline in performance if concepts are flipped, or a strong incline in performance if concepts are inserted. A typical experimental outcome is shown in Figure 3 (left). Here different LRP-rules, pure gradient, activation (mean and sumpooling), and a random perturbation baseline are compared Table 1. Experimental results for evaluating various concept attribution approaches in terms of faithfulness (higher is better (↑)) and complexity (lower is better (↓)). For each approach, the evaluated scores for all models are displayed (UNet/DeepLabV3+/YOLOv5/YOLOv6).  for concepts in layer features.10 of the UNet model. As demonstrated, perturbation of the most activating concepts is significantly less faithful than using relevance-based scores. This is expected, since relevance is object-specific and thereby filters out other activating concepts irrelevant for the current detection. Regarding concept relevances, the scores of gradient and LRP-ε are often characterized by the strongest decline/incline, as the gradient of the model faithfully measures the local sensitivity of the model to changes. However, as more features are perturbed, LRP-γ and LRPz + often perform better, as they better represent the important features in a more global manner by filtering out noisy attributions [7]. In order to receive a score for a whole model, the faithfulness tests are performed in various layers throughout the models on 100 randomly chosen predictions, and the area under or over the curve measured per layer and mean-aggregated to form a final faithfulness score. As can be seen in Table 1, the results depicted in Figure 3 are reflected throughout all tested models, by mostly showing the best scores for gradient and LRP-ε. Layer-wise faithfulness scores for all models can be found in the Appendix A.3.

Explanation Complexity and Interpretation Workload
While high faithfulness suggests that the concept attributions represent the model behavior correctly, they can still be noisy and not human-interpretable [35]. This effect is due to highly non-linear decision boundaries in DNNs [8].
In order to measure the complexity of explanations and the workload a stakeholder has to put in for understanding the explanations, two different measures are computed. First, the standard deviation of latent concept attributions per class is measured, indicating the amount of noise. A low variation suggests, that explanations of the same class are similar and precise, resulting in a lower amount of complexity. As a second measure, the amount of concepts necessary to study in order to comprehend 80 % of all attributions is computed. The more relevance is focused on a small number of concepts, the fewer concepts need to be analyzed.
An example for measuring explanation complexity is shown in Figure 3 (right), where the explanation variation in the first ten convolutional layers of the UNet architecture is shown. Activations exhibit the smallest variation because filters activate on average more often if the feature is present in the image even though not used for inference. Regarding relevance-based concept attributions, gradient and LRP-ε show especially in lower-level layers a high deviation, indicating noisy attributions.
Whereas activation-based approaches lead to a low explanation variation, the distribution of concept attributions is rather uniform, leading to unconcise explanations and a large interpretation workload. Here, relevance-based approaches, which generate object-specific attributions result in smaller relevant concept sets. Plots for all architectures are illustrated in Appendix A.4.
The results of measuring explanation complexity for every model using all predictions in the test datasets are given in Table 1, confirming the previous observations in Figure 3.
Taking into account the results of the faithfulness tests, it is apparent, that relevance-based concept attributions show higher faithfulness than activation, but are not necessarily easier to interpret in terms of explanation complexity alone.
Here, LRP-γ attributions show a good compromise between faithfulness and complexity in most experiments.

Concept Context Scores for Bias Detection
In an ideal world, a DNN trained with enough variety in training data learns abstract and generalized features. However, this is not always the case and several works have shown that DNNs develop biases or Clever Hans features, e.g. detecting specific classes because of correlations in the data, such as watermark signs [5, 55,58].
Also in our experimental datasets, objects are often displayed in the same environments or together with other object classes. A person is often pictured together with a kite, sitting on a horse, walking on the beach, or standing on a surfing board in splashing water. In fact, concepts can be found in the YOLOv6 model for exactly the abovementioned use cases, as shown in Figure 4.

Measuring the Context of Concepts
In order to reveal to what degree the model uses concepts for background features, we in the following compare computation of context scores by three different means: (1) latent concept activation map, (2) latent concept relevance map and (3) concept-conditional L-CRP heatmap. We define the context score C of concept i as the fraction of positive attribution a + outside of the object bounding box compared to the overall sum, as for n samples x j and maskm ∈ {0, 1} marking all background values with a value of one and zero else. Indices p  and q refer to the spatial dimension of the attribution maps.
In case of segmentation, the object is localized through pixel-accurate ground-truth masks, whereas for object detection bounding boxes are available. For measuring the context score in the latent space, masks or boxes are resized. Please note, that the use of latent feature maps is only possible for convolutional layers, but not for dense layers, contained in most classification networks. L-CRP heatmaps, however, can be computed for both layer types.
An example of computing the context score is shown in Figure 5 for the YOLOv6 model and concept 28 of layer ERBlock 5.0.rbr dense, which corresponds to the skin pattern of the giraffe. As can be seen, latent activation maps are not optimal for measuring context scores, as they are not object-and outcome-specific, leading to a broad activation on all giraffes in the image. Thus, context scores are over-estimated using activation. It is further apparent, that for high-level layers the high resolution of L-CRP maps leads to a more accurate context estimation than low-resolution latent maps.
When computing context scores, we found that concepts are used differently throughout object classes, as shown in Figure 6. Here, e.g., the texture of waves can be found in blankets of beds, feathers for birds, or in the water surrounding boats. This illustrates, that it is crucial to measure the context of concepts for each class.

Evaluating Context Scores
For evaluating estimated context scores, we propose to measure the model's sensitivity of concepts on the background of objects. We therefore inspect the influence on the concepts' relevances when the background is perturbed. Concretely, the background sensitivity S of concept i is defined as with concept relevance R i (x j ) and object samplesx j and x j with and without perturbed backgrounds, respectively. Specifically, we applied gray-scale random noise, random noise, gray, and random color perturbation. Each perturbation is further performed with 100 % and 50 % alphablending, totaling 8 perturbations on 60 random detections. The context scores are computed for the 50 most relevant concepts of a class and the corresponding 15 most relevant detections, according to L-CRP attributions with the LRPz + -rule on the test data.
Ideally, concepts with a high context score C also have a high background sensitivity S. As is summarized in Table 2, L-CRP results in both the highest correlation and lowest Root Mean Square Deviation values between context and sensitivity scores. Here, concept scores are evaluated for three layers of each model. It is to note, that using activations leads to relatively high correlations, but large RMSDs as context scores are over-estimated. This is further illustrated by distribution plots in the Appendix A.5.

Context-based Interaction with the Model
The availability of context scores in combination with our method allows to detect the use of single background features, and thus to precisely interact with the model.
To probe the reaction of the model, the object's background can be manipulated. As an example, we noticed that frisbee detections rely on background concepts corresponding to features of dogs, shown in Figure 7. Removing the dog from the image via in-painting leads then to a missed detection of the YOLOv6 model. Alternatively, as we are able to pinpoint background concepts in the latent space, we can also flip the corresponding concepts, and measure the effect on the prediction. When a dog is present in images, concept flipping decreases the predicted output logit by about 6 %, as shown in Figure 7, whereas predictions without any dog present are not influenced significantly.
We further perform latent background concept flipping for frisbee or surfing board detections with "person" context features as well as person detections with "surfing board" concepts, visualized in Figure 7 (bottom). Here, the removal of person concepts affects the surfing board prediction more strongly than the other way around. We can even find examples where flipping three background concepts leads to missed surfing board predictions (shown in Appendix A.5). This is expected, as surfing boards are more likely to be depicted with a person (97 % co-occurrence) than a person with a surfing board (5 % co-occurrence) in the training data, favoring the likeliness of the model to use context features. The variety of "person" contexts is much higher, making it possible for the model to become more generalized in concept utility [37]. The visualization of all flipped background concepts can be found in the Appendix A.5.

Conclusion
We propose L-CRP as an extension of the recent CRP method, enabling local concept-based understanding of segmentation and object detection models. By visualizing and localizing the model's internal representations utilizing L-CRP, the insight that a stakeholder gains can be significantly improved compared to traditional local XAI methods, which tend to merely resemble the object localization output. We evaluate several latent concept attribution methods showing a trade-off between explanation faithfulness and complexity, with relevance-based attributions providing the best compromise. Finally, we apply our method to measure the amount of context by which concepts are used by a model for the prediction of different object classes. We validate our results and show how context scores can be leveraged to reveal context bias and interact with the model.

A. Appendix
In the appendix, we provide additional information and results complementing the main manuscript. Beginning with Section A.1, details about the models, datasets, as well as technical minutae on the computation of LRP attributions are presented. Thereafter, Section A.2 provides additional examples of L-CRP explanations. This is followed by Sections A.3 and A.4, where more results on the evaluation of explanations are given regarding faithfulness and complexity, respectively. Finally, in Section A.5, additional results for the measurement of context scores of concepts are shown and discussed.

A.1. Technical Details
In the following, the models and datasets are presented, as well as details about the used LRP rules and their implementation.

Models and Datasets
The UNet [49] model is implemented using the segmentation models pytorch framework [30] and consists of a VGG-13 [57] encoder with BatchNorm layers, and weights pre-trained on ImageNet [51]. The model is trained for 100 epochs with a batch size of 40 on the CityScapes [15] dataset with an initial learning rate of 10 −3 using the Adam optimizer. The learning rate is reduced to 10 −4 after 50 and 5 · 10 −5 after 75 epochs. Images are resized to a height of 256 and width of 512 pixels and normalized using a mean of (0.485, 0.456, 0.406) and standard deviation of (0.229, 0.224, 0.225) over the three RGB color dimensions 1 . During training, we further apply random cropping to 256 × 256 pixels, a random horizontal flip (50 % probability), brightness, saturation, hue and contrast perturbation (20 % strength, 50 % probability), as well as adding random Gaussian noise (zero mean, variance between 10 and 50, 50 % probability). The final training results in a pixel-wise accuracy of 74.7 % and a mean intersection over union score of 35.1 %.
The YOLOv6 model is based on the PyTorch implementation of the authors [38] and corresponds to the "small" variant named YOLOv6s trained on the MS COCO 2017 dataset [39].
Similarly, the YOLOv5 model is based on the PyTorch implementation of [17] and corresponds to the "medium" variant named YOLOv5m trained on MS COCO 2017.

LRP Canonization
Canonization procedures restructure a model into a functionally equivalent architecture to which established attribution LRP-rules can be applied. Canonization efforts typically concentrate on replacing the Batch-Norm layer [25,29] or handling skip connections [10] as in the ResNet architecture [26].
Regarding canonization, BatchNorm layers are merged into the preceding linear layer for all models. This way, significant improvements in terms of explanations can be achieved [44].
Further, in the Bottleneck modules of DeepLabV3+ (based on the ResNet) and YOLOv5, as well as the RepVGGBlock module (based on [16]) of the YOLOv6, activations of different parallel operations are merged using a sum operation. Here, a Sum module is inserted to handle the summation, and to which a Norm-rule can be applied to normalize relevances during the relevance backpropagation step, as available in [4] and discussed in [10].
The authors of [16] also present a canonization procedure for the RepVGGBlock module of YOLOv6, which merges all linear operations of the module into a single convolutional layer, which we apply to layer model.backbone.stem.
LRP Rules Multiple rules for LRP were defined in the literature [35]. The LRP-ε rule is based on the basic decomposition rule (see Equation (1)) and is given as with z ij describing the contribution of neuron i to the activation of neuron j, and aggregated pre-activations z j = i z ij . The LRP-ε rule ensures that each neuron receives the attribution value (fraction), that it contributed to the output. The added parameter ε ∈ R + stabilizes the division of the denominator, and "dampens" contradicting contributions for |z ij | |z j |, as discussed in [43]. However, ε is usually set to a small value, e.g. 10 −6 in [4], resulting in noisy attributions for deep networks due to large relevance values from contradicting contributions |z ij | |z j |. In classification networks, the LRP-ε rule is often chosen for the dense layers, whereas increasingly stabilizing rules (presented in the following) are chosen for convolutional layers. In the YOLOv6 model implementation, single BatchNorm layers with label rbr identity in RepVGGBlock modules exist without neighbouring linear layers. We apply the LRP-ε rule for these individual BatchNorm layers.
Alternatively, the LRP-γ rule is defined as with positive parameter γ ∈ R + and (·) + = max(0, ·). The function of the LRP-γ rule is to favor positive contributions and at the same limit the unbounded growth potential of positive and negative relevance in the backpropagation step. Note, that the term γ i z + ij (for strictly positive z j ) effectively corresponds to the ε in Equation (1), but scaled to the magnitude of contributions z + ij . The LRP-γ rule has thus been found to be effective in reducing noisy attributions, and the amount of "filtering" can be controlled by one parameter γ, which is by default set to γ = 0.25 in the zennit framework [4]. However, if z j < 0, the denominator can become numerically unstable if |z j | ≈ γ i z + ij . Therefore, we use the generalized version presented in [3], and either favor positive contributions for z j > 0 or negative contributions for z j < 0, depending on the sign of z j : (A. 3) The LRP-z + rule represents a third rule, defined as by only taking into account positive contributions z + ij . The LRP-z + rule can be seen as the most stable or least noisy attribution method, representing Equation A.2 with γ → ∞. To receive explanations with high human interpretability and low amounts of noise, explanations are thus computed with the LRP-z + rule applied to all convolutional layers of the models. For the YOLOv6 an exception is made for visual examples, as the LRP-γ rule also provides stable heatmaps, showing higher faithfulness at the same time, as discussed in 4.1.
Finally, it is established practice to apply the LRP-rule to the first layer [35,43] in order for the attributions to become invariant against normalizations applied in inpu space and making them more human-readable by minimally reducing heatmap fidelity. The LRP-rule is hereby defined as Using the LRP-rule, the relevance of upper-level neuron j is equally distributed to all connected lower-level neurons disregarding any influence of learned weights or input features. For all models, the LRP-rule is applied to the first convolutional layer to smooth the attribution map in input space and yield robust concept localization.

A.2. Explanation Examples Using L-CRP
In the following, two additional L-CRP explanation examples are shown, which extend the LRP heatmaps shown in Figure 2.
The first example shown in Figure A.1 corresponds to the sheep detection of the YOLOv5 model in Figure 2. Here, the traditional LRP heatmap indicates that parts of the sheep are relevant, such as the head or the fur part in the center of the bounding box. By applying L-CRP to layer layer4.0.conv3, we achieve an understanding of what exactly is the model using in terms of concepts here. By analyzing the top-2 most relevant concepts, we learn that the model perceives, e.g., the white fur pattern of the sheep (concept 450) or the frontal face with ears protruding on both sides (concept 173).
The second example shown in Figure  exactly is the model using in terms of concepts. Investigating the top-2 most relevant concepts, shows that the model perceives, e.g., the wheels of the bus (concept 1254) or the school bus color combination of yellow with black (concept 599).

A.3. Faithfulness
In Section 4.1.1, we propose to measure the faithfulness of concept attributions using various approaches based on relevance or activation. In this section, more detailed results for the discussion in Section 4.1.1 are presented. Specifically, we show additional results for the concept flipping and concept insertion experiment for different layers of the UNet, DeepLabV3+, YOLOv5 and YOLOv6 model in Figure A.3. Here, every convolutional layer of the UNet, every second convolutional layer of the DeepLabV3+, and every fourth convolutional layer of the YOLO architectures is analyzed, resulting in approximately 20 layers for each model. Here, the AOC or AUC over the resulting curves are measured for 100 randomly chosen predictions, and the mean values are plotted with the SEM in semi-transparent color. The overall AOC or AUC scores (of all layers) are given in parenthesis. Figure A.3 visualizes that the AOC or AUC scores can vary strongly between layers of an attribution method. This is due to shortcut connections in the network, illustrating that some layers are used more strongly than others. Taking the UNet as an example, it shows that the further the layer in the encoder (up to layer 10), the lower the overall relevance. This indicates, that a large part of features is detected using the lower-level layers.

A.4. Complexity
In Section 4.1.2, we measure and discuss the complexity of concept attribution scores for relevance and activationbased approaches by two means. Firstly, the standard deviation of class concept attribution scores is computed, and secondly, the amount of concepts needed to form 80 % of all attributions is measured. The lower the variation and the smaller the number of concepts to study, the easier it is to understand concept-based explanations.
In the following, we present in Figure A.5 more detailed results of the complexity analysis discussed in Section 4.1.2. Here, the same layers of the models as in previous Section A.3 are investigated. Regarding the analysis, concept attribution scores are collected over all predictions of the test dataset. Thereafter, in order to be interpretable as percentage scores, concept attributions are normalized to an absolute sum of one.
In the first experiment, the mean standard deviation σ t of concept attribution scores R j (x i ) for each class t is measured as with mean attributionR j = 1 ns ns i R j (x i ) over n s class samples and n c concepts. To form a final deviation score σ, the mean over all n t classes is computed as σ = 1 nt nt t σ t . As shown in Figure A.5, it is visible, that especially in lower-level layers gradient and LRP-ε attributions tend to be noisy, which is expected, as this has also been observed for input-level heatmaps [52]. However, gradient and LRP-ε can also show noisy attributions in higher-level layers, e.g., in layer 20 of the UNet model. This indicates, that the commonly applied heuristic [43] for LRP heatmaps to use more faithful methods such as LRP-ε in higher-level layers, and more stable, but less faithful methods (e.g. LRP-z + ) in lower-level layer, might not always lead to stable conceptbased explanations.
In the second experiment, the fraction of concepts forming 80 % of attributions is calculated. Here, attributions are firstly sorted according to their magnitude in descending order. Thereafter, the cumulative distribution is computed, and the smallest number of channels computed for which the cumulative value is smaller than 80 %. Finally, the fraction is computed via division by the total number of channels. This value depends, e.g., on the feature specificity of concepts. For example, relevances are rather focused on a small set of concepts if they have a very specific function, compared to when all concepts are rather generic, leading to a uniform relevance distribution as all concepts are being used.

A.5. Context
In Section 4.2, we compute context scores C of concepts by measuring the number of concept attributions that correspond to the background of detected objects in the spatial dimension, as described by Equation (8). We firstly evaluate the computed context scores comparing the use of latent activation maps, latent relevance maps, and L-CRP heatmaps, and thereafter show how to interact with the model and test its reliance on background features. For both parts, we in the following present additional information.
Context Score Evaluation The context scores are evaluated by computing background sensitivity scores S for each concept via Equation (9). The background sensitivity is hereby measured by perturbing the background of objects and tracking the change in concept attribution.
Ideally, we expect that concepts with a high context score C will also result in a high background sensitivity S. Therefore, we assume of faithful context scores to result in a high correlation ρ as well as a small Root Mean Square Deviation (RMSD) value between context and background sensitivity scores. The correlation ρ is calculated as The final correlation and RMSD values per model shown in Table 2 are computed by taking the mean over three layers. For a visualization of the resulting similarity between context scores and background sensitivity values, distribution plots are shown in Figure A.5. Here, it becomes apparent, that using latent activation leads to an over-estimation of context scores, as they are often significantly higher than the corresponding background sensitivity values, as, e.g., for the three layers of the UNet shown in Figure A.5 (1st row). It can also be seen, that the higher-level layers show the best alignment between C and S values (3rd column), as features are more specialized, becoming better to characterize as either background or foreground concepts.

Context-based Model Interaction
In the background concept flipping experiments in Section 4.2, we interact with the model based on identified (by the model used) background concepts. Therefore, three background concepts are flipped successively and the change in the predicted object logit measured.
In Figure A.6 we show the identified background concepts in detail with reference samples conditioned on the respective target. All concepts correspond to layer ERBlock 5.0.rbr dense of the YOLOv6 model. Please note, that concepts are flipped in the corresponding order as in Figure A.6 from top to bottom.
Interestingly, the perturbation of three concepts of in total 512 leads to missed surfing board predictions. In