extfieldsofstudy:
Engineering
Computer Science

text:
Adaptive Template Enhancement for Improved Person Recognition using Small Datasets

A novel instance-based method for the classification of electroencephalography (EEG) signals is presented and evaluated in this paper. The non-stationary nature of the EEG signals, coupled with the demanding task of pattern recognition with limited training data as well as the potentially noisy signal acquisition conditions, have motivated the work reported in this study. The proposed adaptive template enhancement mechanism transforms the feature-level instances by treating each feature dimension separately, hence resulting in improved class separation and better query-class matching. The proposed new instance-based learning algorithm is compared with a few related algorithms in a number of scenarios. A clinical grade 64-electrode EEG database, as well as a low-quality (high-noise level) EEG database obtained with a low-cost system using a single dry sensor have been used for evaluations in biometric person recognition. The proposed approach demonstrates significantly improved classification accuracy in both identification and verification scenarios. In particular, this new method is seen to provide a good classification performance for noisy EEG data, indicating its potential suitability for a wide range of applications.

I. INTRODUCTION
ITH the recent developments in the field of automatic pattern recognition, particularly the thriving area of deep learning techniques, pattern classification for non-stationary time-series modalities (such as natural language processing and speech recognition) have achieved considerable improvements [1] [2]. However, pattern recognition for complex neurological signals such as electroencephalography (EEG) remains a challenging topic, and has been receiving an increasing attention in the research community [3]. In this paper, we propose a novel instance-based classification method for EEG pattern recognition in biometrics applications. To demonstrate its generality, the method is firstly evaluated using a public benchmarking dataset. EEG classification with a limited amount of training data has been also employed to further explore the effectiveness of the proposed algorithm in dealing with challenging pattern recognition problems.
Compared to eager learning algorithms which perform explicit generalization [4] (e.g., neural networks), one advantage of instance-based learning is its ability to flexibly model the training set by using only a subset of it, hence potentially provide better local approximations. As one important branch of machine learning, instance-based learning is designed to compare the query instance(s) with training instances, and store the results of these comparisons for subsequent decision-making [5]. For example, in a scenario where the size of the training data is dynamic (e.g., additional data enrolment), the templates of interest could be updated adaptably when using instance-based methods, the previously optimized performance may then be well maintained. However, increasing the number of templates without a selection scheme also introduces complexity to the system. Various instance reduction and generation algorithms have been developed to deal with such shortcomings of this type of algorithms [6] [7]. The development of these template (prototype) optimization algorithms has had a number of aims, including 1) speed increase, 2) storage minimisation, 3) improving the learning speed and 4) improved generalization accuracy [6].
One of the earliest instance reduction algorithms is based on a selection rule namely condensed nearest neighbour (CNN) [8]. The traditional CNN, however, may not be able to correctly classify some of the instances due to the equal-weighted evaluation in the feature space [9], given the center(s) of the feature cluster(s) is(are) often considered to be heavily weighted than the outliers in classification. Efforts have been made since then to address and improve various aspects of the instance-based algorithms [6][10] [11]. Typical instance-based learning algorithms where instance selection is paramount include the k-Nearest Neighbour (k-NN) and the Support Vector Machine (SVM) algorithms [9].
Limited availability of training data has long been a challenge for many pattern recognition problems. This may lead to under-generalization of the feature space and as such, result in poor classification performance in real-world scenarios where data volume is much larger than in the experimental scenario. To alleviate this, the algorithm proposed in this work aims to improve the pattern recognition through a multi-stage adaptive mechanism, incorporating by generating new templates through the reconstruction of the existing template data.
The within-class and between-class similarities are critical factors for effective classifications [12]. High between-class similarity may result in a high false positive rate while low within-class similarity may produce a high false negative classification rate. To achieve good recognition performance, instance distributions may be examined to isolate and select key instances with lower between-class similarity and higher within-class similarity. Such subsets of instances can then be used as building blocks to further remodel and provide compact representations for each class. This idea is further explored and implemented using the proposed technique, by subjecting the reconstruction process to be the mutually influenced by the query instance and the training templates, which are simultaneously remodeled for every classification attempt.
Certain time-series data such as EEG signals are characterized by their non-stationary behavior [12][13]: the signal statistics for the same class is often unstable and varies over time. The within-class variance over time, such as longitudinal template ageing effects in person recognition systems, is a great impacting factor in EEG-based pattern recognition. Based on our experimental investigations and other related reports, even an interval of several minutes between independent recordings could cause considerable template variations [3]. Therefore, data-driven optimum instance generation may play an important role for concurrently achieving both satisfactory within-class as well as betweenclass discriminations. Therefore, to explore the full potential of the proposed method, EEG signals, with their time-series nature, provide a suitable test case. This is the main motive of choosing EEG data for the algorithm evaluation in this work. Two distinctively different EEG databases are used to evaluate the effectiveness of the proposed instance-based template reconstruction learning algorithm.
The paper is structured as follows: The detailed description of the algorithm is given in Section II. The performance of the proposed method is evaluated in Section III, including a number of pattern recognition scenarios using EEG, along with a brief comparison with the optimized k-Nearest Neighbours and Support Vector Machine for benchmarking. Section IV provides a discussion of the results followed by the suggestions for future work.

II. INSTANCE-BASED TEMPLATE REGENERATION
The Instance-based Adaptive Template Reconstruction (I-ATR) algorithm proposed here consists of two phases (cf. Figure 1). Phase 1 (Training phase) generates intermediate templates by reconstructing/transforming the initial templates, so that the between-class separation is enhanced. Phase 2 (Test phase) further transforms these intermediate templates based on the query patterns, whilst transforming the query patterns in the meantime, to establish the best class assignment for the given query. This is accomplished by separately reconstructing the intermediate templates for each class, in turn, so that the distance of the query from the nearest class template is the smallest achievable. Each query may be composed of several feature vectors captured from segments of the data recording under observation. This set of instances would then be amalgamated into a template before being assigned to one of the possible classes. The reconstruction of the query is determined by comparing the shortest distances between the reconstructed templates and the transformed query for all the candidate classes in turn. The classification decision would then be determined for the transformed patterns showing the best match.
The following describes the proposed algorithm in more detail. Consider an N-class classification problem. T denotes the instances of the training set for this problem, the query (test) instances are denoted as , and the instances per class as ( ), where n indicates the class label (1 ≤ n ≤ N). Each instance is a L-dimensional feature vector, while each class may contain a different number of instances. A three-dimensional matrix , , , is used to represent the data elements of the feature vector in the training set. Here denotes the index of an instance, n denotes the class to which that instance belongs, and l denotes the feature dimension index (1 ≤ l ≤ L) (cf. Figure 2). , , (1 ≤ r ≤ R), represented the query data where R is the number of feature vectors extracted from one data recording. In a typical pattern classification task, R = 1; but for time-series data (such as EEG), features can be extracted from windows/epochs of the same recording; thus, resulting in multiple feature vectors from one recording (R > 1).
Phase 1 of the proposed I-ATR algorithm is devoted to maximising the between-class separation of the training set. Within each class, n, and for each feature element, l, of a training instance (feature vector), i, within that class, the average distances, ̅ , , , between it and the corresponding feature elements of the training feature vectors of the other classes are exhaustively measured, i.e., step 1) -step 2) in Block In the proposed method the distances between elements of each dimension of the feature vectors from different classes are measured, instead of the distance between the feature vectors. It may be worth noting that the Phase 1 of the algorithm has some similarities with the computation of Hausdorff Distance [14]. The reconstructed instances in ′ , , (here, 1 ≤ i ≤ K) are created to have better between-class separation. As different feature vectors from the original set , , are used to reconstruct/transform these feature instances, the intermediate templates in ′ , , may not have exact equivalents in the original set of feature vectors. However, these reconstructed vectors will always be on the surfaces of, or within the bounding hyper-plane of the original training data set. Phase 2 of the proposed method reconstructs the intermediate templates further, in conjunction with the query vectors, to produce " , , to minimise the distances between " , , and the transformed query feature vectors , . When multiple instances from the same query observation are available (such as EEG data), the adaptive reconstruction is also applied to the instances from the query set, producing transformed query vectors ′ , . As will be demonstrated in Section 3, this phase appears to be effective in alleviating the template longitudinal variations in person recognition using EEG signals.
As in Phase 1, distances , , between the intermediate template , , ′ for each candidate class and the query patterns , are measured in Step 5), Block 2. However, unlike Phase 1, only the feature elements contributing the smallest a few distances are retained to form the newly transformed feature vectors , ′′ and , ′ , the process is illustrated by the Step 6) - Step 9) in Block 2. The distance between , ′′ and , ′ represents the score for the class under consideration. The process is repeated for each class, thus ensuring the best possible match between the training and the test data for each class without bias. The label of the class achieving the lowest distance score(s) (hence, most similar) is assigned to the query // any appropriate distance metric may be used here. In this work, distance (x, y) = |x-y| is used.
2) Compute the mean of the resulting distances: End of for-loops. pattern, as is shown by Step 10). The full algorithmic process for Phase 2 is illustrated in Block 2.
A few features can be distilled from the proposed I-ATR algorithm: it has been pointed in [15], if enough knowledge is available about what make the samples similar/dissimilar in the problem domain, the instance-based methods could be a good choice. In this context, the original samples are purposely reconstructed based on explicitly defined rules (Block 1&2) in the proposed work. This is the motive of performing template reconstruction within the instance-based learning framework.
In addition, it is observed that as the data volume and dimensionality increasing, the mass of a multivariate Gaussian distributions are pushed toward to the distant surface of the hypersphere, which makes many similarity measurements no longer effective [15]. It is proposed in this work to directly construct / generate new templates from the existing data pool, to bypass this predicament with expanded feature extensions in the hyperspace. Studies have also indicated that the rectilinear distance metric (L1 norm) is consistently more preferable than distances with higher norms, such as the Euclidean distance metric (L2 norm) for high dimensional data mining applications [16]. Therefore, without violating the Triangle Inequality, the L1 norm distance measurement between elements from different instances (or feature vectors) is used in the I-ATR algorithm to construct new templates.
A hyper bounding box is established in the Cartesian space subsequently. Within this hyper bounding box: each dimension / attribute from the newly constructed templates is to exhibit the lowest between-class similarities possible, i.e., features distribute around the upper boundary of the hypersurface (after Phase 1); as well as the shortest distances possible between the reconstructed query and template feature vectors for all the training classes, i.e., features distribute around the closest boundaries between the training and query hypersurfaces (in Phase 2). It also worth to point out, though the distance of the reconstructed vectors through the proposed mechanism were directly measured in this study, they may also be effective for other classification algorithms, such as neural nets and linear classifiers.

III. CASE STUDY EVALUATIONS
In this section, the proposed method is evaluated using EEG data to explore its robustness across different application scenarios. Two major subsections are subsequently presented, each devoted to evaluating the I-ATR classification method using two EEG databases respectively.
Evaluation studies are presented to assess the effectiveness of the proposed method for the classification of 1-D time series data. Unlike the 2-D images, the EEG signal in particular, due to its nonstationary nature, could be more challenging for successful pattern recognition. In this section, two EEG databases, one publicly available, the other self-collected are used to further evaluate the proposed method. Section 3.1 reports the evaluations based on a publicly available database collected using BCI2000 instrumentation system [17], namely the EEG Motor Movement/Imagery Dataset (EEG MM/I) [18]. The results for biometric person recognition from 105 selected classes (subjects) from the MM/I database are presented.
In Section 3.2, a more challenging dataset namely Mobile Sensor Database is used to investigate the efficacy of the proposed algorithm for person recognition, where the signal quality is low and only a relatively small amount of training data is available. The impact of the template ageing effect due to multiple sessions with substantial time intervals is also addressed in the last set of the experiments.

MM/I Dataset for Evaluation of Biometric Recognition
This section explores the effectiveness of the I-ATR algorithm in person recognition problem, the results are compared with optimised 1-NN and SVM algorithms. To ensure at least 2 minutes of recording is available for each subject, 105 subjects (classes) were selected from the MM/I dataset. The studies reported in this section used the wavelet-based features were described in [19].
In the work reported here, to simulate application scenarios where ease of deployment and fast data processing is important, only the data from the Oz electrode of the MM/I dataset is used purposely. The experimental arrangement is briefly described below: 1) The EEG signals of four mental tasks (T1-T4) were segmented into multiple 4-second overlapping epochs (50% overlapping). 2) The wavelet packet decomposition (WPD) [20] was carried out for each time-domain window up to level-3. The resulting wavelet coefficients from level-3 between 0 and 60 Hz were retained for feature extraction. Every generated feature vector had six dimensions (each dimension approximately corresponding to a bandwidth of 10 Hz).
3) The variance of the wavelet coefficients in each epoch was used as a feature. 4) The I-ATR algorithm was then invoked for the process of template generation and classification.

Parameter Optimization
Prior to the invocation of the I-ATR algorithm, the EEG recordings were segmented into 240 windows ( = 240). Note, the training set contained two two-minute recordings for each run, each accounted for 120 windows. The highest recognition rate was achieved while the parameter = 160, K indicates the Fig. 3. Preliminary tests for optimizing P%, the analyze was done by dividing the training set into six roughly equal size portions, the preserved data for training is incrementally tested. Classification Precision amount of reconstructed and preserved feature templates for the Phase-1 training. Here the optimal proportion of the reconstructed instances to retain is empirically established, particularly for person recognition applications using the EEG MM/I dataset.
Recognition accuracies were calculated by the leave-one-out three-fold cross-validation approach using in total 6 minutes of the recording from each task. The volume of the retained newly constructed feature vectors is defined as P% (of the available total) which corresponds to the absolute parameter K in Step 4 of Phase 1 shown in Block 1. The recognition results for various P are illustrated in Figure 3. Figure 3 shows the change in classification accuracy while P% of the intermediate templates were preserved in the Phase 1 of the I-ATR algorithm, these intermediate templates were reconstructed from the initial instances after feature extraction (240 windows/observations). As a preliminary exploration for parameter optimisation, here only a small subset of the dataset (S1 -S10) was used to investigate the overall impact of the retained volume of the reconstructed feature instances for the follow-up Phase 2 processing. This is also consistent with some real-world training constraints where availability of data would be limited. The impact of P% on classification performance was investigated where only the Phase 1 of the proposed algorithm was applied (as depicted on the horizontal axis in Figure 3).
Results indicate that higher performances are achieved when the retained volume of reconstructed intermediate templates is about half to 2/3rd of the available training data. Therefore, as a rule of thumb, roughly 2/3 of the reconstructed templates will be retained for the next stage of the algorithm. This parameter should be dependent on the quality of the database and the application scenario: served as a data quality indicator, for lowquality and noisy signals such as the Mobile Sensor Database, a different value of P may be optimum.

Person Identification using MM/I Dataset
Following the empirical optimisation of parameter P, only about 2/3 of the training subset of the MM/I dataset samples in their reconstructed form are retained for subsequent evaluation of the proposed method in Phase 2. Comparisons are also made with the classical k-NN (k = 1) and nonlinear SVM [21], as both of them are popular instance-based classification algorithms. Figure 4 presents the resulting cumulative match characteristic (CMC) [22] curves for person recognition using EEG MM/I dataset. The I-ATR algorithm is seen to result in the best top-1 identification rate of 92.6%. The 1-NN classifier produced the higher recognition rates only after rank-13. Both the SVM and 1-NN had less than 50% accuracy. It is worth to mention that we did not conduct performance comparisons with deep neural nets in this case study for two main reasons: 1) the available data per class for the EEG MM/I dataset is not large enough for the training to converge for this classification problem, 2) the Mobile Sensor Database used for comparison is an even smaller and more challenging cohort. Figure 5 presents the performance comparison of the three algorithms using the detection error trade-off (DET) characteristics curves [23]. From the perspective of biometric person verification, Phase 1 and Phase 2 of the I-ATR algorithm can be viewed as a mechanism to reduce the false acceptance rate (Phase 1, reducing the between-class overlaps in the training set) and false rejection rate (Phase 2, enhancing the provisional within-class matching between training classes and the query), respectively.

Person Verification using MM/I Dataset
For the I-ATR algorithm, each feature vector was generated/reconstructed from 4-second recordings of the EEG signal, which producing 640 samples at 160 Hz. However, for the SVM and 1-NN algorithms, 6400 samples per window was used to maintain their best performances. Figure 5 shows an approximately 2% averaged equal error rate (EER) for the I-ATR algorithm whereas for the 1-NN and SVM classifiers the averaged EER values were approximately 10% and 14% respectively.

Performance using Mobile Sensor Database
The person recognition performance of the I-ATR algorithm was also tested using the locally collected Mobile Sensor Database to show the impact of a more challenging scenario. This database was purposely collected to explore the EEG template ageing effect in biometric scenarios. The experimental  scheme was aimed to mimic the real-life practice: with low-cost single dry sensor and relatively long time-interval (three to eight weeks) between the sessions. The EEG data recorded from 30 participants (classes) were used for this study. For the multi-session evaluation, 4 seconds of continuous data chunks were randomly selected from each 1-minute recording of one session to form a set of the query attempts, and all the data from the other session was used for training the system. The key features of this database are as follows: 1) Data was captured by a gaming-grade single (Fp1 positioning electrode) dry sensor system (NeuroSky MindWave [24]), designed for ease of deployment. 2) EEG records were collected from 30 individuals (age ranges from 21 to 55) without any neurological background. 3) Subjects were required to engage in a simple sub-vocal number-counting activity (with eyes closed), i.e., the subjects sit in a silent room counting numbers from 1 onwards, the EEG data were recorded in the meantime. 4) Data was collected in two sessions with the time interval between the sessions ranging from three to eight weeks. A well-established feature, based on wavelet transform was employed for the analyses presented in this study [19].

Longitudinal Template Ageing Effect
The I-ATR algorithm was in part motivated by the challenge of the template ageing effect in EEG biometric recognition systems, especially where only limited data is available for training [3]. In this section, the impact of using EEG data from different sessions with respect to different classification algorithms are presented. This is highlighted by evaluating the effectiveness of different I-ATR phases in dealing with such template variation phenomenon. Table 1 shows the impact of template ageing when using EEG recordings from the Mobile Sensor Database for person identification. The same parameters as in the last section were used for the 1-NN and SVM classifiers, and their best performances for this database are reported for comparison. For the single-session scenario, the 60 seconds' recording was split into five non-overlapping segments to enable cross-validation. The average accuracies when using SVM and 1-NN classifiers have been found to be comparable at around 93%.
In the case of multi-session data analysis, the entire Session 1 data (60 seconds) was used for training. A randomly selected segment of the data from Session 2 was then used for testing. The training-testing order was then swapped, and the classification results averaged. The impact of template ageing is found to be quite significant, as is shown in Table 1. Only around 10% recognition accuracy were achieved with the conventional instance-based classifiers.
The impact of the two phases with respect to the proposed I-ATR algorithm has also been noticed. As it is illustrated in Table 1, with only the Phase 1, the classification performance in the double-session case (for template ageing) shown substantial improvement compared to 1-NN and SVM with 2 nd order polynomial kernel. This indicates the great effectiveness of constructing new feature vectors with better between-class separation. By only applying Phase 2, similar considerable improvement was also observed; however, due to the absence of Phase 1 to reduce the potential between-class similarity, the false acceptances were not effectively limited, as the feature instances between the classes in the training set were not wellseparated. It is clear from the results that when both the phases are employed, performance revealed drastic improvement, in particular for the multi-session scenario.

Mobile Sensor Database in Identification Scenario
For the biometric identification scenario, the CMC results of the I-ATR algorithm are shown in Figure 6. The recognition results obtained using the conventional classifiers are also included for comparison.
The results for these plots were generated using the Session 1 EEG data (60 sec. recording) for classifier modelling and 4 sec. recordings from Session 2 as query sets. The initial parameter = 240, i.e., the total number of feature instances before template reconstructions. The finally retained number of instances in Phase 2 is = 4 for both the training and query sets. As it can be clearly observed in Figure 6, the I-ATR method provided much higher rank-1 accuracy (85.71%) than the conventional classifiers.

Mobile Sensor Database in Verification Scenario
The DET curves for the Mobile Sensor Dataset are presented in Figure 7. Due to the challenging experimental setup (limited and noisy EEG data from a low-cost single dry sensor, as well as cross-session test data), the EER values of all the illustrated algorithms were found to be not as low as those with the MM/I dataset ( Figure 5). However, it is clear that the I-ATR algorithm still produced much superior EER than the other two conventional instance-based classifiers. The EER of I-ATR method is 7.5%, whereas both the 1-NN and SVM schemes' EER values are higher than 30%. The results in Figure 5 and Figure 7 also highlight the difference in the data quality between the EEG MM/I dataset and the Mobile Sensor Database. The EER using I-ATR for the EEG MM/I database was as low as 2% whereas for the Mobile Sensor Database with only 30 subjects, it was about 7%. These analyses indicate the Mobile Sensor Database is indeed a much more challenging database for algorithm evaluation.

IV. DISCUSSION AND CONCLUSION
In this paper, a novel instance-based template reconstruction classification algorithm, namely the I-ATR recognition method was presented, and its underlying motivation as well as rationale explained. Extensive comparative evaluations using time series data (EEG) from different resources demonstrated the effectiveness and robustness of the proposed method. The I-ATR classification algorithm shows superior performance and surpassed the currently reported results for the respective databases in the explored application scenarios. Specifically, the performance of the I-ATR algorithm was compared with two other popular and closely related instance-based classification algorithms, k-NN and SVM, using two EEG databases with considerably different characteristics. The methods based on deep neural nets were purposely excluded for reporting in this study, as the model training is unable to successfully converge due to the rather limited amount of data volume for each class.
The proposed algorithm was also shown to have resulted in substantial performance improvements in different recognition scenarios. In particular, the resilience of the I-ATR to template ageing even with input samples of limited recording length (2 to 6 minutes) was notable. It is thanks to the proposed approach potentially generates a substantially larger volume of reconstructed data from the limited available data, and subsequently a small subset with reconstructed features is retained to ensure the superior class separation. The intermediate templates (Phase 1 and Phase 2) help to overcome the challenge of limited data.
Compared to conventional classification approaches, the major difference of the proposed method lies in the elementwise template reconstruction for improved class separation, followed by the decision-making based on reconstructed feature vectors. In this approach, each feature dimension is treated independently and thus the space of all possible feature vectors that could be reconstructed is vastly increased. It is a fundamental assumption of this approach that this expanded space is nevertheless broadly representative of the underlying distribution for each class. Furthermore, it is assumed that the resulting feature reconstructions based on this expanded class representations result in new feature vectors that can be better separated using the vector-wise Euclidian distance calculations adopted for classification. The proposed approach shares certain similarities with some recent pattern recognition algorithms in its motivation and conceptual principles. In the image processing field, a number of algorithms have recently been proposed to deal with the issue of pedestrian image alignment [25]. The alignment between the query image and the image in the database is one of the major challenges due to the variant camera angles and pedestrian postures. The pose attention transform [26] and the alignment affine transform [27] were recently developed to construct new images based on the query, which provided better matches in the database to achieve improved identification performances. The proposed I-ATR method also confines the reconstruction within the "bounding box" region [27] in order to guarantee the outcome of the feature transformations does not lose the ability to represent the original signal source.
At the same time, in contrast to the approaches implemented in the image/video processing field, the proposed template reconstruction method is motivated by some special requirements of bio-signals (such as EEG). The size of many publicly annotated EEG data is much smaller compared to the vast quantity of image data available for machine learning. Therefore, as opposed to the usage of deep nets in [26], [27], the I-ATR method is designed to leverage a relatively small amount of data, from which to produce high-quality features in order to achieve improved recognition performance.
Another distinguishing characteristic of the I-ATR algorithm is that it reconstructs the feature vectors of both the training set and the query for each recognition event, thus allowing improved flexibility in controlling the system performance (in terms of sensitivity vs. specificity), as required for specific application scenarios. The proposed approach has, therefore, the potential for applications in other domains using different modalities.
One focus of further work will be to establish the wider applicability of the proposed approach using other databases, including those from domains other than time-series data. Also, attention will be paid to the optimization of the algorithm through the adjustment of its parameters and its sensitivity to outliers. Deep learning methods may be incorporated with the proposed I-ATR algorithm to further improve the performance of the proposed approach.


Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems that are separated by small differences in automatic scores which are commonly observed in practice. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. The results from the analyses point to the need to collect more high-quality human judgments and to improve automatic metrics when differences in system scores are small.

Introduction
Automatic evaluation metrics are the most common method that researchers use to quickly and cheaply approximate how humans would rate the quality of a summarization system (Lin, 2004;Louis and Nenkova, 2013;Zhao et al., 2019;Zhang et al., 2020;Deutsch et al., 2021a, among others). The quality of a metric -how similarly it replicates human judgments of systems -is quantified by calculating the correlation between the metric's scores and human judgments on a set of systems, known as the system-level correlation (Louis and Nenkova, 2013;Deutsch et al., 2021b).
Accurately estimating system-level correlations is critically important. Summarization researchers use automatic metrics during system development to make decisions about which ideas work and which do not, and systems from different research groups are ranked by automatic metrics to define which system is the "state-of-the-art." If we do not have precise estimates of metric quality, it is not clear how much trust the community should put in such evaluation methodologies.
At present, there are disconnects between how automatic metrics are evaluated and how they are used to evaluate systems. First, the metrics' scores which are used in practice are not the ones which are evaluated in system-level correlations; researchers compare systems based on metric scores calculated on the entire test set but calculate scores for system-level correlations when evaluating metrics on a much smaller subset of judged summaries. Second, metrics are evaluated in a setting that is much easier than how they are actually used. Metric correlations are calculated using systems that vary greatly in quality, whereas researchers compare new systems to recent work, which are likely to be very close in quality. Discriminating between two systems of similar quality is much harder than doing so between low and high quality systems.
In this work, we re-examine how system-level correlations are calculated and propose two independent changes to make the evaluation of metrics better aligned to how they are actually used to evaluate systems.
First, we propose to modify the system-level correlation definition to use the entire test set to calculate the system scores for automatic metrics instead of only the subset of summaries judged by humans ( §3). With this change, the scores which are used to compare systems are directly evaluated, and we further demonstrate how the precision of our estimate of system-level correlations improves as a result. Calculating system scores over a larger number of instances reduces the variance of the scores, which results in confidence intervals (CIs) for the correlations that are 16-51% more narrow on average ( §3.2).
Second, we redefine a high quality metric to be one for which a small difference in score reliably indicates a difference in quality ( §4). Then, instead of calculating the correlation with all available system pairs, we only evaluate with pairs of systems whose automatic metric scores differ by some threshold. This allows us to show, for example, that a ROUGE-1 score difference of less than 0.5 between systems has almost no correlation to how humans would rank the same two systems according to our best estimates ( §4.2). For two other metrics, BERTScore (Zhang et al., 2020) and QAEval (Deutsch et al., 2021a), we show their correlations calculated on system pairs of similar quality are much worse than under the standard correlation definition. These results cast doubt on how reliable automatic evaluation metrics are for measuring summarization system quality in realistic scenarios.
Our analyses point to the need to collect more high-quality human judgments of summaries in order to have more accurate estimates of metric correlations as well as the need to improve the ability of automatic metrics to discriminate between similarly performing systems.

Background
Automatic evaluation metrics are most commonly used to argue that one summarization system is better than another, typically by showing that the value of a metric improves with the "better" system. How similarly automatic metrics replicate human judgments of system quality is quantified by system-level correlations as follows.
The summaries from N systems on M jud input documents are judged by humans Z and scored with an automatic metric X . Then, the systemlevel correlation between X and Z is calculated as where x j i and z j i are the scores of X and Z for the summary produced by the i-th system on the j-th input document and CORR is some correlation function. See Fig. 1 for an illustration of this calculation. Figure 1: The system-level correlation is calculated between the average X and Z scores on a set of summarization systems. x j i and z j i are the scores for the summary produced by system i (represented by rows) on input document j (represented by columns).
In this work, we use Kendall's τ (the "b" variant 2 ) as the correlation function because we are most concerned with a metric's ability to correctly determine whether one system is better than another since that is how metrics are used in practice. Kendall's τ is computed based on the number of system pairs out of N 2 which are ranked the same by X and Z. It is defined as where P and Q are the number of pairs ranked the same or different by X and Z, respectively, and T and U are the number of ties only in X or Z, respectively. Because the computation of r SYS involves randomness -its value depends on which M jud input documents (and even which N systems) were used -it is only an approximation of the true correlation between X and Z. As such, Deutsch et al. (2021b) proposed various methods for calculating confidence intervals for r SYS . For instance, their BOOT-INPUTS method uses bootstrapping to repeatedly resample the M jud input documents used to calculate r SYS , thereby calculating a confidence interval for the true r SYS value for X and Z.
Datasets The datasets that are used in this paper's analyses are SummEval (Fabbri et al., 2021) and REALSumm (Bhandari et al., 2020), two recently collected datasets with human annotations for summary quality collected from the CNN/DailyMail dataset (Nallapati et al., 2016). SummEval has M jud = 100 summaries annotated with a summary relevance score for N = 16 systems. REALSumm has M jud = 100 summaries annotated with a Lightweight Pyramid score (Shapira et al., 2019) for N = 25 systems. We correlate the scores of the automatic metrics to these annotations. The CNN/DailyMail test split has 11, 490 instances.

Automatic Metrics
Our experiments will analyze three different reference-based automatic evaluation metrics which were chosen because they were demonstrated to have the best correlations with human judgments on the SummEval and REALSumm datasets (Deutsch et al., 2021b). ROUGE-n (Lin, 2004) evaluates a generated summary by calculating an F 1 score on the number of ngrams it has in common with a human-written reference summary. BERTScore (Zhang et al., 2020) aligns the generated and reference summaries' tokens based on their BERT embeddings (Devlin et al., 2019) and calculates a score based on the similarity of the aligned tokens' embeddings. QA-Eval (Deutsch et al., 2021a) compares the two summaries by automatically generating questions from the reference and calculating what proportion of those questions are answered correctly by the generated summary.

Evaluating with All Available Instances
Although the above definition of the system-level correlation has been used by recent meta-evaluation studies of metrics (Bhandari et al., 2020;Fabbri et al., 2021;Deutsch et al., 2021b), there is a disconnect between how the automatic metrics are evaluated and how they are used in practice.
Researchers who develop summarization systems evaluate those systems with automatic metrics on all M test test instances, not just the subset of M jud instances which were judged by humans. Evaluating a system on a larger number of summaries may end up changing the system's score, which could potentially alter the overall ranking of a set of systems. Therefore, the rankings that are used by practitioners to determine system quality are not the ones which are being evaluated in the standard definition of system-level correlation. 3 Systems 87.0 87.5 88.0 88.5 89.0 89.5 BERTScore Judged Instances All Test Instances Figure 2: The bootstrapped 95% confidence intervals for the BERTScore of each system in the REALSumm dataset using M jud judged instances in blue and M test instances in orange. Evaluating systems with M test instances leads to far better estimate of their true scores.
To that end, we propose to modify the correlation definition to use all M test instances to calculate the system scores for the automatic metrics. That is (differences in bold): In practice with modern, large-scale datasets, this minor change could mean estimating system quality based on ≈10k inputs instead of around 100. This new definition now properly evaluates the way metrics are actually used by researchers. We expect that scoring systems with M test inputs instead of M jud should lead to a better estimate of the true automatic metric score, which would in turn result in a lower-variance estimate of the correlation between X and Z in the form of smaller confidence intervals for r SYS . In the next sections, we carry out analyses to demonstrate that this is true.

Reducing Automatic Metric Variance
First, we empirically show that scoring systems with M test instances instead of M jud does indeed reduce the variance of the estimate of the automatic metric scores and subsequently increases the stabilities of the system rankings.
Ideally, the X score for a system would be its "oracle" X score, equal to the expected value of X for a document sampled from the latent distribution over documents defined by the dataset (e.g., a system's ROUGE score on an infinite number of examples from a dataset). Since this cannot be calculated, it is approximated by averaging the X score on a sample (i.e., either the M jud or M test input documents). Because M test M jud , we expect that the variance of this estimate using M test inputs should be lower than when using M jud . Figure 3: Bootstrapped estimates of the stabilities of the system rankings for automatic metrics and human annotations on SummEval (left) and REALSumm (right). The τ value quantifies how similar two system rankings would be if they were computed with two random sets of M input documents. When all M test test instances are used, the automatic metrics' rankings become near constant. The error regions represent ±1 standard deviation.
To quantify this, we calculated the variance of estimating the oracle X score using both M jud and M test input documents via bootstrapping. We randomly sampled M input documents with replacement, recomputed the system scores, and calculated the variance of those scores over 1k iterations. For all three metrics on both datasets, we found around a 99% reduction in the variance when M test inputs were used instead of M jud , clearly demonstrating that evaluating systems with M test inputs results in a better estimate of the system scores. In Fig. 2, this is visualized for BERTScore on the REALSumm dataset.
However, because we are interested in evaluating the metrics' rankings, we also quantify how much of an effect this reduction in variance has on the stability of the system rankings induced by X . Similarly to the system scores, there is an oracle ranking of systems for X , equal to the ordering of systems by their respective oracle X scores (e.g., systems sorted by their ROUGE scores calculated on an infinite number of examples from a dataset). As the variance of the system score estimates decreases, the computed ranking of systems should begin to converge to the oracle X ranking. We aim to understand to what extent this happens if M test instances are used for evaluation instead of M jud .
To quantify this notion, we calculate the Kendall's τ between two system rankings for X that were based on two sets of M input documents, each sampled with replacement from the set of available documents. This simulates how much the system rankings would change if the evaluation procedure was run twice, each time with M random input documents. This quantity is calculated 1k times for various values of M and plotted in Fig. 3.
As M approaches M test , the automatic metrics' τ values approach 1, which is much higher than the respective values at M jud , typically around 0.6-0.8. A value near 1 means that the rankings calculated using M test inputs are almost constant, implying the rankings have converged to the oracle ranking. Therefore, the reduction in variance from evaluating on M test instances does indeed greatly stabilize the system rankings. Fig. 3 also contains the same analysis performed for the human judgments Z in both datasets, although it is limited to a maximum of M jud input documents. We see that on both datasets the judgments' rankings are still quite variable, reaching a maximum τ of around 0.8-0.85.

Confidence Interval Analysis
Next, we show that the improved estimate of system scores leads to a more precise estimate of r SYS by demonstrating the widths of the confidence intervals for r SYS decrease.
The confidence intervals for r SYS calculated using bootstrapping methods proposed by Deutsch et al. (2021b) are rather wide. For instance, the 95% CI for ROUGE-2 on SummEval is [−.09, .84], demonstrating a rather high level of uncertainty in its value. This is problematic because it means we do not have a good picture of how reliable automatic evaluation metrics are. Reducing the width of the CIs will help us better understand the true metric quality.
We suspect that the large width of the confidence interval is due to the variance of the system rank- ings of the automatic metrics and human judgments. The more unstable the rankings are with respect to the M inputs, the larger the variance of the estimate of r SYS should be since very different system rankings would be compared on each bootstrapping iteration. Deutsch et al. (2021b) used M jud input documents to calculate their CIs. Therefore, we expect the improved stability of the automatic metric system rankings from evaluating on M test instances should result in a more narrow confidence interval for r SYS since some noise has been removed from this computation.
To demonstrate this, we calculated 95% CIs for r SYS using the BOOT-INPUT method on SummEval and REALSumm using both M jud and M test input documents, shown in Fig. 4. We find that the widths of the CIs shrank on average by 51% on SummEval and 16% on REALSumm. The largest decrease in width is in the ROUGE family of metrics on SummEval, potentially because that metric and dataset combination saw the biggest improvement in ranking stability (see Fig. 3). Thus, the improved estimate of the system scores did result in more precise estimates of r SYS . We repeated this analysis using the other bootstrapping methods proposed by Deutsch et al. (2021b), and the results are discussed in Appendix A.

Conclusions & Recommendations
By estimating system quality using automatic metrics on all available instances instead of only those which were judged, we showed that the variances of the system scores and subsequent rankings reduce significantly, resulting in better estimates of r SYS . Because this methodology additionally directly evaluates the system scores used by researchers, we recommend future work do the same.
In order to continue to improve the estimate of r SYS , as much variance as possible needs to be removed from the system rankings. Evaluating systems using M test instances removed a large amount of variance from the automatic metric rankings, but as demonstrated in Fig. 3, the human judgments still have a large amount of variance.
The human rankings' variances can either be reduced by judging more summaries per system or making the judgments more consistent. Since the human rankings' stabilities in Fig. 3 are mostly beginning to plateau -especially for SummEval -it may be prohibitively expensive to collect a sufficient number of judgments to better stabilize the rankings (Wei and Jia, 2021). Therefore, we expect the more feasible solution is to improve the consistency of the human judgments, for example by better training the annotators or improving the annotation protocol.

Evaluating with Realistic System Pairs
Next, we argue that the set of systems used to evaluate metrics is not reflective of how metrics are used in practice and propose a new system-level correlation variant to address this problem.

Evaluating with All System Pairs
The N systems which are used for calculating system-level correlations are typically those which participated in a shared task, as in DUC/TAC (Dang and Owczarzak, 2008, among others), or those which have been published in the previous 3-4 years (Bhandari et al., 2020;Fabbri et al., 2021). As such, they are typically rather diverse in terms of their qualities, both as rated by human annotators and automatic metrics.
The system scores of all of the systems in the REALSumm dataset as evaluated by humans and automatic metrics are shown in Fig. 5. Clearly, the scores are rather diverse. For example, the systems cluster into low, medium, and high quality groups (with an additional outlier) as evaluated by ROUGE. A difference of around 5 ROUGE points between them is a rather large gap for ROUGE scores.
The standard definition for a high quality evaluation metric is one which correctly ranks a set of systems with respect to human judgments. As such, the implementation of the system-level correlation calculated with Kendall's τ will rank all N systems according to the human judgments and an automatic metric, then count how many pairs were ranked the same out of all N 2 pairs (see §2). As a consequence, even pairs of systems which are separated by a large margin according to the automatic metric -likely systems with a clear difference in quality -are included in the evaluation. Therefore, automatic metrics are rewarded for correctly ranking such "easy" system pairs.

Evaluating with Realistic Pairs
This standard evaluation setting does not reflect how summarization metrics are actually used by researchers. New systems are typically only slightly better than previous work. Based on a survey of recent summarization papers in *ACL conferences (see Appendix C), we found that the average improvement over baseline/state-of-the-art models that was reported on the CNN/Dailymail dataset was on average 0.5 ROUGE-1. It is rarely the case that the improvement in automatic metrics is very large. Therefore, evaluating metrics using pairs of systems which are separated by a large margin does not reflect the reality that metrics are very frequently used to compare those separated by a small margin. Including "easy" system pairs in the system-level correlation likely overestimates the quality of the metrics in settings which occur in practice.
To that end, we redefine a high quality evaluation metric to be one for which a small difference in scores reliably indicates a difference in quality. We quantify this by proposing a variant of the system-level τ which is calculated between system pairs which are separated by a pre-defined automatic metric score margin. Instead of using all N 2 system pairs, only pairs whose difference in scores falls within the margin are used to calculate the system-level correlation. We denote this correlation variant as r SYS ∆( , u) where and u are the lower-and upper-bounds of the allowable differences in automatic metrics' scores. This would enable, for example, evaluating how well ROUGE correlates to human judgments on system pairs that are separated by 0.0-0.5 ROUGE points, thereby directly evaluating the scenario in which ROUGE is used to make decisions about system quality.
In Fig. 6 we report the r SYS ∆( , u) correlations for = 0.0 and various values of u on both the SummEval and REALSumm datasets (more combinations of and u are included in Appendix B). That is, we evaluate r SYS only on system pairs which are separated by at most an automatic score of u. The values of u were selected by picking the minimum u which would result in evaluating on 10%, 20%, . . . , 100% of the N 2 possible system pairs closest in score to be consistent across all three metrics.
The correlations for each metric on the system pairs closest in score are far lower than the correlations evaluated on all of the system pairs. For instance, the correlation of BERTScore on Summ-Eval with the closest 20% of system pairs (u ≈ 0.2) is only 0.42 compared to 0.77 under the standard definition of r SYS . Thus, it is clear that the metrics are much less reliable approximations of human judgments when the system scores are close than was previously known. Evaluating on all possible system pairs leads to an overly optimistic view of automatic metric quality.
The r SYS ∆( , u) correlation of ROUGE for = 0.0 and u = 0.5 -a typical improvement reported by researchers -is 0.08 and 0.0 on the Summ-Eval and REALSumm datasets. Therefore, these results suggest the most popular summarization evaluation metric agrees with human judgments of system quality in realistic scenarios only slightly better than or equal to random chance. For instance, 20% of the N 2 system pairs on SummEval are separated by < 0.5 ROUGE-1, and the system-level correlation on those pairs is around 0.08. As more systems are used in the correlation calculation, the allowable gap in scores between system pairs increases, and are therefore likely easier to rank, resulting in higher correlations.
This result also offers an explanation for why a naive metric such as ROUGE achieves moderately strong correlations under the standard definition of the system-level correlation (0.45 and 0.73 on SummEval and REALSumm) despite well known flaws and criticisms (Passonneau et al., 2005;Conroy and Dang, 2008; Deutsch and Roth, 2020, among others): it has benefited from an easy evaluation protocol. Despite its simplicity, it is not too surprising that a large gap of 5-10 ROUGE points actually does correctly rank system pairs. Most of its positive correlation comes from such easy examples.

Conclusions & Recommendations
If it is assumed that we have enough high-quality judgments to accurately discriminate between two similarly performing systems, then the results in Fig. 6 show that the correlations in realistic settings are trending very low, meaning automatic metrics are not nearly sensitive enough to distinguish between systems with only minor differences in quality. This is problematic because this is the scenario in which metrics are most frequently used, and therefore they are not very reliable methods of evaluating summarization systems. However, it is not all bad news. Because the standard systemlevel τ values are moderately positive, consistent improvements in automatic metrics over time will likely result in better quality systems. Similarly to stochastic gradient descent, not every reported improvement is real, but on average over time, the quality does improve. Nonetheless, future work should focus on improving the quality of evaluation metrics when the differences in system performance are small, and researchers who compare systems should invest more effort into their human evaluations since automatic evaluations are not very reliable.
However, because the available number of system pairs to calculate the correlations in Fig. 6 is rather small -especially when evaluating on the closest system pairs -and recent work suggests we may not have enough human judgments to accurately distinguish between similarly performing systems (Wei and Jia, 2021), it could be difficult to reach any definitive conclusions about the metrics' correlations. That being said, these are our best estimates of the correlations with the available data. Not knowing how much we can trust automatic metrics is not a good outcome. In this scenario, future work should focus on collecting more, high-quality human judgments so that we can better meta-evaluate automatic metrics. Since we argue that it is important to distinguish between similarly performing systems, new data collection efforts should consider using targeted pairwise judgments between those systems instead of direct assessments across a variety of systems of diverse quality.
We recommend that proposals of new evaluation metrics also report correlations on system pairs with various differences in scores in addition to the standard system-level correlation definition. Re-porting this information would better inform users of metrics about how likely humans would agree their observed improvement is real based on its value.

Related Work
The methodology behind meta-evaluating summarization evaluation metrics was established during the DUC/TAC shared tasks (Dang and Owczarzak, 2008, among others). In addition to competitions for developing high-quality summarization systems, there were also shared tasks for creating automatic metrics that correlated well with human judgments. The benchmark datasets created during DUC/TAC were small in size by today's standards because they were manually collected multi-document summarization datasets, which are hard to create at scale. As such, all of the model-generated summaries on the full test set were judged (so M jud = M test ; §3), unlike for current datasets which are too large to fully judge.
Recently, there has been growing interest in revisiting the meta-evaluation of automatic evaluation metrics for summarization, in part due to the large differences between currently popular summarization datasets and those used in DUC/TAC. We view our work as continuing this direction of research, described next.
Peyrard (2019) argues that current evaluation metrics do not work as well when they are used to evaluate high-performing systems compared to those which were evaluated in DUC/TAC. Both Fabbri et al. (2021) and Bhandari et al. (2020) re-evaluated how well existing evaluation metrics work on the popular CNN/DailyMail dataset (Nallapati et al., 2016) by collecting judgments of summary quality using recent state-ofthe-art systems. These datasets were used in our analyses. While the goal of these works was to identify which metrics correlated best with human judgments, our goal is to point out the ways in which the current methodology of meta-evaluating metrics is inconsistent with how they are used.
Then, the work of Deutsch et al. (2021b) proposed statistical methods for estimating and comparing correlation values. In contrast to our work, they provide statistical tools for analyzing correlations, whereas we propose new definitions of correlations.
Finally, Wei and Jia (2021) provided a theoretical analysis of the bias and variance of automatic and human evaluations of machine translations and summaries. Among their conclusions, they argue for evaluating metrics with pairwise accuracy (Kendall's τ ) and that it may be prohibitively expensive to collect enough human judgments to distinguish between two systems with very similar quality. Our work further argues that metrics should be evaluated with a variant of Kendall's τ calculated using realistic system pairs ( §4). Unfortunately, their results suggest that collecting enough human judgments to accurately measure how well automatic metrics perform in this setting may be very difficult.
Related studies to ours have examined how the choice of which systems to include in metric evaluations impacts the correlation values. Both Mathur et al. (2020) and Bhandari et al. (2020) identify that metrics perform worse when scoring only the top-k systems in machine translation and summarization, respectively, and examine the use of pairwise comparisons for metric evaluation. Further, Mathur et al. (2020) demonstrate that outlier systems have an out-sized influence on the correlation values and recommend removing them from the metric evaluations. In contrast, our work proposes to change the evaluation methodology for metrics so that it more closely resembles how they are used in practice. This results in evaluating only on system pairs which are realistically compared by researchers, that is, those separated by small margins in automatic metric scores. We believe that this is a more principled approach to how to select which system pairs to evaluate on compared to previous work.

Conclusion
In this work, we proposed two independent changes to how the system-level correlation of metrics is calculated to better align with how they are used to evaluate systems. Our analyses showed that these modifications led to lower-variance estimates of correlations and that commonly reported improvements in metric scores may not reliably predict how humans would judge system quality. The results from the analyses point to the need for future data collection efforts of high-quality human judgments and improving automatic evaluation metrics when differences in system performance are small.

A Additional Confidence Interval Results
In of the paper, Fig. 4). The widths of the BOOT-BOTH CIs decreased by 14% and 12%, whereas the BOOT-SYSTEMS CIs only decreased by 1% and 6%.
The BOOT-SYSTEMS widths likely decreased less because its estimation of r SYS is not dependent on the variance of the system score estimates. Since the set of M input documents is fixed, the system scores do not change at all during bootstrapping, so increasing the number of summaries used to estimate those scores should not have a major effect on the estimation of r SYS . B Additional r SYS ∆( , u) Results = 0 for ROUGE-1, ROUGE-2, and ROUGE-L, equivalent to those shown in Fig. 6 in the main body of the paper (ROUGE-1 is shown in both). The ROUGE-2 and ROUGE-L results are largely consistent with those of ROUGE-1. The metrics' correlations to human annotations are low (or even negative) when the differences between system scores are small. As more pairs are added that differ by larger margins, the correlations increase.
Figs. 10 and 11 contain the r SYS ∆( , u) correlations for ROUGE, BERTScore, and QAEval for various combinations of and u on both the Summ-Eval and REALSumm datasets. The first rows of each heatmap are plotted in Figs. 6 and 9. We see that as the allowed score gap between system pairs is allowed to increase (i.e., adding "easier" pairs to rank), the correlation increases by a large margin over the correlation on pairs close in score. All of the metrics have nearly perfect correlation when the system pairs are separated by large margins.

C Summarization Paper Survey
To estimate the difference in ROUGE-1 score that is commonly reported in papers, we performed a survey of recently published summarization papers. We selected papers from 2020 and 2021 that were published in a *ACL conference (including Findings), had "summary" or "summarization" in the title, proposed a new system, and compared systems on the CNN/DailyMail dataset with ROUGE. We selected the differences between the best two models that were compared with ROUGE on the test set. We did not include ablation experiments for which the differences are likely smaller than the differences between the top two performing systems. The results are shown in Table 1    2) for ROUGE (top), BERTScore (middle), and QAEval (bottom) on SummEval (left) and REALSumm (right). The values of and u were chosen so that each value in the heatmaps evaluates on 10% more system pairs than the value to its left. For instance, the first row evaluates on 10%, 20%, . . . , 100% of the system pairs. The second row evaluates on 10%, 20%, . . . , 90% of the system pairs, never including the 10% of pairs which are closest in score. The first row of each of the heatmaps is plotted in Fig. 6. The correlations on realistic score differences between systems are in the upper left portion of the heatmaps and contain the lowest correlations overall. Evaluating on all pairs is the top-rightmost entry, and the "easiest" pairs (those separated by a large score margin) are in the bottom right.


Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models

World models learn the consequences of actions in vision-based interactive systems. However, in practical scenarios such as autonomous driving, there commonly exists noncontrollable dynamics independent of the action signals, making it difficult to learn effective world models. To tackle this problem, we present a novel reinforcement learning approach named Iso-Dream, which improves the Dream-to-Control framework in two aspects. First, by optimizing the inverse dynamics, we encourage the world model to learn controllable and noncontrollable sources of spatiotemporal changes on isolated state transition branches. Second, we optimize the behavior of the agent on the decoupled latent imaginations of the world model. Specifically, to estimate state values, we roll-out the noncontrollable states into the future and associate them with the current controllable state. In this way, the isolation of dynamics sources can greatly benefit long-horizon decision-making of the agent, such as a self-driving car that can avoid potential risks by anticipating the movement of other vehicles. Experiments show that Iso-Dream is effective in decoupling the mixed dynamics and remarkably outperforms existing approaches in a wide range of visual control and prediction domains.

Introduction
Humans can infer and predict real-world dynamics by simply observing and interacting with the environment. Inspired by this, many cutting-edge AI agents use self-supervised learning [38,20,12] or reinforcement learning [39,22,42] techniques to acquire knowledge from their surroundings. Among them, world models [20] have received widespread attention in the field of robot visual control, and led the recent progress in model-based reinforcement learning (MBRL) [22,42,24,30]. A typical approach [22] is to use the trajectories of observations and control signals collected by an RL agent to learn a differentiable simulator of the environment, namely the world model, and then update the RL agent by optimizing the behaviors on the latent imaginations of the world model. However, since the observation sequence is high-dimensional, non-stationary, and often driven by multiple sources of physical dynamics, how to learn effective world models in complex visual scenes remains an open problem. In realistic scenarios such as autonomous driving, we can generally divide spatiotemporal dynamics in the system into controllable parts that perfectly respond to action signals, and parts beyond the control of the agent, such as the movement of other vehicles and other external changes. The isolation of controllable and noncontrollable states can improve MBRL in two aspects: • Modular representation improves the generalization of the agent to non-stationary environments with noises, such as the time-varying background in our modified DeepMind Control Suite. • More importantly, it improves long-horizon RL tasks that can greatly benefit from decisions based on predictions of future noncontrollable dynamics. For example, in autonomous driving, potential risks can be better avoided by predicting the movement of other vehicles.
We present Iso-Dream, a novel MBRL framework that learns to decouple and leverage the controllable and noncontrollable state transitions. Accordingly, it improves the original Dreamer [22] from two perspectives: (i) a new form of world model representations and (ii) a new actor-critic algorithm to derive the behavior from the world model. As shown in Figure 1, the foundation of decoupling the world model is to separate the mixed latent states into an action-conditioned branch and an action-free branch, which can individually transit different sources of visual dynamics. The components are jointly trained to maximize the variational lower bounds. To further isolate the controllable states, the action-conditioned branch is also optimized with inverse dynamics, that is, to reason about the actions that have driven the state transitions between adjacent time steps.
Another contribution of Iso-Dream is to find that disentangling physical dynamics can greatly benefit the downstream decision-making tasks by more accurately foreseeing the inherent changes in the environment. Intuitively, humans can decide how to interact with the environment at each moment based on their anticipation of future changes in the surroundings. To make more forward-looking decisions, as shown by the blue arrows in Figure 1, the policy network integrates the current controllable state and multiple steps of predicted noncontrollable states through an attention mechanism. It enables the agent to thoroughly consider possible future interactions with the environment.
We evaluate Iso-Dream in the following domains: The modified DeepMind Control Suite with noisy video background; The CARLA autonomous driving environment in which other vehicles can be naturally viewed as noncontrollable components; The real-world BAIR robot dataset and the RoboNet dataset that are helpful to validate the effectiveness of the world model for disentanglement. On all benchmarks, Iso-Dream remarkably outperforms the existing approaches by large margins.

Related Work
Action-conditioned video prediction. A straightforward deep learning solution to visual control problems is to learn action-conditioned video prediction models [38,14,8,53] and then perform Monte-Carlo importance sampling and optimization algorithms, such as the cross-entropy methods, over available behaviors [15,12,29]. Hot topics in video prediction mainly includes long-term and high-fidelity future frames generation [44,43,51,5,52,50,54,41,40,36,56,28,2], dynamics uncertainty modeling [1,10,48,31,7,16,55], object-centric scene decomposition [47,27,18,58,3], and space-time disentanglement [49,27,19,6]. The corresponding technical improvements mainly involve the use of more effective neural architectures, novel probabilistic modeling methods, and specific forms of video representation. The disentanglement methods are closely related to the world model in Iso-Dream. They commonly separate visual dynamics into content and motion vectors, or long-term and short-term states. In contrast, Iso-Dream is designed to learns a decoupled world model based on controllability, which contributes more to the downstream behavior learning process.
Visual MBRL. In visual control tasks, the agents have to learn the action policy directly from high-dimensional observations. They can be roughly grouped into two categories, that is, model-free methods [34,57,32,33,25] and model-based methods [15,39,20,23,22,30,42,59,4]. Among them, the MBRL approaches explicitly model the state transitions and generally yield higher sample efficiency than the model-free methods. Ha and Schmidhuber [20] proposed the World Models that first learn compressed latent states of the environment in a self-supervised manner, and then train the agent on the latent states generated by the world model. Following the two-stage training procedure, PlaNet [23] uses an action-conditioned, recurrent state-space model (RSSM) as the world model, and optimizes the action policy on the recurrent states with the cross-entropy methods. In Dreamer [22] and DreamerV2 [24], agents learn behaviors by optimizing the expected values over the predicted latent states in RSSM. InfoPower [4] prioritizes functional-related information from visual observations to obtain a more robust representation for MBRL. Notably, Iso-Dream is very different from InfoPower in two aspects. First, we explicitly model the state transitions of controllable and noncontrollable dynamics, so that it is possible to choose whether to take the noncontrollable states into behavior learning according to the prior knowledge of a specific domain. Second, we propose a new behavior learning method that greatly benefits from the decoupled world model, so that we can preview possible future states of noncontrollable patterns before making decisions at this moment.

Method
In this section, we first present basic assumptions and the general framework of Iso-Dream for decoupling and leveraging controllable and noncontrollable dynamics for visual control (Section 3.1).
For representation learning, we introduce the three-branch world model and its training objectives of inverse dynamics (Section 3.2). For behavior learning, we present an actor-critic method that is trained on the imaginations of the decoupled world model latent states, so that the agent may consider possible future states of noncontrollable dynamics (Section 3.3). Finally, we discuss how Iso-Dream is deployed to interact with the environment (Section 3.4).

Basic Assumptions of Iso-Dream
As shown in Figure 1, when the agent receives a sequence of visual observations o 1:T , the underlying spatiotemporal dynamics can be defined as u 1:T . Our goal is to understand the inner relationships of different dynamics by decoupling u 1:T into controllable latent states s 1:T and noncontrollable latent states z 1:T that vary in spacetime, such that: where a t is the action signal. To achieve long-term prediction, we isolate s t and z t to each other and model their state transitions of p(s t+1 | s t , a t ) and p(z t+1 | z t ) respectively.
According to our prior knowledge of the environment, we can optionally choose whether to roll out the noncontrollable states and consider them during behavior learning. For tasks where the noncontrollable components can be viewed as time-varying noise, we simply derive the action policy by a t ∼ π(a t | s t ). The isolation of controllable states improves the generalization of the agent to non-stationary systems. For tasks like autonomous driving, the behaviors are derived by a t ∼ π(a t | s t , z t:t+τ ), where we calculate the relationships between s t and the imagined noncontrollable states over time horizon τ . It assumes that, in specific long-horizon tasks, the agent can greatly benefit from predicting the consequences of external noncontrollable forces.

Representation Learning of Controllable and Noncontrollable Dynamics
Inspired by previous approaches [37,17] showing that modular structures are effective for disentanglement learning, we leverage a three-branch architecture to decouple u t into controllable dynamics state s t , noncontrollable dynamics state z t , and time-invariant representation of the background. As shown in Figure 2(a), the action-conditioned branch models p(s t+1 | s t , a t ). It follows the RSSM architecture from PlaNet [23] to use a recurrent neural network GRU s (·), the deterministic hidden state h t , and the stochastic state s t to form the transition model, where the GRU keeps the historical information of the controllable dynamics. The action-free branch models p(z t+1 | z t ) with similar network structures. The transition models with separate parameters can be written as follows: Action-free branch Static branch Action-conditioned branch Action-free branch We here uses t andz t to denote the prior representations. We optimize the transition models with posterior representations that are derived from We learn the posteriors from the observation at current time step o t ∈ R 3×H×W by a shared encoder Enc θ and subsequent branch-specific encoders Enc φ1 and Enc φ2 .
To enhance the disentanglement representation learning corresponding to the control signals, we introduce the training objective of inverse dynamics. Accordingly, we design an Inverse Cell of a 2-layer MLP to infer the actions that lead to certain transitions of the controllable states: where the inputs are the posterior representations in the action-conditioned branch. By learning to regress the true behavior a t−1 , the Inverse Cell facilitates the action-conditioned branch to isolate the representation of the controllable dynamics. To avoid the training collapse where the actionconditioned branch captures most of the useful information, while the action-free branch learns almost nothing, in the process of image reconstruction, we respectively use the prior states t and the posterior state z t to generate the controllable visual componentô s t ∈ R 3×H×W with mask M s t ∈ R 1×H×W and the noncontrollable componentô z t ∈ R 3×H×W with M z t ∈ R 1×H×W . By further integrating the time-invariant information extracted from the first K frames, we havê For reward modeling, we have two options with the action-free branch. In one case, the noncontrollable dynamics can be considered as noises that are not related to the task, and therefore z t is no longer useful during imagination. In other words, the policy and the predicted reward are only related to the controllable states. In the other case, future noncontrollable states would affect how the agent makes decisions, and we consider the action-free components during behavior learning. For this, we learn alternative reward models p(r t | s t ) or p(r t | s t , z t ) in forms of MLPs.
For a sequence of (o t , a t , r t ) T t=1 sampled from the replay buffer during training, the world model can be optimized using the following loss functions, where α, β 1 , and β 2 are hyper-parameters: Roll-out the noncontrollable states {zi} t+L+τ i=t+1 from zt through the action-free branch alone. 10 for time step j = i . . . i + L do 11 Compute latent state ej ∼ Attention(sj,zj:j+τ ) using Eq. (7). 12 Imagine an action aj ∼ π(aj|ej). 13 Predict the next controllable statesj+1 ∼ p(sj, aj) using the action-conditioned branch alone.
Roll-out the noncontrollable stateszt+1:t+τ from zt through the action-free branch alone. 22 Generate at ∼ π(at | st, zt,zt+1:t+τ ) using future state attention in Eq. (7). The world model training approach can be partly customized for different environments. In situations where noncontrollable states are indeed involved in behavior learning, minimizing the ELBO objective can maintain the semantics ofz t . Otherwise, if the action-free features are only used to prevent noisy distractions from affecting the training process of Iso-Dream, rather than being used for behavior learning, we can simply train the action-free branch with the reconstruction loss alone.

Behavior Learning in Decoupled Imaginations
Thanks to the decoupled world model, we can optimize the agent behaviors to adaptively consider the relations between available actions and possible future states of the noncontrollable dynamics. A practical example is autonomous driving, where the motion of other vehicles can be naturally viewed as noncontrollable but predictable components. As shown in Figure 2(b), we here propose an improved actor-critic learning algorithm that 1) allows the action-free branch to foresee the future ahead of the action-conditioned branch, and 2) exploits the predicted future information of noncontrollable dynamics to make more forward-looking decisions.
Suppose we are making decisions at time step t in the imagination period. A straightforward solution from the original Dreamer method is to learn an action model and a value model based on the isolated controllable states t ∈ R 1×d . However, we notice that by employing an attention mechanism, we can explicitly calculate its relations to a sequence of future noncontrollable statesz t:t+τ ∈ R τ ×d , where τ is the length of a sliding window from now on.
In this way,s t evolves to a more "visionary" representation e t ∈ R 1×d . We update the action model and the value model in Dreamer [22] as follows: Action model: a t ∼ π(a t | e t ), where L is the imagination time horizon. As shown in Alg. 1, during imagination, we first use the action-free transition model to obtain sequences of noncontrollable states of length L + τ , denoted . At each time step in the imagination period, the agent draws an action a j from the visionary state e j , which is derived from Eq. (7). The action-conditioned branch uses the action a j in latent imagination and predicts the next controllable state s j+1 . We follow DreamerV2 [24] to train the action model to maximize the λ-return [45], and train the value model to regress the λ-return 3 .

Policy Deployment by Rolling-out Noncontrollable Dynamics
As discussed above, in the cases that noncontrollable dynamics are irrelevant to the control task, when interacting with the environment, we only use the state of controllable dynamics to generate the policy at each time step t. However, for the situation where noncontrollable dynamics should be closely related to the behavior of the agent, as shown in Lines 21-22 in Alg. 1, the action-free branch consecutively predicts the next τ − 1 noncontrollable statesz t+1:t+τ starting from the current posterior state z t . Similar to Eq. (7) in the process of behavior learning, we here use the learned future state attention network to adaptively integrate s t , z t andz t+1:t+τ . Based on the integrated feature e t , the Iso-Dream agent draws a t from the action model to interact with the environment.

Experimental Setup
Benchmarks. We quantitatively and qualitatively evaluate Iso-Dream on two reinforcement learning environments, i.e., DeepMind Control Suite [46] and CARLA [11], and two real-world datasets for action-conditioned video prediction, i.e., BAIR robot pushing [13] and RoboNet [9]. The video prediction experiments can provide more intuitive visualizations of disentanglement learning.

DeepMind Control Suite
Implementation details. In order to verify the enhancement of Iso-Dream by disentangling different components under complex visual dynamics, we evaluate Iso-Dream on environments from DMC Generalization Benchmark. Instead of training on original DeepMind Control Suite environments, agents are trained and tested both with natural video backgrounds (i.e. video_easy environments). In this environment, since the background is randomly replaced by a real-world video, the noncontrollable motion of the background can affect the procedure of dynamics learning and behavior learning of agents. Therefore, to obtain a better decision policy and avoid the disruption from noisy backgrounds, the agent may decouple noncontrollable representation (i.e., dynamic background) and controllable representation in spacetime, and only use controllable representation for control. To this end, we simply train the action-free branch with only reconstruction loss and discard it in imagination and policy deployment. We evaluate our model with baselines in 4 tasks from four different domains. The number of environment steps is limited to 500k.
Quantitative results. To evaluate the performance, we train and test the agents in environments with video backgrounds. As shown in Table 1, Iso-Dream exceeds the performance of DreamerV2 and other baselines in all tasks, indicating that the three-branch structure can effectively learn task-related visual representations and alleviate complex background interference in visual data. For each sequence, we use the first 5 images as context frames. Iso-Dream successfully disentangles controllable and noncontrollable components.
Qualitative results. We leverage Iso-Dream to complete video prediction tasks in video_easy environments. The sequence of frames and actions are randomly collected during test episodes. The first 5 frames are given to the model and the next 45 frames are predicted only based on action inputs. To show the qualitative results, we visualize the masks and visual decoupled components from the action-conditioned and action-free branches. The overall visualization is shown in Figure 3(left). From this prediction result, we can find that Iso-Dream has the ability to predict long-term sequence and disentangle controllable and noncontrollable dynamics from images in video_easy environments. As shown in the third and fourth row of action-conditioned branch output in Figure 3, the controllable representation has been successfully isolated and matches its mask. Besides, in this visualization, the action-free component in this background video is the motion of sea waves, which is captured by the fifth and sixth row of action-free branch outputs.

CARLA Autonomous Driving Environment
Implementation details. In the autonomous driving task, We use a camera with 60 degree view on the roof of the ego-vehicle, which obtains images of 64 × 64 pixels. Following the setting in the DBC [59], in order to encourage highway progression and penalise collisions, the reward is formulated as: where v ego is the velocity vector of the ego-vehicle, projected onto the highway's unit vectorû h , and multiplied by time discretization ∆t = 0.05 to measure highway progression in meters. Impulse I ∈ R + is caused by collisions, and a steering penalty steer ∈ [−1, 1] facilitates lane-keeping. The hyper-parameters ξ 1 and ξ 2 are set to 10 −4 and 1, respectively. We use β 1 = 1, β 2 = 1 and α = 1 in Eq. (6) and τ = 5 in Eq. (7).
Quantitative results. As shown in Figure 4(a), Iso-Dream has significant advantages compared to other baselines and outperforms DreamerV2 by a large margin. Furthermore, we conduct ablation studies to confirm the validity of inverse dynamics and the rolling-out strategy of noncontrollable states. Figure 4(b) shows that the performance drops when Inverse Cell is removed, indicating the importance of modeling inverse dynamics to isolate controllable and noncontrollable components from the whole dynamics. In order to verify the effectiveness of the proposed attention mechanism, we conduct experiments to evaluate Iso-Dream where policy networks directly concatenate the current controllable state and the noncontrollable state as input. Comparing the blue curve and green curve,   we observe that rolling-out noncontrollable states in the action-free branch can significantly improve the agent's decision-making results. The red curve shows that the performance of Iso-Dream degrades by about 15% in the absence of a separate network branch that captures the static information.
Qualitative results. Reconstruction results of predictions in CARLA environment are shown in Figure 3(right column). In CARLA, we observe that the agent actions potentially affect all pixel values in the observation, as the camera on the main car (i.e., the agent) moves. Therefore, we view the visual dynamics of other vehicles as a combination of controllable and noncontrollable states. Accordingly, our model can determine which component is dominant by learning attention masks (values between 0 and 1) across the action-conditioned and action-free branches. The "action-free masks" present hot spots around other vehicles, while the attention values in corresponding areas on the "action-cond masks" are still greater than 0. The agent can avoid collisions by rolling-out noncontrollable components to preview possible future states of other vehicles. We include more showcases with different numbers of vehicles in the supplementary materials.

BAIR & RoboNet for Action-Conditioned Video Prediction
Implementation details. In order to evaluate the effectiveness of our world model in a more complex environment, we test the video prediction ability of the proposed structure on BAIR and RoboNet dataset. Moreover, we add predictable visual dynamics unrelated to the control signals to the raw observations, i.e., bouncing balls of the same size and speed. In the training phase, we train the model to predict 10 frames into the future from 2 observations. For testing, we use the first 2 frames as input to predict the next 28 frames in the BAIR dataset, and the next 18 frames in the RoboNet dataset. All inputs for training and testing are resized to 64 × 64. Considering the simplicity and predictability of bouncing balls, in the action-free branch, we use a similar structure as in the DMC experiment. Moreover, we replace the GRU cell with two layers of ST-LSTM unit [52] in both branches. The optimization objective consists of image reconstruction loss and action reconstruction loss of Inverse Cell. SSIM and PSNR are adopted as evaluation metrics.  Figure 5: Showcases of video prediction results on the BAIR robot pushing dataset. We display every 3 frames in the prediction horizon. The generated masks show that each branch of Iso-Dream captures coarse localisation of controllable representations and noncontrollable representations.
Quantitative results. Table 2 gives the quantitative results on BAIR and RoboNet datasets with bouncing balls in the training and testing phase. Compared with other models, Iso-Dream shows the competitive performance in two datasets. For PSNR, Iso-Dream improves SVG by 7.7% in BAIR and 9.3% in RoboNet. Compared with PhyDNet, which also disentangles features in two branches, Iso-Dream achieves better performance in both PSNR and SSIM. It shows that our Iso-Dream has a stronger ability of disentanglement learning to achieve long-term prediction.
Qualitative results. We visualize a sequence of predicted frames on BAIR with bouncing balls in Figure 5. Specifically, the output of two branches and corresponding masks are provided. We can see from these demonstrations that the world model of Iso-Dream is more accurate in modeling future dynamics for long-term prediction. It shows the fact that the action-free branch learns noncontrollable dynamics, while the action-conditioned branch learns controllable dynamics related to input action.

Conclusions
In this paper, we proposed an MBRL framework named Iso-Dream, which mainly tackles the difficulty of vision-based prediction and control in the presence of complex visual dynamics. Our approach has two novel contributions to world model representation learning and corresponding MBRL algorithms. First, it learns to decouple controllable and noncontrollable latent state transitions via modular network structures and inverse dynamics. Further, it makes long-horizon decisions by rolling-out the noncontrollable dynamics into the future and learning their influences on current behavior. Iso-Dream achieves competitive results on the CARLA autonomous driving task, where other vehicles can be naturally viewed as noncontrollable components, indicating that with the help of decoupled latent states, the agent can make more forward-looking decisions by previewing possible future states in the action-free network branch. Besides, Iso-Dream was shown to effectively improve the visual control task in a modified DeepMind Control Suite, as well as the visual prediction task on the BAIR robot pushing dataset and the RoboNet dataset.
One limitation of Iso-Dream is the computational efficiency. Compared with DreamerV2, it requires longer training time per episode due to more intensive state transitions in behavior learning. But fortunately, from Figure 4(a), Iso-Dream is more sample-efficient than existing MBRL methods. Another limitation is the special treatment for different environments. In our preliminary experiments, we attempted to use the same model architecture for all test benchmarks. However, we observed that different benchmarks have specific requirements on the network structure, which we found should be dependent on our prior knowledge of the environments.

A Benchmarks
We quantitatively and qualitatively evaluate Iso-Dream on the following two environments for visual control and two real-world datasets for action-conditioned video prediction.
• DeepMind control suite [46]: A set of stable, well-tested continuous control tasks that are easy to use and modify. For vision-based control, we use a modified version of the DeepMind control suite in DMControl Generalization Benchmark [26] to evaluate Iso-Dream. In this environment, agents are trained to complete different tasks with random natural video as backgrounds, namely video_easy and video_hard benchmarks. We use 4 tasks to test our Iso-Dream, i.e., Finger Spin, Cheetah Run, Walker Walk, Hopper Stand.
• CARLA [11]: An open-source simulator with more complex and realistic visual observations for autonomous driving research. In our experiments, we evaluate Iso-Dream in a first-person highway driving task in "Town04". The agent's goal is to drive as far as possible in 1000 time steps without colliding with the 30 other moving vehicles or barriers.
• BAIR robot pushing [13]: An action-conditioned video prediction dataset composed of hours of self-supervised learning with the robotic arm Sawyer. In each video, a random moving robotic arm pushes a variety of objects on similar tables with a static background. Each video also has recorded actions taken by the robotic arm which correspond to the commanded gripper pose.
• RoboNet [9]: A large-scale dataset contains action-conditioned videos of seven robotic arms interacting with a variety of objects from four different research laboratories, i.e., Berkeley, Google, Penn, and Stanford.

B Compared Methods
For visual MBRL, we compare our method with the following baselines and existing approaches: • DreamerV2 [24]: A model-based RL method that learns directly from latent variables in world models. The latent representation enables agents to imagine thousands of trajectories in parallel.
• CURL [34]: A model-free RL method that extracts high-level features from raw pixels using contrastive learning, maximizing agreement between augmented versions of the same observation.
• SVEA [25]: A framework for data augmentation in deep Q-learning algorithms that improves stability and generalization on off-policy RL.
• SAC [21]: A model-free actor-critic method that optimizes a stochastic policy in an off-policy way.
• DBC [59]: It learns a bisimulation metric representation without reconstruction loss, which are invariant to different task-irrelevant details in the observation.
For video prediction, we compare the proposed world model with the following approaches: • SVG [10]: This model introduces random variables into latent space, which ensures that the future trajectory is inherently random.
• SA-ConvLSTM [35]: Based on the self-attention mechanism, this model uses the self-attention memory to capture long-term spatial dependency.
• PhyDNet [19]: This model uses a two-branch architecture to disentangle PDE dynamics from unknown complementary information.

C Additional Visualization in DMC and CARLA
DeepMind Control suite. In Figure 6, more showcases on the DeepMind Control are presented with different noisy backgrounds. We show the visualization of the masks and decoupled components from three branches of Iso-Dream.
CARLA autonomous driving simulator. In Figure 7, we visualize the video prediction results on the CARLA environment with different numbers of vehicles. We train Iso-Dream with 30 vehicles and test with 10 vehicles and 20 vehicles respectively. D Additional Results on the BAIR Robot Pushing Dataset Figure 8 shows an interesting result of the different training sets (i.e., BAIR, BAIR+bouncing balls) and the same testing set (i.e., BAIR). Iso-Dream is the only approach that achieves improvements when training on noisy data with bouncing balls, as shown in Figure 8(red bars). In this training setup, it performs best on the standard test set without balls. Iso-Dream is built on a more efficient architecture than the baseline models. It provides a general framework that can be easily extended to other backbones.
Ablation study. In Table 3, the first row shows the results of removing the action-free branch in the world model of Iso-Dream. The performance has decreased from 21.43 to 20.47 and from 19.51 to 18.51 in PSNR for predicting the next 18 frames and next 28 frames respectively, indicating that modular network structures are effective for predictive learning by decoupling the controllable and noncontrollable representations. Comparing the second row and third row in the Table 3, we observe that modeling inverse dynamics can improve the performance by learning more deterministic state transitions given particular actions in the action-conditioned branch.

E Network Architectures for Different Environments
The networks and hyper-parameters used for different environments are shown in Table 4. The results of models trained on BAIR (blue) and BAIR + bouncing balls (red), and tested on BAIR. We use the first 2 frames as input to predict the next 18 frames. The horizontal axis represents the different models, and the vertical axes represent test results of PSNR and SSIM.  conv3-3 conv3-3 -