Music theory studies the regularity of patterns in music to capture concepts underlying music styles and composers’ decisions. This paper continues the study of building automatic theorists (rovers) to learn and represent music concepts that lead to human interpretable knowledge and further lead to materials for educating people. Our previous work took a first step in algorithmic concept learning of tonal music, studying high-level representations (concepts) of symbolic music (scores) and extracting interpretable rules for composition. This paper further studies the representation hierarchy through the learning process, and supports adaptive 2D memory selection in the resulting language model. This leads to a deeper-level interpretability that expands from individual rules to a dynamic system of rules, making the entire rule learning process more cognitive. The outcome is a new rover, MUS-ROVER II, trained on Bach’s chorales, which outputs customizable syllabi for learning compositional rules. We demonstrate comparable results to our music pedagogy, while also presenting the differences and variations. In addition, we point out the rover’s potential usages in style recognition and synthesis, as well as applications beyond music.
1 INTRODUCTION
Forming hierarchical concepts from low-level observations is key to knowledge discovery. In the field of artificial neural networks, deep architectures are employed for machine learning tasks, with the awareness that hierarchical representations are important (Bengio et al., 2013). Rapid progress in deep learning has shown that mapping and representing topical domains through increasingly abstract layers of feature representation is extremely effective. Unfortunately, this layered representation is difficult to interpret or use for teaching people. Consequently, deep learning models are widely used as algorithmic task performers (e.g. AlphaGo), but few act as theorists or pedagogues. In contrast, our goal is to achieve a deeper-level interpretability that explains not just what has been learned (the end results), but also what is being learned at every single stage (the process).
On the other hand, music theory studies underlying patterns beneath the music surface. It objectively reveals higher-level invariances that are hidden from the low-level variations. In practice, the development of music theory is an empirical process. Through manual inspection of large corpora of music works, theorists have summarized compositional rules and guidelines (e.g. J. J. Fux, author of Gradus ad Parnassum, the most influential book on Renaissance polyphony), and have devised multi-level analytical methods (e.g. H. Schenker, inventor of Schenkerian analysis) to emphasize the hierarchical structure of music, both of which have become the standard materials taught in today’s music theory classes. The objective and empirical nature of music theory suggests the possibility of an automatic theorist — statistical techniques that perform hierarchical concept learning — while its pedagogical purpose requires human interpretability throughout the entire learning process.
The book title Gradus ad Parnassum, means “the path towards Mount Parnassus,” the home of poetry, music, and learning. This paper presents MUS-ROVER II, an extension of our prior work (Yu et al., 2016a;b), to independently retake the path towards Parnassus. The rover acts more as a pathfinder than a generative model (e.g. LSTM), emphasizing the path more than the destination.
We compare the paths taken by this improved automatic theorist to paths taken by human theorists (say Fux), studying similarities as well as pros and cons of each. So advantages from both can be jointly taken to maximize the utility in music education and research. In this paper in particular, we highlight the concept hierarchy that one would not get from our prior work, as well as enhanced syllabus personalization that one would not typically get from traditional pedagogy.
2 MUS-ROVER OVERVIEW
As the first algorithmic pathfinder in music, MUS-ROVER I introduced a “teacher student” model to extract compositional rules for writing 4-part chorales (Yu et al., 2016a;b). The model is implemented by a self-learning loop between a generative component (student) and a discriminative component (teacher), where both entities cooperate to iterate through the rule-learning process (Figure 1). The student starts as a tabula rasa that picks pitches uniformly at random to form sonorities (a generic term for chord) and sonority progressions. The teacher compares the student’s writing style (represented by a probabilistic model) with the input style (represented by empirical statistics), identifying one feature per iteration that best reveals the gap between the two styles, and making it a rule for the student to update its probabilistic model. As a result, the student becomes less and less random by obeying more and more rules, and thus, approaches the input style. Collecting from its rule-learning traces, MUS-ROVER I successfully recovered many known rules, such as “Parallel perfect octaves/fifths are rare” and “Tritons are often resolved either inwardly or outwardly”.
What is Inherited from MUS-ROVER I MUS-ROVER II targets the same goal of learning interpretable music concepts. It inherits the self-learning loop, as well as the following design choices.
(Dataset and Data Representation) We use the same dataset that comprises 370 C scores of Bach’s 4-part chorales. We include only pitches and their durations in a piece’s raw representation, notated as a MIDI matrix whose elements are MIDI numbers for pitches. The matrix preserves the twodimensional chorale texture, with rows corresponding to melodies, and columns to harmonies.
(Rule Representation) We use the same representation for high-level concepts in terms of rules, unrelated to rules in propositional logic. A (compositional) rule is represented by a feature and its distribution: r = (φ, pφ), which describes likelihoods of feature values. It can also be transformed to a linear equality constraint (Aφpstu = pφ) in the student’s optimization problem (Γ’s in Figure 1).
(Student’s Probabilistic Model) We still use n-gram models to represent the student’s style/belief, with words being sonority features, and keep the student’s optimization problem as it was. To reiterate the distinctions to many music n-grams, we never run n-grams in the raw feature space, but only collectively in the high-level feature spaces to prevent overfitting. So, rules are expressed as probabilistic laws that describe either (vertical) sonority features or their (horizontal) progressions.
What is New in MUS-ROVER II We study hierarchies on features, so rules are later presented not just as a linear list, but as hierarchical families and sub-families. In particular, we introduce conceptual hierarchy that is pre-determined by feature maps, and infer informational hierarchy that is post-implied from an information-theoretic perspective. We upgrade the self-learning loop to adaptively select memories in a multi-feature multi-n-gram language model. This is realized by constructing hierarchical filters to filter out conceptual duplicates and informational implications. By further following the information scent spilled by Bayesian surprise (Varshney, 2013), the rover can effectively localize the desired features in the feature universe.
3 RELATED WORK
Adversarial or Collaborative MUS-ROVER’s self-learning loop between the teacher (a discriminator) and student (a generator) shares great structural similarity to generative adversarial nets (Goodfellow et al., 2014) and their derivatives (Denton et al., 2015; Makhzani et al., 2015). However, the working mode between the discriminator and generator is different. In current GAN algorithms, the adversarial components are black-boxes to each other, since both are different neural networks that are coupled only end to end. The learned intermediate representation from one model, no matter how expressive or interpretable, is not directly shared with the other. Contrarily in MUSROVER, both models are transparent to each other (also to us): the student directly leverages the rules from the teacher to update its probabilistic model. In this sense, the learning pair in MUSROVER is more collaborative rather than adversarial. Consequently, not only the learned concepts have interpretations individually, but the entire learning trace is an interpretable, cognitive process.
Furthermore, MUS-ROVER and GAN contrast in the goal of learning and the resulting evaluations. The rover is neither a classifier nor a density estimator, but rather a pure representation learner that outputs high-level concepts and their hierarchies. Training this type of learner in general is challenging due to the lack of a clear objective or target (Bengio et al., 2013), which drives people to consider some end task like classification and use performance on the task to indirectly assess the learned representations. In MUS-ROVER, we introduce information-theoretic criteria to guide the training of the automatic theorist, and in the context of music concept learning, we directly evaluate machine generated rules and hierarchies by comparison to those in existing music theory.
Interpretable Feature Learning In the neural network community, much has been done to first recover disentangled representations, and then post-hoc interpret the semantics of the learned features. This line of work includes denoising autoencoders (Vincent et al., 2008) and restricted Boltzmann machines (Hinton et al., 2006; Desjardins et al., 2012), ladder network algorithms (Rasmus et al., 2015), as well as more recent GAN models (Radford et al., 2015). In particular, InfoGAN also introduces information-theoretic criteria to augment the standard GAN cost function, and to some extent achieves interpretability for both discrete and continuous latent factors (Chen et al., 2016). However, beyond the end results, the overall learning process of these neural networks are still far away from human-level concept learning (Lake et al., 2015), so not directly instructional to people.
Automatic Musicians Music theory and composition form a reciprocal pair, often realized as the complementary cycle of reduction and elaboration (Laitz, 2016) as walks up and down the multilevel music hierarchy. Accordingly, various models have been introduced to automate this up/down walk, including music generation (Cope & Mayer, 1996; Biles, 1994; Simon et al., 2008), analysis (Taube, 1999), or theory evaluation (Rohrmeier & Cross, 2008). In terms of methodologies, we have rule-based systems (Cope, 1987), language models (Google Brain, 2016; Simon et al., 2008), and information-theoretic approaches (Jacoby et al., 2015; Dubnov & Assayag, 2002). However, all of these models leverage domain knowledge (e.g. human-defined chord types, functions, rules) as part of the model inputs. MUS-ROVER takes as input only the raw notations (pitches and durations), and outputs concepts that are comparable to (but also different from) our domain knowledge.
4 HIERARCHICAL RULE LEARNING
MUS-ROVER II emphasizes hierarchy induction in learning music representations, and divides the induction process into two stages. In the first stage, we impose conceptual hierarchy as pre-defined structures among candidate features before the self-learning loop. In the second stage, we infer informational hierarchy as post-implied structures through the rule learning loops.
Interpretable Features A feature is a function that computes a distributed representation of the building blocks that constitute data samples. For Bach’s 4-part chorales, we model every piece (4-row matrix) as a sequence of sonorities (columns). So every sonority is the building block of its composing piece (like a word in a sentence). Then a feature maps a sonority onto some feature space, summarizing an attribute. To formalize, let Ω = {R, p1, . . . , pn} be an alphabet that comprises a rest symbol R, and n pitch symbols pi. In addition, the alphabet symbols — analogous to image pixels — are manipulable by arithmetic operations, such as plus/minus, modulo, and sort. More precisely, every pi is an integer-valued MIDI number (60 for middle C, granularity 1 for semi-tone), and R is a special character which behaves like a python nan variable. The four coordinates of every sonority p ∈ Ω4 denote soprano, alto, tenor, and bass, respectively. We define a feature as a surjective function φ : Ω4 7→ φ(Ω4), and the corresponding feature space by its range. As a first and brutal categorization, we say a feature (space) is raw (or lowest-level) if |φ(Ω4)| = |Ω4|, and high-level if |φ(Ω4)| < |Ω4|. For instance, Ω4 or any permutation of Ω4 is a raw feature space. MUS-ROVER II employs a more systematic way of generating the universe of interpretable features. A (sonority) feature is constructed as the composition of a window and a descriptor. A window is a function that selects parts of the input sonority: wI : Ω4 7→ Ω|I|, where I is an index set. For instance, w{1,4}(p) = (p1, p4) selects soprano and bass. A descriptor is constructed inductively from a set of basis descriptors B, consisting of atomic arithmetic operations. We currently set B = {order, diff, sort, mod12} (Appendix A.2). We define a descriptor of length k as the composition of k bases: d(k) = bk ◦ · · · ◦ b1, for all bi ∈ B, where d(0) is the identity function. We collect the family of all possible windows: W = {wI | I ∈ 2{1,2,3,4}\{∅}}, and the family of all descriptors of length less than or equal to k: D[k] = {d(k′) | 0 ≤ k′ ≤ k}, and form the feature universe:
Φ = {d ◦ w | w ∈W,d ∈ D[k]}. (1) The fact that every candidate feature in Φ is systematically generated as composition of atomic operators ensures its interpretability, since one can literally read it out step-by-step from the composition.
Feature-Induced Partition On the one hand, a feature function has all the mathematic specifications to name the corresponding feature and feature values. On the other hand, we only care about the partition of the input domain (Ω4) induced by the feature but not the (superficial) naming of the clusters. In other words, we only identity the sonority clusters whose members are mapped to the same function value, but not the value per se. As a result, we use a partition to refer to the essence of a concept, and the inducing function as a mathematical name to interpret the concept. To formalize, a feature function φ induces a partition of its domain
Pφ = { φ−1({y}) | y ∈ φ(Ω4) } . (2)
Given a feature universe Φ, (2) defines an equivalence relation on Φ: φ P∼ φ′ if Pφ = Pφ′ , which induces the corresponding partition family PΦ as the resulting equivalence classes. For two partitions P,Q ∈ PΦ, we say P is finer than Q (or Q is coarser), written as P Q, if for all p, p′ ∈ Ω4, p, p′ are in the same cluster under P ⇒ p, p′ are in the same cluster under Q. We say P is strictly finer, written as P Q, if P Q and Q P . Conceptual Hierarchy Based on the binary relation , we construct the conceptual hierarchy for the partition family PΦ, and represent it as a directed acyclic graph (DAG) with nodes being partitions. For any pair of nodes v, v′, v → v′ if and only if the partition referred by v is (strictly) finer than that referred by v′. The DAG grows from a single source node, which represents the finest partition — every point in the domain by itself is a cluster — and extends via the edges to coarser and coarser partitions. In terms of features, we say a feature φ′ is at a higher level than another feature φ, if the induced partitions satisfy Pφ Pφ′ . In other words, a higher-level feature induces a coarser partition that ignores lower-level details by merging clusters. One can check that the finest partition (the source node) is indeed induced by a raw feature. We attach an efficient algorithm for pre-computing the conceptual hierarchy in Appendix A.3.
We emphasize the necessity of this multi-step process: features→ partitions→ hierarchy (DAG), as opposed to a simple hierarchical clustering (tree). The latter loses many inter-connections due to the tree structure and its greedy manner, and more importantly, the interpretability of the partitions.
Informational Hierarchy We infer informational hierarchy from a many-to-one relation, called implication, along a rule trace. More formally, let {ri}ki=1 := {(φi, p̂φi)}ki=1 be the extracted trace
of rules (in terms of feature and feature distribution) by the kth iteration of the self-learning loop. We say a feature φ is informationally implied from the trace {ri}ki=1 with tolerance γ > 0, if
gap ( p 〈k〉 φ,stu ∥∥ p̂φ ) := D ( p 〈k〉 φ,stu ∥∥ p̂φ ) < γ, and gap ( p 〈k′〉 φ,stu ∥∥ p̂φ ) ≥ γ,∀k′ < k,
where D(·‖·) is the KL divergence used to characterize the gap of the student’s style (probabilistic model) against Bach’s style (input). One trivial case happens when φ is extracted as the kth rule, i.e. φ = φk, then gap(p 〈k〉 φ′,stu
∥∥ p̂φ′) = 0 < γ, ∀φ′ ∈ {φ′ | Pφ Pφ′}, meaning that feature φ, once learned as a rule, informationally implies itself and all its descendants in the conceptual hierarchy. However, what is more interesting is the informational implication from other rules outside the conceptual hierarchy, which is typically hard for humans to “eyeball”.
One might question the necessity of conceptual hierarchy since it can be implied in the informational hierarchy. The answer is yes in principle, but no in practice. The main difference is that conceptual hierarchy is pre-computed over the entire feature universe before the loop, which is global, precise, and trace independent. On the contrary, informational hierarchy is trace specific and loose, due to tolerance γ and the precision of the optimization solver. As a result, informational hierarchy alone tends to lose the big picture and require more post-hoc interpretations, and is unstable in practice.
Hierarchical Filters Beyond their benefits in revealing inter-relational insights among distributed representations, we build hierarchical filters from both conceptual and informational hierarchies, for the purpose of pruning hierarchically entangled features and speeding up feature selection. This upgrades MUS-ROVER II into a more efficient, robust, and cognitive theorist. Recall the skeleton of the teacher’s optimization problem in Figure 1, we flesh it out as follows:
maximize φ∈Φ
gap ( p 〈k−1〉 φ,stu ∥∥ p̂φ )
(3)
subject to H(p̂φ) ≤ δ (Regularity Condition) φ /∈ C〈k−1〉 := { φ ∣∣ Pφ Pφ′ , φ′ ∈ Φ〈k−1〉 } (Conceptual-Hierarchy Filter)
φ /∈ I〈k−1〉 := { φ ∣∣ gap ( p 〈k−1〉 φ,stu ∥∥ p̂φ ) < γ } (Informational-Hierarchy Filter)
In the above optimization problem, Φ is the feature universe defined in (1) and φ ∈ Φ is the optimization variable whose optimal value is used to form the kth rule: φk = φ?, rk = (φ?, p̂φ?). We decouple the regularity condition from the objective function in our previous work (which was the generalized cultural hole function), and state it separately as the first constraint that requires the Shannon entropy of the feature distribution to be no larger than a given threshold (Pape et al., 2015). The second constraint encodes the filter from conceptual hierarchy, which prunes coarser partitions of the learned features Φ〈k−1〉 := {φ1, . . . , φk−1}. The third constraint encodes the filter from informational hierarchy, which prunes informationally implied features.
There are two hyper-parameters δ and γ in the optimization problem (3), whose detailed usage in syllabus customization will be discussed later in Sec. 6. At a high level, we often pre-select γ before the loop to express a user’s satisfaction level: a smaller γ signifies a meticulous user who is harder to satisfy; the threshold δ upper bounds the entropic difficulty of the rules, and is adaptively adjusted through the loop: it starts from a small value (easy rules first), and auto-increases whenever the feasible set of (3) is empty (gradually increases the difficulty when mastering the current level).
5 ADAPTIVE MEMORY SELECTION
MUS-ROVER II considers a continuous range of higher order n-grams (variable memory), and adaptively picks the optimal n based on a balance among multiple criteria. The fact that every ngram is also on multiple high-level feature spaces opens the opportunities for long-term memories without exhausting machine memory, while effectively avoiding overfitting.
Two-Dimensional Memory In light of a continuous range of n-grams, say n ∈ N = {2, 3, . . . }, the feature universe adds another dimension, forming a two-dimensional memory (N×Φ) — length versus depth — for the language model (Figure 2: left). The length axis enumerates n-gram orders, with a longer memory corresponding to a larger n; the depth axis enumerates features, with a deeper
memory corresponding to a higher-level feature. Every cell in the memory is indexed by two coordinates (n, φ), referring to the feature φ under the n-gram, and stores the corresponding feature distribution. As a consequence, the rule extraction task involves picking the right feature under the right n-gram, which extends the space of the optimization problem (3) from Φ to N × Φ. Accordingly, the constraints of (3) jointly forge a mask on top of the 2D memory (Figure 2: right).
Criteria and Balance We propose three criteria to extract rules from the 2D memory: confidence, regularity, and efficacy. Confidence is quantified by empirical counts: the more relevant examples one sees in Bach’s chorales, the more confident. Regularity is quantified by Shannon entropy of the rule’s feature distribution: a rule is easier to memorize if it is less entropic (Pape et al., 2015). Efficacy is inversely quantified by the gap between the student’s probabilistic model and the rule’s feature distribution: a rule is more effective if it reveals a larger gap. There are tradeoffs among these criteria. For instance, a lower-level feature is usually more effective since it normally reflects larger variations in the gap, but is also unlikely to be regular, thus harder to memorize and generalize. Also a feature under a higher-order n-gram may be both regular and effective, but the number of examples that match the long-term conditionals is likely to be small, reducing confidence.
Adaptive Selection: Follow the (Bayesian) Surprise The teacher’s optimization problem (3) explicitly expresses the efficacy factor in the objective, and the regularity condition as the first constraint. To further incorporate confidence, we cast the rule’s feature distribution p̂φ in a Bayesian framework rather than a purely empirical framework as in our previous work. We assume the student’s belief with respect to a feature φ follows a Dirichlet distribution whose expectation is the student’s probabilistic model. In the kth iteration of the self-learning loop, we set the student’s prior belief as the Dirichlet distribution parameterized by the student’s latest probabilistic model:
priorφ,stu ∼ Dir ( c · p〈k−1〉φ,stu ) ,
where c > 0 denotes the strength of the prior. From Bach’s chorales, the teacher inspects the empirical counts qφ associated with the feature φ and the relevant n-gram, and computes the student’s posterior belief if φ were selected as the rule:
posteriorφ,stu ∼ Dir ( qφ + c · p〈k−1〉φ,stu ) .
The concentration parameters of the Dirichlet posterior show the balance between empirical counts and the prior. If the total number of empirical counts is small (less confident), the posterior will be smoothed more by the prior, de-emphasizing the empirical distribution from qφ. If we compute
p̂φ ∝ ( qφ + c · p〈k−1〉φ,stu ) in the objective of (3), then
gap ( p 〈k−1〉 φ,stu ∥∥ p̂φ ) = D ( E [ priorφ,stu ] ∥∥ E [ posteriorφ,stu ]) . (4)
The right side of (4) is closely related to Bayesian surprise (Varshney, 2013), which takes the form of KL divergence from the prior to posterior. If we remove the expectations and switch the roles between the prior and posterior, we get the exact formula for Bayesian surprise. Both functionals
capture the idea of comparing the gap between the prior and posterior. Therefore, the efficacy of concept learning is analogous to seeking (informational) surprise in the learning process.
The subtlety in (4) where we exchange the prior and posterior, makes a distinction from Bayesian surprise due to the asymmetry of KL divergence. As a brief explanation, adopting (4) as the objective tends to produce rules about what Bach hated to do, while the other way produces what Bach liked to do. So we treat it as a design choice and adopt (4), given that rules are often taught as prohibitions (e.g. “parallel fifths/octaves are bad”, “never double the tendency tones”). There are more in-depth and information-theoretic discussions on this point (Huszár, 2015; Palomar & Verdú, 2008).
6 EXPERIMENTS
MUS-ROVER II’s main use case is to produce personalized syllabi that are roadmaps to learning the input style (customized paths to Mount Parnassus). By substituting the student module, users can join the learning cycle, in which they make hands-on compositions and get iterative feedback from the teacher. Alternatively, for faster experimentation, users make the student their learning puppet, which is personalized by its external parameters. This paper discusses the latter case in detail.
Math-to-Music Dictionary MUS-ROVER II conceptualizes every rule feature as a partition of the raw space, and uses the inducing function as its mathematical name. To get the meanings of the features, one can simply work out the math, but some of them already have their counterparts as music terminologies. We include a short dictionary of those correspondences in Appendix A.1.
Pace Control and Syllabus Customization We present a simple yet flexible pace control panel to the users of MUS-ROVER II, enabling personalized set-up of their learning puppet. The control panel exposes four knobs: the lower bound, upper bound, and stride of the rule’s entropic difficulty (δmin, δmax, δstride), as well as the satisfactory gap (γ). These four hyper-parameters together allow the user to personalize the pace and capacity of her learning experience. The entropic difficulty δ caps the Shannon entropy of a rule’s feature distribution in (3), a surrogate for the complexity (or memorability) of the rule (Pape et al., 2015). It is discretized into a progression staircase from δmin up to δmax, with incremental δstride. The resulting syllabus starts with δ = δmin, the entry level difficulty; and ends whenever δ ≥ δmax, the maximum difficulty that the user can handle. Anywhere in between, the loop deactivates all rules whose difficulties are beyond current δ, and moves onto the next difficulty level δ + δstride if the student’s probabilistic model is γ-close to the input under all currently active rule features.
To showcase syllabus customization, we introduce an ambitious user who demands a faster pace and a patient user who prefers a slower one. In practice, one can collectively tune the stride parameter δstride and the gap parameter γ, with a faster pace corresponding to a larger δstride (let’s jump directly to the junior year from freshman) and a larger γ (having an A- is good enough to move onto the next level, why bother having A+). Here we simply fix δstride, and let γ control the pace. We illustrate two syllabi in Table 1, which compares the first ten (1-gram) rules in a faster (γ = 0.5) syllabus and a slower one (γ = 0.1). Notice the faster syllabus gives the fundamentals that a music student will typically learn in her first-year music theory class, including rules on voice crossing,
pitch class set (scale), intervals, and so on (triads and seventh chords will appear later). It effectively skips the nitty-gritty rules (marked by an asterisk) that are learned in the slower setting. Most of these skipped rules do not have direct counterparts in music theory (such as taking the diff operator twice) and are not important, although occasionally the faster syllabus will skip some rules worth mentioning (such as the second rule in the slower pace, which talks about spacing among soprano, alto, and bass). Setting an appropriate pace for a user is important: a pace that is too fast will miss the whole point of knowledge discovery (jump to the low-level details too fast); a pace that is too slow will bury the important points among unimportant ones (hence, lose the big picture).
Fundamentals: Hierarchical 1-gram Similar to our teaching of music theory, MUS-ROVER II’s proposed syllabus divides into two stages: fundamentals and part writing. The former is under the 1-gram setting, involving knowledge independent of the context; the latter provides online tutoring under multi-n-grams. We begin our experiments with fundamentals, and use them to illustrate the two types of feature hierarchies.
Let’s take a closer look at the two syllabi in Table 1. The specifications (left) and hierarchies (right) of the four common rules are illustrated in Table 2. The rules’ translations are below the corresponding bar charts, all of which are consistent with our music theory. Extracted from the conceptual hierarchy, the right column lists the partition sub-family sourced at each rule, which is pictorially simplified as a tree by hiding implied edges from its corresponding DAG. Every coarser partition in
a sub-family is indeed a higher-level representation, but has not accumulated sufficient significance to make itself a rule. A partition will never be learned if one of its finer ancestors has been made a rule. Observe that all of the coarser partitions are not typically taught in theory classes.
MUS-ROVER II measures the student’s progress from many different angles in terms of features. With respect to a feature, the gap between the student and Bach is iteratively recorded to form a trajectory when cycling the loop. Studying the vanishing point of the trajectory reveals the (local) informational hierarchy around the corresponding feature. Taking the second and seventh rule in the slower syllabus for example, we plot their trajectories in Figure 3. Both illustrate a decreasing trend1 for gaps in the corresponding feature spaces. The left figure shows that the second rule is largely but not entirely implied by the first, pointing out the hierarchical structure between the two: the first rule may be considered as the dominant ancestor of the second, which is not conceptually apparent, but informationally implied. On the contrary, the right figure shows that the seventh rule is not predominantly implied by the first, which instead is informationally connected to many other rules. However, one could say that it is probably safe to skip both rules in light of a faster pace, since they will eventually be learned fairly effectively (with small gaps) but indirectly.
Part Writing: Adaptive n-grams Unlike fundamentals which studies sonority independently along the vertical direction of the chorale texture, rules on part writing (e.g. melodic motion, chord progression) are horizontal, and context-dependent. This naturally results in an online learning framework, in which rule extractions are coupled in the writing process, specific to the realization of a composition (context). Context dependence is captured by the multi-n-gram language model, which further leads to the 2D memory pool of features for rule extraction (Sec. 5). Consider an example of online learning and adaptive memory selection, where we have the beginning of a chorale:
〈s〉 → (60, 55, 52, 36)→ (60, 55, 52, 36)→ (62, 59, 55, 43)→ (62, 59, 55, 43)→ (62, 59, 55, 43), and want to learn the probabilistic model for the next sonority. Instead of starting from scratch, MUS-ROVER II launches the self-learning loop with the ruleset initialized by the fundamentals (incremental learning), and considers the 2D memoryN×Φ, forN = {2, 3, 4, 5}. The first extracted rule is featured by order ◦ sort ◦ mod12 ◦ w{3,4}. The rule is chosen because its corresponding feature has a large confidence level (validated by the large number of matched examples), a small entropy after being smoothed by Bayesian surprise, and reveals a large gap against the Bach’s style. Figure 4 shows the relative performance of this rule (in terms of confidence, regularity, and style gap) to other candidate cells in the 2D memory. Among the top 20 rules for this sonority, 12 are 5-gram, 5 are 4-gram, 3 are 2-gram, showing a long and adaptive dependence to preceding context.
Visualizing Bach’s Mind With the hierarchical representations in MUS-ROVER II, we are now able to visualize Bach’s music mind step by step via activating nodes in the DAG of rule features
1Fluctuations on the trajectory are largely incurred by the imperfect solver of the optimization problem.
(similar to neuron activations in a brain). The hierarchical structure, as well as the additive activation process, is in stark contrast with the linear sequence of rules extracted from our prior work (Appendix A.5). Figure 5 shows a snapshot of the rule-learning status after ten loops, while the student is writing a sonority in the middle of a piece. The visualization makes it clear how earlier independent rules are now self-organized into sub-families, as well as how rules from a new context overwrite those from an old context, emphasizing that music is highly context-dependent.
7 CONCLUSIONS AND DISCUSSIONS
Learning hierarchical rules as distributed representations of tonal music has played a central role in music pedagogy for centuries. While our previous work achieved the automation of rule extraction, and to certain level, the interpretability of the rules, this paper yields deeper interpretability that extends to a system of rules and the overall learning process. In summary, it highlights the importance of disentangling the rule features, sorting out their interconnections, and making the concept learning process more dynamic, hierarchical, and cognitive.
MUS-ROVER is targeted to complement music teaching and learning. For instance, to many music students, learning and applying rules in part-writing is like learning to solve a puzzle (like Sudoku). Rules themselves are quite flexible as opposed to 0-1 derivatives, and may sometimes be contradictory. In addition, due to the limitation of human short-term memory and the difficulty of foreseeing implications, one has to handle a small set of rules at a time in a greedy manner, make some trials, and undo a few steps if no luck. Hence, solving this music puzzle could become a struggle (or maybe interesting): according to personal preferences, one typically begins with a small set of important rules, and via several steps of trial and error, tries one’s best to make the part-writing satisfy a majority of rules, with occasional violations on unimportant ones. On the other hand, a machine is often good at solving and learning from puzzles due to its algorithmic nature. For instance, MUSROVER’s student can take all rules into consideration: load them all at a time as constraints and figure out the global optimum of the optimization problem in only a few hours. The same level of efficiency might take a human student years to achieve.
We envision the future of MUS-ROVER as a partner to humans in both music teaching and research, which includes but is not limited to, personalizing the learning experience of a student, as well as suggesting new methodologies to music theorists in analyzing and developing new genres. It also has practical applications: as by-products from the self-learning loop, the teacher can be made into a genre classifier, while the student can be cast into a style synthesizer. We are also eager to study the rover’s partnership beyond the domain of music.
ACKNOWLEDGMENTS
We thank Professor Heinrich Taube, President of Illiac Software, Inc., for providing Harmonia’s MusicXML corpus of Bach’s chorales (https://harmonia.illiacsoftware.com/), as well as his helpful comments and suggestions. This work was supported by the IBM-Illinois Center for Cognitive Computing Systems Research (C3SR), a research collaboration as part of the IBM Cognitive Horizons Network.
A APPENDIX
A.1 MATH-TO-MUSIC DICTIONARY
A.2 ATOMIC ARITHMETIC OPERATORS
In MUS-ROVER II, we set B = {order, diff, sort, mod12}, where diff(x) = (x2 − x1, x3 − x2, · · · ), ∀x ∈ Ω2 ∪ Ω3 ∪ Ω4; sort(x) = (x(1), x(2), · · · ), ∀x ∈ Ω2 ∪ Ω3 ∪ Ω4; mod12(x) = (mod(x1, 12),mod(x2, 12), · · · ), ∀x ∈ Ω ∪ Ω2 ∪ Ω3 ∪ Ω4;
and order(x), similar to argsort, maps x ∈ Ω2 ∪ Ω3 ∪ Ω4 to a string that specifies the ordering of its elements, e.g. order((60, 55, 52, 52)) = “4=3<2<1”. The numbers in an order string denote the indices of the input vector x.
A.3 ALGORITHM FOR CONCEPTUAL HIERARCHY
Input: A family of distinct partitions, represented by a sorted list P = [p1, . . . , pn]: pi 6= pj, for all i 6= j, and |p1| ≤ . . . ≤ |pn|;
Output: The conceptual hierarchy as a DAG, represented by the n by n adjacency matrix T: T[i, j] = 1 if there is an edge from node i to node j in the DAG;
initialize T[i, j] = 0 for all i, j; for i = n : 1 do
for j = (i + 1) : n do if T[i, j] == 0 then
if is coarser(pi, pj) then for k in {k | pj ≺ pk} ∪ {j} do
T[i, k] = 1; end
end end
end end T = Transpose(T);
Algorithm 1: Algorithm for computing the conceptual hierarchy
A.4 HEURISTICS FOR COMPARING TWO PARTITIONS
Given two partitions P,Q from the partition family, the function is coarser(P,Q) in Algorithm 1 returns True if P ≺ Q. A brute-force implementation of this function involves studying all (unordered) pairs of elements in the input domain (Hubert & Arabie, 1985), which incurs computational
burdens if the size of the input domain is large. Therefore, we try to get around this brute-force routine whenever certain heuristic can be used to infer the output of is coarser directly. We propose a few of these heuristics as follows.
Transitivity Heuristic If Pφ Pφ′ and Pφ′ Pφ′′ , then Pφ Pφ′′ . Window Heuristic Let Pφ and Pφ′ be two partitions induced by features φ and φ′, respectively. In addition, φ and φ′ are generated from the same descriptor that preserves the orders of the inputs’ coordinates, e.g. diff, mod12: φ = d ◦ wI , φ′ = d ◦ wI′ . We claim that Pφ Pφ′ , if I ⊃ I ′ and |φ(Ω4)| > |φ′(Ω4)|. To see why this is the case, pick any x, y ∈ Ω4 from the same cluster in Pφ, then φ(x) = φ(y). Since d preserves the orders of the inputs’ coordinates, and I ′ extracts coordinates from I , then φ′(x) = φ′(y), i.e. x, y are in the same cluster in Pφ′ . So, by definition, Pφ Pφ′ . Since |φ(Ω4)| > |φ′(Ω4)|, Pφ Pφ′ . Descriptor Heuristic Let Pφ and Pφ′ be two partitions induced by features φ and φ′, respectively. In addition, φ and φ′ are generated from the same window:
φ = d ◦ wI ; φ′ = d′ ◦ wI . We claim that Pφ Pφ′ , if d′ = b ◦ d for some function b and |φ(Ω4)| > |φ′(Ω4)|. To see why this is the case, pick any x, y ∈ Ω4 from the same cluster in Pφ, then φ(x) = φ(y). Since d′ = b ◦ d for some b, then φ′ = b ◦ d ◦ wI = b ◦ φ, thus, φ′(x) = φ′(y), i.e. x, y are in the same cluster in Pφ′ . So, by definition, Pφ Pφ′ . Since |φ(Ω4)| > |φ′(Ω4)|, Pφ Pφ′ . Combined Heuristic Combining the above heuristics, one can show that for Pφ and Pφ′ where
φ = d ◦ wI ; φ′ = d′ ◦ wI′ , we have Pφ Pφ′ , if the following conditions are satisfied:
1) d, d′ both preserve the orders of the inputs’ coordinates,
2) d′ = b ◦ d for some b,
3) I ⊃ I ′, 4) |φ(Ω4)| > |φ′(Ω4)|.
A.5 SAMPLE RULE TRACES FROM MUS-ROVER I
Table 4 is essentially the same as Table 2 in our previous publication (Yu et al., 2016a), with feature notations following the current fashion. α is the pace control parameter that we used in our previous system. No hierarchy was present in any of the three rule traces. For instance, the ordering features were learned as independent rules in a trace, even if they are apparently correlated, e.g. the ordering of w{1,2,3,4} (S,A,T,B) implies the ordering of w{1,4} (S,B).

Summary:
The paper presents an advanced self-learning model that extracts compositional rules from Bach's chorales, which extends their previous work in: 1) the rule hierarchy in both conceptual and informational dimensions; 2) adaptive 2-D memory selection which assumes the features follow Dirichlet Distribution. Sonority (column of 4 MIDI numbers) acts as word in language model: unigram statistics have been used to learn the fundamental rules in music theory, while n-grams with higher order help characterize part writing. Sonorities have been clustered together based on feature functions through iterations. The partition induced by the features is recognized as a rule if it is sufficiently significant. As a result, two sample syllabi with different difficulty strides and "satisfactory gaps" have been generated in terms of sets of learned rules.
1. Quality:
 a) Strengths: In the paper, the exploration of hierarchies in two dimensions makes the learning process more cognitive and interpretable. The authors also demonstrate an effective memory selection to speed up the learning.
 b) Flaws: The paper only discussed N<=5, which might limit the learning and interpretation capacities of the proposed model, failing to capture long-distance dependence of music. (In the replies to questions, the authors mentioned they had experimented with max N=10, but I'm not sure why related results were not included in the paper). Besides the elaborated interpretation of results, a survey seeking the opinions of students in a music department might make the evaluation of system performance more persuasive.
2. Clarity:
 a) Pros: The paper clearly delivers an improved automatic theorist system which learns and represents music concepts as well as thoroughly interprets and compares the learned rules with music theory. Proper analogies and examples help the reader perceive the ideas more easily.
 b) Cons: Although detailed definitions can be found in the authors' previous MUS-ROVER I papers, it would be great if they had described the optimization more clearly (in Figure 1. and related parts).  The "(Conceptual-Hierarchy Filter)" row in equations (3): the prime symbol should appear in the subscript.
3. Originality:
The representation of music concepts and rules is still an open area, the paper investigate the topic in a novel way. It illustrates an alternative besides other interpretable feature learning methods such as autoencoders, GAN, etc.
4. Significance:
It is good to see some corresponding interpretations for the learned rules from music theory. The authors mentioned students in music could and should be involved in the self-learning loop to interact, which is very interesting. I hope their advantages can be combined in the practice of music theory teaching and learning.


In this work we propose a simple unsupervised approach for next frame prediction in video. Instead of directly predicting the pixels in a frame given past frames, we predict the transformations needed for generating the next frame in a sequence, given the transformations of the past frames. This leads to sharper results, while using a smaller prediction model. In order to enable a fair comparison between different video frame prediction models, we also propose a new evaluation protocol. We use generated frames as input to a classifier trained with ground truth sequences. This criterion guarantees that models scoring high are those producing sequences which preserve discriminative features, as opposed to merely penalizing any deviation, plausible or not, from the ground truth. Our proposed approach compares favourably against more sophisticated ones on the UCF-101 data set, while also being more efficient in terms of the number of parameters and computational cost.
1 INTRODUCTION
There has been an increased interest in unsupervised learning of representations from video sequences (Mathieu et al., 2016; Srivastava et al., 2015; Vondrick et al., 2016). A popular formulation of the task is to learn to predict a small number of future frames given the previous K frames; the motivation being that predicting future frames requires understanding how objects interact and what plausible sequences of motion are. These methods directly aim to predict pixel values, with either MSE loss or adversarial loss.
In this paper, we take a different approach to the problem of next frame prediction. In particular, our model operates in the space of transformations between frames, directly modeling the source of variability. We exploit the assumption that the transformations of objects from frame to frame should be smooth, even when the pixel values are not. Instead of predicting pixel values, we directly predict how objects transform. The key insight is that while there are many possible outputs, predicting one such transformation will yield motion that may not correspond to ground truth, yet will be realistic; see fig. 1. We therefore propose a transformation-based model that operates in the space of affine transforms. Given the affine transforms of a few previous frames, the model learns to predict the local affine transforms that can be deterministically applied on the image patches of the previous frame to generate the next frame. The intuition is that estimation errors will lead to a slightly different yet plausible motion. Note that this allows us to keep using the MSE criterion, which is easy to optimize, as long as it is in transformation space. No blur in the pixel space will be introduced since the output of the transformation model is directly applied to the pixels, keeping sharp edges intact. Refer to fig. 5 and our online material 1 for examples.
The other contribution of this work is the evaluation protocol. Typically, generative models of video sequences are evaluated in terms of MSE in pixel space (Srivastava et al., 2015), which is not a good choice since this metric favors blurry predictions over other more realistic looking options that just happen to differ from the ground truth. Instead, we propose to feed the generated frames to a video
∗Work done as part of internship with FAIR 1see: http://joo.st/ICLR/GenerationBenchmark
classifier trained on ground truth sequences. The idea is that the less the classifier’s performance is affected by the generates frames the more the model has preserved distinctive features and the more the generated sequences are plausible. Regardless of whether they resemble the actual ground truth or not. This protocol treats the classifier as a black box to measure how well the generated sequences can serve as surrogate for the truth sequence for the classification task. In this paper we will validate our assumption that motion can be modelled by local affine transforms, after which we will compare our method with networks trained using adversarial training and simple regression on the output frame, using both this new evaluation protocol and by providing samples for qualitative inspection.
Our experiments show that our simple and efficient model outperforms other baselines, including much more sophisticated models, on benchmarks on the UCF-101 data set (Soomro et al., 2012). We also provide qualitative comparisons to the moving MNIST digit data set (Srivastava et al., 2015).
1.1 RELATED WORK
Early work on video modeling focused on predicting small patches (Michalski et al., 2014; Srivastava et al., 2015); unfortunately, these models have not shown to scale to the complexity of highresolution videos. Also these models require a significant amount of parameters and computational power for even relatively simple data.
In Ranzato et al. (2014), the authors circumvented this problem by quantizing the space of image patches. While they were able to predict a few high-resolution frames in the future, it seems dissatisfying to impose such a drastic assumption to simplify the prediction task.
Mathieu et al. (2016) recently proposed to replace MSE in pixel space with a MSE on image gradients, leveraging prior domain knowledge, and further improved using a multi-scale architecture with adversarial training (Goodfellow et al., 2014). While producing better results than earlier methods, the models used require a very large amount of computational power. We make an explicit comparison to this paper in the experiments section 3.
In Oh et al. (2015), frames of a video game are predicted given an action (transformation) taken by the player. While the paper shows great results, the movement in a natural video cannot be described by a simple action and is therefore not widely applicable. Finally, our work is also related to optical flow estimation (Brox et al., 2004). Instead of estimating the flow of pixels, here we estimate the flow of patches and separately predict how these patches transform in future frames.
Prior work relating to the evaluation protocol can be found in Yan et al. (2015). The authors generate images using a set of predefined attributes and later show that they can recover these using a pretrained neural network. Our proposal extends this to videos, which is more complicated since both appearance and motion are needed for correct classification.
2 MODEL
The model we propose is based on three key assumptions: 1) just estimating object motion yields sequences that are plausible and relatively sharp, 2) global motion can be estimated by tiling highresolution video frames into patches and estimating motion “convolutionally” at the patch level, and 3) patches at the same spatial location over two consecutive time steps undergo a deformation which can be well described by an affine transformation.
The first assumption is at the core of the proposed method: by considering uncertainty in the space of transformations we produce sequences that may still look plausible. The other two assumptions state that a video sequence can be composed by patches undergoing affine transformations. We agree that these are simplistic assumptions, which ignore how object identity affects motion and do not account for out of plane rotations and more general forms of deformation. However, our qualitative and quantitative evaluation shows the efficacy of these assumptions to real video sequence as can be seen in section 3 and from visualizations in the supplementary material2.
Our approach consists of three steps. First, we estimate affine transforms of every video sequence to build a training set for our model. Second, we train a model that takes the past N affine transforms and predicts the next M affine transforms. Finally, at test time, the model uses the predicted affine transforms to reconstruct pixel values of the generated sequence. We describe the details of each phase in the following sections.
2.1 AFFINE TRANSFORM EXTRACTOR
Given a frame x and the subsequent frame y, the goal of the affine transform extractor is to learn mappings that can warp x into y. Since different parts of the scene may undergo different transforms, we tile x into overlapping patches and infer a transformation for each patch. The estimation process couples the transformations at different spatial locations because we minimize the reconstruction error of the entire frame y, as opposed to treating each patch independently.
2see: http://joo.st/ICLR/ReconstructionsFromGroundTruth and http://joo.st/ ICLR/GenerationBenchmark
Let x and y have size Dr ×Dc. Let image x be decomposed into a set of overlapping patches, each containing pixels from patches of size dr×dc with dr ≤ Dr and dc ≤ Dc. These patches are laid out on a regular grid with stride sr and sc pixels over rows and columns, respectively. Therefore, every pixel participates in drsr dc sc
overlapping patches, not taking into account for the sake of simplicity border effects and non-integer divisions. We denote the whole set of overlapping patches by {Xk}, where index k runs over the whole set of patches. Similarly and using the same coordinate system, we denote by {Yk} the set of overlapping patches of y. We assume that there is an affine mapping Ak that maps Xk to Yk, for all values of k. Ak is a 2 × 3 matrix of free parameters representing a generic affine transform (translation, rotation and scaling) between the coordinates of output and input frame. Let Ỹk be the transformed patches obtained when Ak is applied to Xk. Since coordinates overlap between patches, we reconstruct y by averaging all predictions at the same location, yielding the estimate ỹ. The joint set of Ak is then jointly determined by minimizing the mean squared reconstruction error between y and ỹ.
Notice that our approach and aim differs from spatial transformer networks (Jaderberg et al., 2015) since we perform this estimation off-line only for the input frames, computing one transform per patch.
In our experiments, we extracted 16 × 16 pixel patches from the input and we used stride 4 over rows and columns. The input patches are then matched at the output against smaller patches of size 8× 8 pixels, to account for objects moving in and out of the patch region.
2.2 AFFINE TRANSFORM PREDICTOR
The affine transform predictor is used to predict the affine transforms between the last input frame and the next frame in the sequence. A schematic illustration of the system is shown in fig. 2. It receives as input the affine transforms between pairs of adjacent frames, as produced by the affine transform extractor described in the previous section. Each transform is arranged in a grid of size 6 × n × n, where n is the number of patches in a row/column and 6 is the number of parameters of each affine transform. Therefore, if four frames are used to initialize the model, the actual input consists of 18 maps of size n × n, which are the concatenation of At−2, At−1, At, where At is the collection of patch affine transforms between frame at time t− 1 and t. The model consists of a multi-layer convolutional network without any pooling. The network is the composition of convolutional layers with ReLU non-linearity, computing a component-wise thresholding as in v = max(0, u). We learn the parameters in the filters of the convolutional layers by minimizing the mean squared error between the output of the network and the target transforms.
Notice that we do not add any regularization to the model. In particular, we rely on the convolutional structure of the model to smooth out predictions at nearby spatial locations.
2.3 MULTI-STEP PREDICTION
In the previous section, we described how to predict the set of affine transforms at the next time step. In practice, we would like to predict several time steps in the future.
A greedy approach would: a) train as described above to minimize the prediction error for the affine transforms at the next time step, and b) at test time, predict one step ahead and then re-circulate the model prediction back to the input to predict the affine transform two steps ahead, etc. Unfortunately, errors may accumulate throughout this process because the model was never exposed to its own predictions at training time.
The approach we propose replicates the model over time, also during training as shown in fig. 3. If we wish to predict M steps in the future, we replicate the CNN M times and pass the output of the CNN at time step t as input to the same CNN at time step t + 1, as we do at test time. Since predictions live in a continuous space, the whole system is differentiable and amenable to standard back-propagation of the error. Since parameters of the CNN are shared across time, the overall system is equivalent to a peculiar recurrent neural network, where affine transforms play the role of recurrent states. The experiments in section 3 demonstrate that this method is more accurate and robust than the greedy approach.
2.4 TESTING
At test time, we wish to predict M frames in the future given the past N frames. After extracting the N − 1 affine transforms from the frames we condition upon, we replicate the model M times and feed its own prediction back to the input, as explained in the previous section.
Once the affine transforms are predicted, we can reconstruct the actual pixel values. We use the last frame of the sequence and apply the first set of affine transforms to each patch in that frame. Each pixel in the output frame is predicted multiple times, depending on the stride used. We average these predictions and reconstruct the whole frame. As required, we can repeat this process for as many frames as necessary, using the last reconstructed frame and the next affine transform.
In order to evaluate the generation, we propose to feed the generated frames to a trained classifier for a task of interest. For instance, we can condition the generation using frames taken from video clips which have been labeled with the corresponding action. The classifier has been trained on ground truth data but it is evaluated using frames fantasized by the generative model. The performance of the classifier on ground truth data is an upper bound on the performance of any generative model. This evaluation protocol does not penalize any generation that deviates from the ground truth, as standard MSE would. It instead check that discriminative features and the overall semantics of the generated sequence is correct, which is ultimately what we are interested in.
3 EXPERIMENTS
In this section, we validate the key assumptions made by our model and compare against state-ofthe-art generative models on two data sets. We strongly encourage the reader to watch the short video clips in the Supplementary Material to better understand the quality of our generations.
In section 2, we discussed the three key assumptions at the foundations of our model: 1) errors in the transformation space look still plausible, 2) a frame can be decomposed into patches, and 3) each patch motion is well modeled by an affine transform. The results in the Supplementary Material 3 validate assumption 2 and 3 qualitatively. Every row shows a sequence from the UCF101 dataset (Soomro et al., 2012). The column on the left shows the original video frames and the one on the right the reconstructions from the estimated affine transforms, as described in section 2.1. As you can see there is barely any noticeable difference between these video sequences, suggesting that video sequences can be very well represented as tiled affine transforms. For a quantitative comparison and for an assessment of how well the first assumption holds, please refer to section 3.2.
3see: http://joo.st/ICLR/ReconstructionsFromGroundTruth
In the next section, we will first report some results using the toy data set of “moving MNIST digits” (Srivastava et al., 2015). We then discuss generations of natural high-resolution videos using the UCF-101 dataset and compare to current state-of-the-art methods.
3.1 MOVING MNIST
For our first experiment, we used the dataset of moving MNIST digits (Srivastava et al., 2015) and perform qualitative analysis4. It consists of one or two MNIST digits, placed at random locations and moving at constant speed inside a 64 × 64 frame. When a digit hits a boundary, it bounces, meaning that velocity in that direction is reversed. Digits can occlude each other and bounce off walls, making the data set challenging.
Using scripts provided by Srivastava et al. (2015), we generated a fixed dataset of 128,000 sequences and used 80% for training, 10% for validation and 10% for testing. Next, we estimated the affine transforms between every pair of adjacent frames to a total of 4 frames, and trained a small CNN in the space of affine transforms. The CNN has 3 convolutional layers and the following number of feature maps: 18, 32, 32, 6. All filters have size 3× 3. Fig. 4 shows some representative test sequences and the model outputs. Each subfigure corresponds to a sequence from the test set; the top row corresponds to the ground truth sequence while the bottom row shows the generations. The input to the CNN are three sets of affine transforms corresponding to the first four consecutive frames. The network predicts the next six sets of affine transforms from which we reconstruct the corresponding frames. These results should be compared to fig. 5 in Srivastava et al. (2015). The generations in fig. 4 show that the model has potential to represent and generate video sequences, it learns to move digits in the right direction, to bounce them, and it handles multiple digits well except when occluion makes inputs too ambiguous. The model’s performance is analyzed quantitatively in the next section using high resolution natural videos.
3.2 UCF 101 DATA SET
The UCF-101 dataset (Soomro et al., 2012) is a collection of 13320 videos of 101 action categories. Frames have size 240× 320 pixels. We train a CNN on patches of size 64× 64 pixels; the CNN has 6 convolutional layers and the following number of feature maps: 18, 128, 128, 128, 64, 32, 16, 6. All filters have size 3 × 3. The optimal number of filters has been found using cross-validation in order to minimize the estimation error of the affine transform parameters. Unless otherwise stated, we condition generation on 4 ground truth frames and we predict the following 8 frames.
We evaluate several models5: a) a baseline which merely copies the last frame used for conditioning, b) a baseline method which estimates optical flow (Brox et al., 2004) from two consecutive frames
4A quantitative analysis would be difficult for this data set because metrics reported in the literature like MSE (Srivastava et al., 2015) are not appropriate for measuring generation quality, and it would be difficult to use the metric we propose because we do not have labels at the sequence level and the design of a classifier is not trivial.
5Unfortunately, we could not compare against the LSTM-based method in Srivastava et al. (2015) because it does not scale to high-resolution videos, but only to small patches.
and extrapolates flow in subsequent frames under the assumption of constant flow speed, c) an adversarially trained multi-scale CNN (Mathieu et al., 2016) and several variants of our proposed approach.
Qualitative comparisons can be seen in the fig. 5 and in the supplementary material6. The first column on the page shows the input, the second the ground truth, followed by results from our model, Mathieu et al. (2016) and optical flow (Brox et al., 2004). Note especially the severe deformations in the last two columns, while our model keeps the frame recognizable. It produces fairly sharp reconstructions validating our first hypothesis that errors in the space of transformations still yield plausible reconstructions (see section 2). However it is also apparent that our approach underestimates movement, which follows directly from using the MSE criterion. As discussed before, MSE in pixel space leads to blurry results, however using MSE in transformation space also has some drawbacks. In practice, the model will predict the average of several likely transformations, which could lead to an understimation of the true movement.
6see: http://joo.st/ICLR/GenerationBenchmark
In order to quantify the generation quality we use the metric described in section 2.4. We use C3D network (Tran et al., 2015) as the video action classifier: C3D uses both appearance and temporal information jointly, and is pre-trained with Sports1M (Karpathy et al., 2014) and fine tuned on UCF 101. Due to the model constraints, we trained only two models, that takes 4 and 8 frames as input, respectively.
We evaluate the quality of generation using 4 (the first four predicted frames) and the whole set of 8 predicted frames, for the task of action classification. At test time, we generate frames from each model under consideration, and then use them as input to the corresponding C3D network.
Table 1 shows the accuracy of our approach and several baselines. The best performance is achieved by using ground truth frames, a result comparable to methods recently appeared in the literature (Karpathy et al., 2014; Tran et al., 2015). We see that for ground truth frames, the number of frames (4 or 8) doesn’t make a difference. There is not much additional temporal or spatial signal provided by having greater than four frames. Next, we evaluate how much we lose by representing frames as tiled affine transforms. As the second row shows there is negligible if any loss of accuracy when using frames reconstructed from the estimated affine transforms (using the method described in section 2.1), validating our assumptions at the beginning of section 2 on how video sequences can be represented. The next question is then whether these affine transforms are predictable at all. The last two rows of Table 1 show that this is indeed the case, to some extent. The longer the sequence of generated frames the poorer the performance, since the generation task gets more and more difficult.
Compared to other methods, our approach performs better than optical flow and even the more sophisticated multi-scale CNN proposed in Mathieu et al. (2016) while being computationally cheaper. For instance, our method has less than half a million parameters and requires about 2G floating point operations to generate a frame at test time, while the multi-scale CNN of Mathieu et al. (2016) has 25 times more parameters (not counting the discriminator used at training time) and it requires more than 100 times more floating point operations to generate a single frame.
Finally, we investigate the robustness of the system to its hyper-parameters: a) choice of patch size, b) number of input frames, and c) number of predicted frames. The results reported in Table 2 demonstrate that the model is overall pretty robust to these choices. Using patch sizes that are too big makes reconstructions blocky but within each block motion is coherent. Smaller patch sizes give more flexibility but make the prediction task harder as well. Mapping into patches of size smaller than 16 × 16 seems a good choice. Using only 2 input frames does not seem to provide enough context to the predictor, but anything above 3 works equally well. Training for prediction of the next frame works well, but better results can be achieved by training to predict several frames in the future, overall when evaluating longer sequences.
4 CONCLUSIONS
In this work, we proposed a new approach to generative modeling of video sequences. This model does not make any assumption about the spatio-temporal resolution of video sequences nor about object categories. The key insight of our approach is to model in the space of transformations as opposed to raw pixel space. A priori we lack a good metric to measure how well a frame is reconstructed under uncertainty due to objects motion in natural scenes. Uncertainty about object motion and occlusions causes blurry generations when using MSE in pixel space. Instead, by operating in the space of transformations we aim at predicting how objects move, and estimation errors only yield a different, and possibly still plausible, motion. With this motivation we proposed a simple CNN operating in the space of affine transforms and we showed that it can generate sensible sequences up to about 4 frames. This model produces sequences that are both visually and quantitatively better than previously proposed approaches.
The second contribution of this work is the metric to compare generative models of video sequences. A good metric should not penalize a generative model for producing a sequence which is plausible but different from the ground truth. With this goal in mind and assuming we have at our disposal labeled sequences, we can first train a classifier using ground truth sequences. Next, the classifier is fed with sequences produced by our generative model for evaluation. A good generative model should produce sequences that still retain discriminative features. In other words, plausibility of generation is assessed in terms of how well inherent information is preserved during generation as opposed to necessarily and merely reproducing the ground truth sequences.
The proposed model is relatively simple; straightforward extensions that could improve its prediction accuracy are the use of a multi-scale architecture and the addition of recurrent units. These would enable a better modeling of objects of different sizes moving at varying speeds and to better capture complex temporal dynamics (e.g., cyclical movements like walking). A larger extension would be the addition of an appearance model, which together with our explicit transformation model could lead to learning better feature representations for classification.
In our view, the proposed approach should be considered as a stronger baseline for future research into next frame prediction. Even though our analysis shows improved performance and better looking generations, there are also obvious limitations. The first such limitation is the underestimation of transformations due to usage of the MSE as a criterion. We consider two main avenues worth pursuing in this space. First, we consider modelling a distribution of transformations and sampling one from it. The challenge of this approach is to sample a consistent trajectory. One could model the distribution of an entire trajectory, but that is a complex optimization problem. A second option is to use adversarial training to force the model to pick a plausible action. This option does not guarantee that underestimation of movement will be avoided. This will depend on the discriminator model accepting this as a plausible option.
Another limitation is that the current model does not factor out the “what” from the “where”, appearance from motion. The representation of two distinct objects subject to the same motion, as well as the representation of the same object subject to two different motion patterns are intrinsically different. Instead, it would be more powerful to learn models that can discover such factorization and leverage it to produce more efficient and compact representations.
ACKNOWLEDGMENTS
Authors thank Camille Couprie and Michael Mathieu for discussions and helping with evaluation of their models.

Paper Summary
This paper makes two contributions -
(1) A model for next step prediction, where the inputs and outputs are in the
space of affine transforms between adjacent frames.
(2) An evaluation method in which the quality of the generated data is assessed
by measuring the reduction in performance of another model (such as a
classifier) when tested on the generated data.
The authors show that according to this metric, the proposed model works better
than other baseline models (including the recent work of Mathieu et al. which
uses adversarial training).
Strengths
- This paper attempts to solve a major problem in unsupervised learning
  with videos, which is evaluating them.
- The results show that using MSE in transform space does prevent the blurring
  problem to a large extent (which is one of the main aims of this paper).
- The results show that the generated data reduces the performance of the C3D
  model on UCF-101 to a much less extent than other baselines.
- The paper validates the assumption that videos can be approximated to quite a
  few time steps by a sequence of affine transforms starting from an initial
frame.
Weaknesses
- The proposed metric makes sense only if we truly just care about the performance
  of a particular classifier on a given task. This significantly narrows the
scope of applicability of this metric because arguably, one the important
reasons for doing unsupervised learning is to come up a representation that is
widely applicable across a variety of tasks. The proposed metric would not help
evaluate generative models designed to achieve this objective.
- It is possible that one of the generative models being compared will interact
  with the idiosyncrasies of the chosen classifier in unintended ways.
Therefore, it would be hard to draw strong conclusions about the relative
merits of generative models from the results of such experiments. One way to
ameliorate this would be to use several different classifiers (C3D,
dual-stream network, other state-of-the-art methods) and show that the ranking
of different generative models is consistent across the choice of classifier.
Adding such experiments would help increase certainty in the conclusions drawn
in this paper.
- Using only 4 or 8 input frames sampled at 25fps seems like very little context
  if we really expect the model to extrapolate the kind of motion seen in
UCF-101. The idea of working in the space of affine transforms would be much
more appealing if the model can be shown to really generated non-trivial motion
patterns. Currently, the motion patterns seem to be almost linear
extrapolations.
- The model that predicts motion does not have access to content at all. It only
  gets access to previous motion. It seems that this might be a disadvantage
because the motion predictor cannot use any cues like object boundaries, or
decide what to do when two motion fields collide (it is probably easier to argue
about occlusions in content space).
Quality/Clarity
The paper is clearly written and easy to follow. The assumptions are clearly
specified and validated. Experimental details seem adequate.
Originality
The idea of generating videos by predicting motion has been used previously.
Several recent papers also use this idea. However the exact implementation in
this paper is new. The proposed evaluation protocol is novel.
Significance
The proposed evaluation method is an interesting alternative, especially if it
is extended to include multiple classifiers representative of different
state-of-the-art approaches. Given how hard it is to evaluate generative models
of videos, this paper could help start an effort to standardize on a benchmark
set.
Minor comments and suggestions
(1) In the caption for Table 1: ``Each column shows the accuracy on the test set
when taking a different number of input frames as input" - ``input" here refers
to the input to the classifier (Output of the next step prediction model). However
in the next sentence ``Our approach maps 16 \times 16 patches into 8 \times 8
with stride 4, and it takes 4 frames at the input" - here ``input" refers to
the input to the next step prediction model. It might be a good idea to rephrase
these sentences to make the distinction clear.
(2) In order to better understand the space of affine transform
parameters, it might help to include a histogram of these parameters in the
paper. This can help us see at a glance, what is the typical range of these
6 parameters, should we expect a lot of outliers, etc.
(3) In order to compare transforms A and B, instead of ||A - B||^2, one
could consider A^{-1}B being close to identity as the metric. Did the authors
try this ?
(4) "The performance of the classifier on ground truth data is an upper bound on
the performance of any generative model." This is not *strictly* true. It is
possible (though highly unlikely) that a generative model might make the data
look cleaner, sharper, or highlight some aspect of it which could improve the
performance of the classifier (even compared to ground truth). This is
especially true if the the generative model had access to the classifier, it
could then see what makes the classifier fire and highlight those discriminative
features in the generated output.
Overall
This paper proposes future prediction in affine transform space. This does
reduce blurriness and makes the videos look relatively realistic (at least to the
C3D classifier). However, the paper can be improved by showing that the model can
predict more non-trivial motion flows and the experiments can be strengthened by
adding more classifiers besides than C3D.