Music Transformer: Generating Music with Long-Term Structure
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity is quadratic in the sequence length. We propose an algorithm that reduces the intermediate memory requirements to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long (thousands of steps) compositions with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-competition, and obtain state-of-the-art results on the latter.
 This paper describes a method for improving the (sequence-length) scalability of the Transformer architecture, with applications to modeling long-range interactions in musical sequences. The proposed improvement is applied to both global and local relative attention formulations of self-attention, and consists of a clever re-use (and re-shaping) of intermediate calculations. The result shaves a factor of L (sequence length) from the (relative) memory consumption, facilitating efficient training of long sequences. The method is evaluated on MIDI(-like) data of Bach chorales and piano performances, and compares favorably to prior work in terms of perplexity and a human listener evaluation. The results in this paper seem promising, though difficult to interpret. The quantitative evaluation consists of perplexity scores (Tables 2 and 3), and the qualitative listening study is analyzed by pairwise comparisons between methods. While the proposed method achieves the highest win-rate in the listening study, other results in the study (LSTM vs Transformer) run contrary to the ranking given by the perplexity scores in Table 3. This immediately raises the question of how perceptually relevant the (small) differences in perplexity might be, which in turn clouds the overall interpretation of the results. Of course, perplexity is not the whole story here: the focus of the paper seems to be on efficiency, not necessarily accuracy, but one might expect improved efficiency to afford higher model capacity and improve on accuracy. The core contributions of this work are described in sections 3.4 and 3.5, and while I get the general flavor of the idea, I find the exposition here both terse and difficult to follow. Figures 1 and 2 should illustrate the core concept, but they lack axis labels (and generally sufficient detail to decode properly), and seem to use the opposite color schemes from each-other to convey the same ideas. Concrete image maps using real data (internal feature activations) may have been easier to read here, along with an equation that describes how the array indices map after skewing. The description in 3.4 of the improved memory enhancement is also somewhat difficult to follow. The claim is a reduction from O(DL^2) to O(DL), but table 1 lists this as O(DL^2) to O(DL + L^2). In general, I would expect L to dominate D, which still leaves the memory usage in quadratic space, so it*s not clear how or why this constitutes an improvement. The improvement due to moving from global to local attention is clear, but this does not appear to be a contribution of this work.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Music Transformer: Generating Music with Long-Term Structure
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity is quadratic in the sequence length. We propose an algorithm that reduces the intermediate memory requirements to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long (thousands of steps) compositions with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-competition, and obtain state-of-the-art results on the latter.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper describes a method for improving the (sequence-length) scalability of the Transformer architecture, with applications to modeling long-range interactions in musical sequences. The proposed improvement is applied to both global and local relative attention formulations of self-attention, and consists of a clever re-use (and re-shaping) of intermediate calculations. The result shaves a factor of L (sequence length) from the (relative) memory consumption, facilitating efficient training of long sequences. The method is evaluated on MIDI(-like) data of Bach chorales and piano performances, and compares favorably to prior work in terms of perplexity and a human listener evaluation. The results in this paper seem promising, though difficult to interpret. The quantitative evaluation consists of perplexity scores (Tables 2 and 3), and the qualitative listening study is analyzed by pairwise comparisons between methods. While the proposed method achieves the highest win-rate in the listening study, other results in the study (LSTM vs Transformer) run contrary to the ranking given by the perplexity scores in Table 3. This immediately raises the question of how perceptually relevant the (small) differences in perplexity might be, which in turn clouds the overall interpretation of the results. Of course, perplexity is not the whole story here: the focus of the paper seems to be on efficiency, not necessarily accuracy, but one might expect improved efficiency to afford higher model capacity and improve on accuracy. The core contributions of this work are described in sections 3.4 and 3.5, and while I get the general flavor of the idea, I find the exposition here both terse and difficult to follow. Figures 1 and 2 should illustrate the core concept, but they lack axis labels (and generally sufficient detail to decode properly), and seem to use the opposite color schemes from each-other to convey the same ideas. Concrete image maps using real data (internal feature activations) may have been easier to read here, along with an equation that describes how the array indices map after skewing. The description in 3.4 of the improved memory enhancement is also somewhat difficult to follow. The claim is a reduction from O(DL^2) to O(DL), but table 1 lists this as O(DL^2) to O(DL + L^2). In general, I would expect L to dominate D, which still leaves the memory usage in quadratic space, so it*s not clear how or why this constitutes an improvement. The improvement due to moving from global to local attention is clear, but this does not appear to be a contribution of this work.
assistant
 This paper describes a method for improving the (sequence-length) scalability of the Transformer architecture, with applications to modeling long-range interactions in musical sequences. The proposed improvement is applied to both global and local relative attention formulations of self-attention, and consists of a clever re-use (and re-shaping) of intermediate calculations. The result shaves a factor of L (sequence length) from the (relative) memory consumption, facilitating efficient training of long sequences. The method is evaluated on MIDI(-like) data of Bach chorales and piano performances, and compares favorably to prior work in terms of perplexity and a human listener evaluation. The results in this paper seem promising, though difficult to interpret. The quantitative evaluation consists of perplexity scores (Tables 2 and 3), and the qualitative listening study is analyzed by pairwise comparisons between methods. While the proposed method achieves the highest win-rate in the listening study, other results in the study (LSTM vs Transformer) run contrary to the ranking given by the perplexity scores in Table 3. This immediately raises the question of how perceptually relevant the (small) differences in perplexity might be, which in turn clouds the overall interpretation of the results. Of course, perplexity is not the whole story here: the focus of the paper seems to be on efficiency, not necessarily accuracy, but one might expect improved efficiency to afford higher model capacity and improve on accuracy. The core contributions of this work are described in sections 3.4 and 3.5, and while I get the general flavor of the idea, I find the exposition here both terse and difficult to follow. Figures 1 and 2 should illustrate the core concept, but they lack axis labels (and generally sufficient detail to decode properly), and seem to use the opposite color schemes from each-other to convey the same ideas. Concrete image maps using real data (internal feature activations) may have been easier to read here, along with an equation that describes how the array indices map after skewing. The description in 3.4 of the improved memory enhancement is also somewhat difficult to follow. The claim is a reduction from O(DL^2) to O(DL), but table 1 lists this as O(DL^2) to O(DL + L^2). In general, I would expect L to dominate D, which still leaves the memory usage in quadratic space, so it*s not clear how or why this constitutes an improvement. The improvement due to moving from global to local attention is clear, but this does not appear to be a contribution of this work.