Controllable Natural Language Generation with Contrastive Prefixes

To guide the generation of large pretrained language models (LM), previous work has focused on directly fine-tuning the language model or utilizing an attribute discriminator. In this work, we propose a novel lightweight framework for controllable GPT2 generation, which utilizes a set of small attribute-specific vectors, called prefixes (Li and Liang, 2021), to steer natural language generation. Different from Li and Liang (2021), where each prefix is trained independently, we take the relationship among prefixes into consideration and train multiple prefixes simultaneously. We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control. Experimental results on both single-aspect and multi-aspect control show that our methods can guide generation towards the desired attributes while keeping high linguistic quality.

Introduction
The goal of controllable Natural Language Generation (NLG) is to guide generation towards the desired attributes in the concerned aspects of the text. For example, the aspect can be topic or sentiment, and sentiment may have two attributes: positive and negative. Previous work has focused on directly fine-tuning the existing models (Keskar et al., 2019;Hu et al., 2017;Ficler and Goldberg, 2017) or using a discriminator to guide generation (Dathathri et al., 2020;Krause et al., 2020;Holtzman et al., 2018). CTRL (Keskar et al., 2019) achieves controllability at the expense of training a large conditional LM. GeDi (Krause et al., 2020) also trains conditional LMs but uses them as discriminators to guide generation, introducing additional 345M parameters. Besides, GeDi focuses on single-aspect control, ignoring the need for multi-aspect control. PPLM (Dathathri et al., 2020) guides generation by iteratively updating the LM's hidden activations. However, this decoding strategy is extremely computationally intensive, resulting in a slow generation speed (Gehman et al., 2020). Prefix-tuning (Li and Liang, 2021) proposes to optimize a prefix, which is a small continuous taskspecific vector, as a lightweight alternative to finetuning an NLG task, such as table-to-text generation or summarization. Inspired by Li and Liang (2021), we propose to use prefixes, a set of small continuous attribute-specific vectors, to steer NLG. Compared with using an attribute model or a generative discriminator (Dathathri et al., 2020;Krause et al., 2020), using learned prefixes to achieve controllability has the following benefits. First, it introduces fewer additional parameters (~0.2%-2% of GPT2 parameters in our experiments). Second, using prefixes keeps the inference speed comparable to that of the original GPT2 model.
In a general sense, prefix-tuning (Li and Liang, 2021) can be considered as controlling the genera-tion of language models. Prefix-tuning views each prefix as an independent control task thus trains each prefix separately (top in Figure 1). However, one aspect of controllability in NLG involves multiple attributes, which might have a relationship with each other. For example, the sentiment aspect usually has two attributes: positive and negative, which are in opposition to each other. We think that this opposite relationship can be helpful to improve the controllability of a prefix. Therefore, we propose a novel supervised method and a novel unsupervised one in our framework, which takes the relationship among prefixes into consideration and trains multiple prefixes simultaneously with novel training objectives, as illustrated in Figure 1.
Experimental results on the single-aspect control tasks (sentiment control, detoxification, and topic control) show that our proposed methods can guide generation towards the target attribute while keeping high linguistic quality, even when only several dozen labeled examples are available. In addition to single-aspect control, multi-aspect control can be achieved by combining the proposed supervised method with the unsupervised method in our framework. Experimental results on the sentiment and topic control show that the prefixes trained with our method can successfully control these two aspects simultaneously.
Our main contributions are as follows: • We propose a novel framework that utilizes prefixes with frozen LMs as a lightweight alternative for controllable GPT2 generation.
• We propose a supervised method and an unsupervised method with novel objectives for prefix training, where the relationship among prefixes are considered and multiple prefixes are trained simultaneously.
• This work provides a unified perspective for single-aspect control and multi-aspect control. Experimental results show that our methods can effectively guide generation in both single-aspect control and multi-aspect control.
2 Related Work Ficler and Goldberg (2017) control the stylistic aspects of the generated text with a conditioned RNN (Recurrent Neural Network) LM. Holtzman et al. (2018) compose a committee of discriminators to guide an RNN generator towards the generations with the desired linguistic quality. Hu et al. (2017) aim at controlling the sentiment and tense of the generated text by combining variational autoencoders (VAE) and attribute discriminators. More recently, with the advent of Transformers and large pretrained language models, such as GPT2, an extensive body of work has focused on controlling the generation of these Transformerbased models. Keskar et al. (2019) train a 1.63 billion-parameter conditional transformer LM from scratch with 55 attribute control codes to guide generation. However, this method is expensive and lacks flexibility since the control codes are fixed. Dathathri et al. (2020) address these limitations by developing a plug-and-play model which leverages an attribute discriminator to perturb the LM's hidden activations. However, updating gradients at the token level results in slow inference. Instead of updating the hidden activations, Krause et al. (2020); Yang and Klein (2021); Lin and Riedl (2021) introduce generative discriminators to re-weight the next token distributions on the fly during inference, thus improving the inference speed.
Our work is mostly related to Yu et al. (2021); Li and Liang (2021). Yu et al. (2021) use a pretrained LM followed by an attribute alignment function to encode the tokens of the target attributes and the resulting hidden states are used to control generation. Different from their work, we do not take the tokens of the target attributes as input. Instead, we directly train a set of parameters, which acts as the prepended hidden states of GPT2, to control generation. Avoiding using attribute tokens can circumvent the problems when it is difficult to describe the desired attribute with only one word. Besides, Yu et al. (2021) focus on attributes disentanglement, which is not a focus in our work, so our training methods are different. Prefix-tuning (Li and Liang, 2021) can, in a general sense, be viewed as controlling the generation of LMs, where the LM is controlled to depict a specific NLG task, while in this work, the LM is controlled to carry specific attributes in a generation. Besides, our proposed methods for prefix training are different from Li and Liang (2021), as stated in Section 1.

Method
Our method uses prefixes to guide GPT2 generation, where a prefix is a continuous attributespecific vector prepended to the activations of GPT2. Prefixes are free parameters denoted as H θ . Different from Li and Liang (2021), where each prefix is trained independently, we consider the relationship among attributes and train multiple prefixes simultaneously, so H θ is of dimension N × M × D, where N is the number of prefixes. In single-aspect control, N equals the number of attributes in the concerned aspect. M is the length of a prefix. D = 2 × L × E is the dimension of the activation in GPT2, where L is the number of transformer layers, E is the hidden size, and 2 indicates one key vector and one value vector. Following Li and Liang (2021), we reparametrize H θ [i, j, :] = W i H θ [i, j, :] by a smaller parameter (H θ ) composed with a large matrix (W i ). After the training finishes, only H θ needs to be saved for generation while W and H θ can be discarded. Since the GPT2 parameters are kept frozen during training, they do not need to be saved either. Figure 2 shows an example of the generation process under the control of a trained prefix. The prefixes can be trained in a supervised, semi-supervised, or unsupervised way. Since the semi-supervised method is a combination of the supervised and the unsupervised method, we introduce the supervised and the unsupervised method in this section. For clarity, we introduce these methods under the single-aspect control setting.

Supervised Method
Suppose the concerned aspect has the attribute set Y , each training example is a pair of (x, y) where x is the input text and y ∈ Y is the attribute label of x. Note that the attribute label also indicates the ground truth index of the prefix in H θ , so y also refers to the prefix index in the following description. As mentioned in Section 1, we introduce an additional discriminative loss to train multiple prefixes simultaneously. Therefore, the training loss L sup is a weighted sum of the language model loss L LM and the discriminative loss L d : The computation of log p(x t |x <t , y) is parameterized as log p θ,γ (x t |x <t , H θ [y, :, :]), where γ is the set of fixed GPT2 parameters, and θ represents learnable prefix parameters. log p(x|y) = t log p(x t |x <t , y), so the parameterization of . "The book" is the given prompt. "is good" is the generated completion.
Note that each prefix can be trained independently using L LM alone, which would be the same as prefix-tuning (Li and Liang, 2021). Intuitively, prefixes trained by L LM are infused with the information of what is encouraged to generate. However, we observe that in controllable NLG, it is helpful to also infuse a prefix with the information of what is discouraged to generate. Given a training example (x, y), the prefix H θ [y, :, :] should be optimized towards generating x, while the other prefixes should be discouraged to generate x. To achieve this goal, all the prefixes in H θ should be trained simultaneously. Therefore, the discriminative loss L d is introduced. As in equation 3, optimizing L d improves the attribute alignment p(y|x) by increasing p(x|y) and lowering p(x|ȳ),ȳ ∈ Y \{y} at the same time. We assume uniform prior, so p(y) and p(y ) can be canceled out in Equation 3. Figure 3 illustrates the training process with two prefixes.

Unsupervised Method
In the unsupervised setting, we assume the attribute set Y of the concerned aspect is known. The training example consists of input text x only. The attribute label y is no longer available and thus the index of the prefix associated with x is unknown. In other words, the index of the prefix corresponding to x is a latent variable z, whose posterior distribution follows a categorical distribution. Inspired by VQ-VAE (van den Oord et al., 2017), we consider the prefixes as discrete latent representations. We take the backbone model in the above supervised method as the decoder and introduce an encoder to parameterize the categorical distribution q(z|x). According to q(z|x), a prefix index z is selected and the prefix H θ [z, :, :] is then fed into the decoder  to reconstruct the input text x. Since the selection process of the prefixes is non-differentiable, we use Gumbel-Softmax (GS) relaxation (Jang et al., 2017;Maddison et al., 2017) following Sønderby et al. (2017); Ramesh et al. (2021). Formally, q(z|x) is computed as follows: where τ is the temperature of Gumbel-Softmax, and Enc is the encoder function. We use a pretrained GPT-2 model followed by a linear layer as the encoder. To train the prefixes, the loss function is a weighted sum of the three loss terms: where L LM is the language model loss. Similar as that in the supervised method, the computation of log p(x t |x <t , z) is parameterized as log p θ,γ (x t |x <t , H θ [z, :, :]). L KL is the Kullback-Leibler divergence, where we assume the prior p(z) to be uniform. Note that these two terms constitute the loss function of VAE. Optimizing these two loss terms improves the evidence lower bound of log p(x). Similar to the intuition behind L d in the supervised method, if the ground truth prefix for x is H θ [y, :, :], then the other prefixes should be discouraged to generate x. However, L d requires the ground truth label y for computation. Instead, we introduce an unsupervised contrastive loss L c .
where m is a pre-set margin andz is another latent variable indicating the index of the opposite prefix of x. q(z|x) is computed as follows: L c is aimed at increasing the attribute alignment by pushing p(z|x) away from p(z|x) by a margin. The computation of p(z|x) is as follows: We assume uniform prior, so p(z) and p(z ) can be canceled out. Similar as the parameterization of log p(x|y) in the supervised method, the parameterization of log p(x|z) is the sum of log p θ,γ (x t |x <t , H θ [z, :, :]) over t. The training process is illustrated in Figure 4.

Experiments
We experiment with three tasks: sentiment control, detoxification, and topic control. We compare our method to GPT2, PPLM, and GeDi. We focus on English text in all the experiments and we experiment with GPT2-medium (345M parameters) for all the methods. We use the original implementation of PPLM and GeDi released by Dathathri et al. (2020) and Krause et al. (2020), and the hyperparameters are set to the reported value in the original paper. The detailed hyperparameters in each task are listed in appendix A. For the GPT2 model, we do experiments under two settings. First, the GPT2 model generates completions of each prompt in the evaluation dataset, which is denoted as GPT2medium. Second, GPT2-medium + prompt engineering prepends a guiding sentence to each testing prompt and then generates completions of each augmented prompt. We evaluate the linguistic quality and attribute alignment of the generation. The linguistic quality is evaluated using the perplexity calculated by GPT2-large (774M parameters).
To evaluate the robustness of our supervised method with the size of the training dataset, we experiment with the following three different settings: 1) using the complete training dataset; 2) using 1,000 examples per attribute for training; 3) using 24 examples per attribute for training. We evaluate our unsupervised method on the sentiment control task and the detoxification task, which are binary tasks. Note that different from the supervised method, our unsupervised method does not use any attribute labels, so the order of the attributes in the trained prefixes is undetermined. After the prefixes finish training using the unsupervised method, we manually check the order of the attributes.

Tasks
Sentiment Control Same as GeDi, we use IMDb movie reviews (Maas et al., 2011) to train our model. The number of prefixes is 2. Note that GeDi only uses 11.25k examples from the dataset for training. To be a fair comparison, we randomly sample 11.25k examples from the dataset to train our model. To evaluate the sentiment alignment of the generated text, we finetune a RoBERTa (Liu et al., 2019) classifier using the Yelp Review dataset (Zhang et al., 2015). The prompts used for evaluation are the same as those in the PPLM experiment (Dathathri et al., 2020). For each of the 15 prompts, 45 completions are generated. In the GPT2-medium + prompt engineering setting, we prepend each prompt with the guiding sentence "This is a negative review:" for negative sentiment control, and similarly, we prepend each prompt with "This is a positive review:" for positive sentiment control.
Detoxification We use Jigsaw Toxic Comment Classification Challenge Dataset 1 to train our model. The number of prefixes is 2. Google Perspective API 2 is used for toxicity evaluation. The testing prompts are collected from RealTox-icityPrompts (Gehman et al., 2020). We use the prompts categorized as "challenging" in the dataset. We further filter out the prompts with toxicity larger than 0.5, scored by Perspective. The resulted evaluation dataset consists of 203 prompts. For each of these prompts, 20 completions are generated. In the GPT2-medium + prompt engineering setting, we prepend each prompt with the guiding sentence "This is a non-toxic comment:".
Topic Control We experiment with the AGNews dataset and DBPedia dataset (Zhang et al., 2015). The number of prefixes is 4 and 14, respectively. The prompts used for evaluation are the same as those in the PPLM experiment (Dathathri et al., 2020). For each of the 20 prompts, 45 completions are generated. Same as that in GeDi, we split each of the original training datasets in half. One half is used to train prefixes, while the other half is used to train a RoBERTa topic classifier for topic relevance evaluation. In the GPT2-medium + prompt engineering setting, the guiding sentence follows the template "The following is about [TOPIC]". We do not compare with PPLM in the topic control task since PPLM uses a bag-of-words attribute model to do topic control, where the 7 predefined topics are different from the topics in the AGNews dataset or the DBPedia dataset.
All the experiments are conducted on NVIDIA Tesla V100 GPUs. The detailed hyper-parameters for each experiment are listed in appendix A.

Results
In the unsupervised setting, GPT2-medium + prompt engineering shows controllability on sentiment control (Table 1) and topic control (Table 3). However, this method does not work on the detoxification task (Table 2). Our unsupervised method significantly lowers the toxicity on the detoxification task and the ablation study shows that the contrastive loss L c is crucial. On the sentiment control task, our unsupervised method does not achieve good attribute alignment when the target sentiment is negative, but it performs well when the target sentiment is positive. One possible reason is that compared with the differences between toxic and normal sentences, the difference between positive sentiment and negative sentiment is more subtle, so it is more challenging for the GPT2 encoder in our unsupervised model to accurately separate the unlabeled data into two sentiments. As a result, the encoder's implicit criterion to categorize the input text may not be exactly the sentiment, which is also the reason that after removing the contrastive loss L c in the unsupervised loss function, the attribute relevance on the negative sentiment is higher while that on the positive sentiment is lower.
In the supervised setting with full data, our supervised method consistently achieves better controllability than PPLM while maintaining the linguistic quality of the generations (Table 1, 2). Although GeDi achieves a high attribute alignment score on the three tasks, it severely sacrifices the linguistic quality, as indicated by the high perplexity. In the few-shot setting, where the number of labeled training examples is reduced to 1000 or 24 examples per attribute, our supervised method can still maintain good controllability on the three tasks, showing the robustness of our method to the size of the training data.
Ablation study shows the importance of the discriminative loss L d in our supervised method. As mentioned in section 3, training without L d is equivalent to prefix-tuning. Comparing the results of Ours−L d and GPT2-medium show that directly using prefix-tuning can achieve controllability on the sentiment or the topic. However, it is less effective on detoxification. The reason is that different from topic control or sentiment control, detoxification requires the model to avoid generating some words or phrases according to the context, which can not be achieved by prefix-tuning. L d fills this gap by increasing p(x|y) and lowering p(x|ȳ) at the same time. Therefore, incorporating L d is of critical importance to the detoxification task. In the DBPedia topic control task, adding L d also achieves a large improvement on attribute alignment. The number of attributes in this task is much larger than that in the other tasks, so incorporating L d can effectively push the prefixes to capture the unique features of each topic.
We compare the average inference speed of our methods with the baselines ( Table 5). The inference speed of PPLM is several dozen times slower than that of the original GPT2 model. GeDi's inference speed is much faster than that of PPLM. The inference speed of our method is the closest to that of the original GPT2.

Human Evaluation
Besides automatic evaluation, we also conduct human evaluations on Amazon Mechanical Turk to compare the performance of the baselines and our methods. In each task, workers are presented with a prompt along with the completions generated by different methods. Workers are instructed to answer two questions:"Which one has the best linguistic quality?" and "The target attribute is [ATT]. Which one aligns best with the target attribute?". [ATT] is the control attribute used when generating the completions. In order to evaluate the linguistic quality and the attribute alignment separately, the workers are instructed not to consider the control aspect or the factual errors when answering the first question and not to consider the linguistic quality when answering the second question. The user interface provided to the workers is shown in the appendix ( Figure 5). We conduct human evaluations on the results of the sentiment control experiment and those of the AGNews topic control experiment separately. 100 tasks are randomly sampled from the results of each control experiment. Each task is assigned to 3 different Mechanical Turk workers and the annotations are aggregated by majority voting. To ensure data quality, we restrict the workers to be in Canada or United States with     a HIT approval rate higher than 95%. In total, 81 workers participated in the human evaluation. For the sentiment control task, we compare the results of GPT2-medium + prompt engineering, PPLM, GeDi, and our supervised method (with full training dataset). For the AGNews topic control task, PPLM is not evaluated as explained above. The results are shown in Table 4. The inter-annotator agreement on the sentiment task and the AGNews task is 0.39 and 0.30 in Fleiss' κ, respectively. Appendix B lists other details of the human evaluation.
In the sentiment control task, the result of human evaluation on linguistic quality is generally consistent with the result of automatic evaluation. However, different from the result of the automatic evaluation, annotators are more inclined to select Ours and GPT2 + prompt engineering when evaluating attribute alignment. Although the annotators are instructed not to consider linguistic quality when evaluating sentiment alignment, they tend to select the one with better linguistic quality when multiple completions exhibits equally good attribute alignment. In the AGNews topic control task, the result of human evaluation on attribute alignment is generally consistent with the result of automatic evaluation. However, in more than half of the linguistic quality questions, the annotators select Ours, although GPT2-medium + prompt engineering achieves lower perplexity than Ours. On inspection, we find that GPT2-medium + prompt  Table 6: Experimental results of the multi-aspect control task. "PPL.": perplexity scores. "Senti. Rel.": sentiment relevance. "Topic Rel.": topic relevance. "−L d / −L enc ": ablating loss terms as described in Eq. 3 and Eq. 12.
engineering in this task exhibits a more severe repetition problem compared to that in the sentiment control task. This inconsistency shows the limitation of using automatic evaluations, as alluded to in Welbl et al. (2021). Both human evaluation and automatic evaluation show that the linguistic quality of GeDi is inferior to that of the other methods. One possible reason is the length of the prompt. In the original experiment in Krause et al. (2020), each prompt is at least 150 characters for sentiment control evaluation and at least 30 characters for topic control evaluation. However, we use the prompts as in Dathathri et al. (2020), where the average prompt length is 11.8 characters for sentiment control evaluation and 14.5 characters for topic control evaluation. The generated examples are shown in the appendix (Table 7).

Multi-Aspect Control
Our method can also be applied to multi-aspect control. Directly applying our supervised method to multi-aspect control requires training examples with multi-aspect labels. However, such datasets are usually not readily available since most of the datasets are labeled for a single task. Although multi-aspect labeled examples are limited, we have training examples with single-aspect labels from multiple aspects, which can be utilized to achieve multi-aspect control. One method is to train a set of prefixes for each aspect separately using our supervised method and then concatenate the prefixes from different aspects for generation. This method is denoted as Ours (concatenation) in the result table. Another method is to train the prefixes of multiple aspects simultaneously by considering each single-aspect labeled example as partially labeled. We use a semi-supervised method for training, which is a combination of our supervised method and unsupervised method in Section 3. The model structure is the same as in the unsupervised method ( Figure 4). The loss function is as follows: where the latent variable z is the concatenation of the latent variable of each aspect, including both the supervised aspects and the unsupervised ones z = [z sup ; z uns ]. L enc is used to train the encoder. It is introduced because the partially labeled examples imply the ground truth indexes of the prefixes in the labeled aspect, providing supervision for both the prefix and the encoder. σ is the softmax function. We experiment with controlling the following two aspects simultaneously: sentiment and topic. We use the binary sentiment dataset from Amazon review (Zhang et al., 2015) and the DBPedia topic dataset. The prompts used for evaluation are the same as those in the topic control experiment. For each of the 20 prompts, 45 completions are generated. In the GPT2-medium + prompt engineering setting, the guiding sentence follows the template "This is a [SENTIMENT] review on [TOPIC]:". In Ours (concatenation), the sentiment prefixes and the topic prefixes are trained separately using our supervised method and then concatenated as multiaspect prefixes. In Ours (semi-supervised), we reuse the prefixes trained in the single-aspect control tasks to initialize H θ . All the experiments are conducted on NVIDIA Tesla V100 GPUs. The hyper-parameters are listed in appendix A.
Experimental results on multi-aspect control (Table 6) show that simply concatenating the prefixes trained for single-aspect control can effectively control the sentiment and topic simultaneously, and our experiments show that the order of the prefixes does not impact the result. On the other hand, training using the combination of our supervised and unsupervised methods can further improve the attribute alignment without sacrificing too much linguistic quality. Same as the observations stated in Section 4.1.2, removing the discriminative loss L d will significantly degrade the attribute relevance, especially the topic relevance. Removing the encoder loss L enc may achieve higher overall attribute relevance at the cost of linguistic quality, indicated by a higher perplexity. We present the generated examples in the appendix (Table 7).

Conclusion
We propose a novel framework for controllable GPT2 generation with frozen LMs, which utilizes contrastive prefixes to guide generation. Experimental results show that our framework can not only successfully guide generation from a single aspect but also achieve promising results on multiaspect control tasks. Besides the control tasks we experimented with, our proposed framework can be freely applied to other desired attributes.

Ethical Considerations
With our controlling methods, it is not one hundred percent guaranteed that the generations will have the desired attributes, but the probability for the generations to exhibit the desired attributes will increase. When applied to detoxification, although the probability of toxicity degeneration will decrease, the controlled language model may still produce unsafe text. We would like to clarify that the offensive language generated by the language model controlled with our methods does not represent any opinion of the authors.
Besides, our proposed methods control the highlevel attributes of the generation, such as toxicity, topic, or sentiment, but there is no guarantee of factual accuracy for the generation, which is a wellknown problem in NLG models. Our controlling methods may not be used for factual accuracy controlling. While reducing hallucination is not the focus of this work, knowledge-grounded generation techniques can be used to alleviate this problem.

Appendix A Hyperparameters
For PPLM and GeDi, we use the hyperparameters reported in their original work (Dathathri et al., 2020;Krause et al., 2020). Note that GeDi has multiple versions of submission available online and we refer to the latest one on OpenReivew.
Our methods are implemented using the Hugging face Transformers package. In all the experiments with our methods, the random seed is fixed to 42, and the optimizer is AdamW with a learning rate of 2e-5. D = 24 × 2 × 1024, where 24 is the number of hidden layers in GPT2-medium, 1024 is the size of hidden states in GPT2-medium, and 2 represent one key and one value. In the sentiment control task and the topic control tasks, the maximum generation length is set to 50 during evaluation while in the detoxification task the maximum generation length is set to 20. Unless stated otherwise, the prefix length M = 10.

Sentiment Control
In the Ours (unsupervised) setting, the training batch size is 8. ω 1 = 0.8, ω 3 = 2.0. The weight of the KL loss term ω 2 anneals from 0.001 to 0.1 during training while the temperature τ reduces from 1.0 to 0.5. The number of training epochs is 60. During training, we randomly mask the input tokens when computing the next token probabilities so as to force the prefix to preserve the key information of the input text. The mask rate is 0.5.
Detoxification In the Ours (unsupervised) setting, the training batch size is 8. ω 1 = 0.8, ω 3 = 2.0. The weight of the KL loss term ω 2 anneals from 0.001 to 0.1 during training while the temperature τ reduces from 1.0 to 0.5. The number of training epochs is 4. Same as in the sentiment control task, the mask rate is 0.5.
Multi-Aspect Control In the Ours (concatenation) setting, the sentiment prefix with length M = 10 and the topic prefix with length M = 10 are concatenated, so the resultant multi-aspect prefix has a length M = 20.

B Human Evaluation
The payment for each approved annotation is set to $0.6. The average completion time is 3 minutes 45 seconds per HIT (prorated to an hourly wage of $9.6). In fact, Bynum allegedly responded to Eley on Facebook: "A. Someone should tell me who my NBA team ____. Nice kid (10 pounds . . .

Ours
In fact, Bynum allegedly responded to Eley on Facebook: "Thanks for your suggestion. I'm going to start by using the phrase 'if you have something to . . .

GPT2+
The book contains some serious errors in spelling and grammar and there is little in the way of a coherent story. This review is therefore the negative review, for lack of a better term. It is a story about "The Dark Knight." It follows one. . .

PPLM
The book, The Big Empty, is the first of a series of three, to be released by the American Public Health Association in the fall. The report's goal is to "The new report ( ). A: It is . . .

Ours
The book I had read before I heard about the film and it was a huge disappointment. I am a huge fan of The Hobbit, so I was looking forward to watching this and this is not going to hold me back from enjoying the film. I am very . . .

Positive GPT2+
The potato chip is fantastic! I love it! My friends really liked it and the food is good as well. Went here for a quick lunch. We sat in the outdoor patio area next to a few of our friends. My sister and I . . .

PPLM
The potato, which is a staple of many people's diet, has been linked to the development of type 1 diabetes and heart problems. A group of British scientists has found the first evidence of an association between high-fiber eating and type 1 diabetes . . .

GeDi
The potato grower will adjust perfectly to your farm and farm animals. We offer high-quality fresh, all-natural produce that is grown outdoors. About Us Bib Flowermachine provides composting, hydroponics, water, fertile . . .

Ours
The potato chip is the classic American family meal. And while it's been around for decades, it still is the perfect dinner option for any family, whether you're a member of the household or not. But it is also an incredibly versatile meal. For example . . .

Sports GPT2+
Views on football. Football is the subject of a number of sports-related articles by the public domain, so this will not be repeated here. This article may not contain legal advice or should be considered legal advice in relation to your own legal . . .

GeDi
Views on Beckham MVP derby got into the mix Sunday weekend, as ESPN's Adam Schefter produced a great (& entire list we'll get to below) breakdown of all things Beckham. Basically, we popped the top of the pitcher (who may win to clear . . .

Ours
Views on this season are split. Some, like former Miami Dolphins quarterback Peyton Manning, believe the Patriots are a Super Bowl contender. Others, like former New England Patriots head coach Bill Belichick, say the Pats are a perennial loser.

World GPT2+
The central theme of the novel is the search for purpose and for meaning. However, the novel isn't just about these goals and meanings. It is also about life and death, personal relationships, and the way that life and death are often intertwined in the lives of . . .

GeDi
The central theme campaigner Najim Hasina uses is Kashmir peace, and with the Privy Council review being conducted towards the beginning of January, critical comments were placed on Delhi's artificiality andness in defence of watchdog. As has been stated, Rajesh G. . .

Ours
The central theme of the next few weeks will be the battle against terrorism, with Iraq at the top of the list.

{Negative, Company} Ours
The issue focused on accessories and software was one of the main reasons why Apple Inc. dropped the product line. The company did not realize that its product line would be the downfall of the company.

{Positive, Athlete} Ours
The issue focused on his game as a center back. He is an excellent athlete who has a strong work ethic. He is a good defensive midfielder who can make plays and get his team points. He plays a natural position as a right midfielder. Table 7: Examples of the generation. In the first column are control codes. "Negative": Negative Sentiment. "Positive": Positive Sentiment. The second column lists the methods. "GPT2+": GPT2-medium + prompt engineering. The given prompts are in bold. The guiding sentences of GPT2+ are omitted for brevity.


Static Seeding and Clustering of LSTM Embeddings to Learn from Loosely Time-Decoupled Events

Humans learn from the occurrence of events in a different place and time to predict similar trajectories of events. We define Loosely Decoupled Timeseries (LDT) phenomena as two or more events that could happen in different places and across different timelines but share similarities in the nature of the event and the properties of the location. In this work we improve on the use of Recurring Neural Networks (RNN), in particular Long Short-Term Memory (LSTM) networks, to enable AI solutions that generate better timeseries predictions for LDT. We use similarity measures between timeseries based on the trends and introduce embeddings representing those trends. The embeddings represent properties of the event which, coupled with the LSTM structure, can be clustered to identify similar temporally unaligned events. In this paper, we explore methods of seeding a multivariate LSTM from time-invariant data related to the geophysical and demographic phenomena being modeled by the LSTM. We apply these methods on the timeseries data derived from the COVID-19 detected infection and death cases. We use publicly available socio-economic data to seed the LSTM models, creating embeddings, to determine whether such seeding improves case predictions. The embeddings produced by these LSTMs are clustered to identify best-matching candidates for forecasting an evolving timeseries. Applying this method, we show an improvement in 10-day moving average predictions of disease propagation at the US County level.

Introduction
In many real-life applications a dataset can consist of instances that have features that are both static and dynamic. For example, consider patient health data such as age and gender which are relatively static features compared to high-frequency dynamic heartbeat data as collected from electrode sensors connected to the patient. Sequence classification models such as Recurring Neural Networks (RNN) [1], Long Short-Term Memory (LSTM) [2], or Hidden Markov Models (HMM) [3] can be used to model the dynamic time-variant features of the event but are not suitable to address the static features [4]. In the patient health data example, an LSTM structure could be used to model the heartbeat timeseries data across multiple patients; however, it is not suitable for processing static and dynamic data simultaneously [4]. Ensemble methods such as those provided by Dietterich in [5] and Bagnall et al in [6] provide another way to address the issue: predictions made by temporal models such as Dynamic Time Warping (DTW) [7], Rotation Forests [8] and COTE [9] on dynamic data are combined with the predictions of a discriminative classifier on static data by performing distance measures as presented in [10]. Tzirakis et al. in [11] develop a methodology accomplishing simultaneously: (1) hierarchical clustering of raw dynamic data, (2) learning of deep end-to-end representations, and (3) temporal segments boundaries identification. They compute similarity between timeseries segments using an extension of DTW. A global loss function is used to optimize all three objectives. While this method results in representations learned from the clusters detected in this process, it does not intrinsically tie these representations to each timeseries.
In this work we introduce the concept of a Loosely Decoupled Timeseries (LDT) phenomenon and improve LSTM networks to enable Artificial Intelligence (AI) solutions to offer better timeseries predictions informed by static features or features that vary at a different frequency than the main event being modeled. A key feature of LSTMs is that they maintain a dual-purpose internal state (memory) that can aid in the learning and forecasting process [2]. We use similarity measures between timeseries based on the trends and introduce embeddings [12] representing those trends. The embeddings are constructed from features that are either static or changing at a different frequency than the timeseries of the main event being modeled. We apply this method to improve predictions of COVID-19 detected infections and death cases. We treat COVID-19 detected infections and death cases [13] as the main time-variant dynamic event and use socio-economic data at the US-County level as static features to inform predictions among counties of similar socio-economic structure but differing time lag in COVID-19 disease propagation among their population.
This work develops ideas from such disparate sources as COVID-19 forecasting, signature verification, and useful-life estimation from sensor data. Li et. al. [14] demonstrate improved signature verification by casting signatures as static representations of dynamic pseudo processes, using the dynamic process to generate an attention mechanism for the static representation. This has obvious ties to the COVID-19 pandemic as a dynamic process. We choose geospatial and demographic characteristics of communities as our static representation for two reasons: latent handling of mobility-impacted disease transmission and data augmentation. COVID-19 spread is heavily impacted by population mobility, which Panagopoulos et. al. [15] attempt to capture directly using graph neural networks, with vertices as cities and edges as movement between cities, while Xiao et. al. [16] use intra-city mobility patterns to train an adversarial encoder framework to predict next-at-risk communities. Both groups suffer from a lack of training data, which Panagopoulos et. al. [15] attempt to alleviate using transfer-learning between graphs generated from different countries. Wang et. al. [17] attempt to use augmented data for training, using an ABM to generate synthetic data based on an SEIR epidemic model. However, in our experience (unpublished), the SEIR model is not a great description of COVID-19 and the efficacy of epidemic models is highly dependent on their internal social-interaction model and estimated parameter values. Therefore, we propose a clustering approach based on geospatial and demographic attributes to augment our training data with other US counties that are similar in latent space and known-pandemic trajectory (matching COVID-19 spread based on where each county is in their respective pandemic trajectory) Our novel contributions are as follows: 1) Definition and demonstration of Loosely-Decoupled Timeseries using static and trajectorymatched dynamic features for improved spatio-temporal prediction. 2) Computationally simple (K-means or K-medoids) latent-space clustering of static geospatial and demographic features accounting for mobility patterns and socioeconomic behaviors. 3) Built-in data augmentation through clustering of data with trajectory-matched pandemic behavior, effectively increasing the training data by reducing the number of prediction classes, thus obviating the need for potentially problematic synthetic data augmentation.

Loosely Decoupled Timeseries
We define a Loosely Decoupled Timeseries (LDT) phenomenon as the relationship between two or more events that could happen at the same place or at different places but across different timelines sharing similarities in the nature of the event and the properties of the location. We contrast LDT with event-coupled timeseries [18] and tightly coupled timeseries [19]. Eventcoupled timeseries consist of phenomena starting at the same time, whereas LDT allow for a lag between the event onsets. Tightly coupled timeseries start at the same time and are coupled in time throughout the event, such as the case of audio or speech and the corresponding video of lip gestures; whereas LDT events can happen at varying time frequencies such as the loose coupling of birth rates measured annually, and unemployment measured monthly. Other examples of LDT include the timeseries associated with a news cycle (hourly) in relation to the timeseries associated with the spread of violence (daily or weekly victim counts) or with the spread of disease (daily infection or death counts) being covered by the news cycle. LDT can also span two or more events occurring at different locations.
We represent LDT as: where ( , ) is a time sequenced event [0, …., T] conditioned on the environment , while provide for a varied frequency time series and provide for a time lag between the two events.

Our Approach
A key feature of LSTMs is the maintenance of dual-purpose internal state (memory) that aids in learning and forecasting. This ameliorates the exploding or vanishing gradients problem experienced by RNNs [2], at the expense of slightly higher memory and computational complexity. This internal state convolves more distant and more recent information input; acting as a compression or embedding mechanism for the timeseries.
We use this internal memory state as an embedded representation of the timeseries, after appropriately training an LSTM model on subsequences of the timeseries.
We represent the LSTM model trained on timeseries ( , ), as [2]: We also have Which produces a trained LSTM model represented as: The summation indices u, based on [2], can stand for input units, gate units, memory cells, or even conventional hidden units. These different types of units convey useful information about the current state of the LSTM. These may also be recurrent self-connections like . At time t, 's output ( ) is computed as: The "internal state" or embedding representation ( ) is: The differentiable function g "squashes" ; the differentiable function h scales memory cell outputs computed from the internal state . Figure 1. Architecture of memory cell (the box), the j-th memory cell block, and its gate units , and . The self-recurrent connection (with weight 1.0) indicates feedback with a delay of 1 time step. The index k ranges over hidden units u. [2] We represent the embedding representation of ( , ) as: We enforce the same cell size, u, for all LSTMs trained to ensure the equidimensional representation of the internal states (3) to facilitate comparison independent of their length.
Our general focus is on the using the equidimensional embedding representations of (3) along with the time invariant LSTM-based prediction: given an evolving timeseries ( , ), search for and select longer/more evolved timeseries � * + , � that can be associated to produce a better prediction for the next time step(s) of (2) for ( , ).
To locate an associated timeseries, we expand the internal state (3) of an LSTM to include the static properties of : We combine (3) and (4) to form: We call (5) the embedding representation of the phenomena at . We then adopt a clustering method to cluster the embedding representations of several phenomena to identify the LDT tuples. In this paper we explore two clustering methods. K-means: With k centroid points and minimizes the sum over each cluster of the sum of the square of the distance between the point and its centroid -the centroid is not necessarily a point from the data set.
And K-medoids: which is similar to K-means but uses medoids chosen from points in the data set.
Once the LDT clustered tuples are identified, we then seed the main LSTM model based on this static data prior to each training episode, as well as prior to making a forecast. We call this seeding the local static embedding. This seeding takes the form of a dense embedding layer from the static data vector to the (initialization of the) hidden layer in the LSTM model. This dense embedding is trained by back propagation simultaneously with the LSTM training regimen. The embedding layer is reused at estimation time to reseed the hidden layer of the LSTM at the initial step.
Forecasting COVID-19 COVID-19 is a disease (SARS-CoV-2) which spread rapidly around the world. Various metrics have been put forth to monitor its spread and evolution. While there is significant variation in the data collected, two main metrics of interest are generally available. One is the number of people "infected" (those that tested positive on a standardized testing platform), and the other is the number of people that died due to a recorded affiliation with the virus.
Both metrics are subject to reporting discrepancies. While some jurisdictions only report PCRpositive tests, the gold standard, others report results from less-reliable methods or even antibody tests (antibody tests rarely show positive until the end of an infection and then show positive for months or more after viral clearance). Additionally, death counts often represent both deaths due directly to COVID-19 complications and deaths due to unrelated causes but with positive test result (from preventative screening).
Simultaneously, there is a need from the public (and public health officials) to predict the evolution of these metrics days, potentially weeks, ahead for resource allocation and policy formulation. This prompted numerous efforts to build and apply predictive models [20] [21].
The standard models used in public health are derivatives of the SIR model [22]. These models are based on the "evolution" of an individual through the stages of a disease, from Susceptible "S" (has potential to get infected) to Infected "I" (virus is present) to Recovered "R" (disease ran its course). Variations considering asymptomatic, or unreported infections, as well as death as an outcome, are also used. While well understood, both from a theoretical and a practical (estimation) perspective, these models are necessarily limited by the assumption of compartmentalization (disease evolving in isolation). In reality, human movement patterns lead to diffusion of infection across boundaries. Solving coupled SIR compartmental models subject to constraints and diffusion becomes significantly more difficult and potentially intractable without deeper (longer history) samples.
Another aspect of compartmental models is their focus on inference rather than prediction. The primary focus of SIR-type models is on estimating disease characteristics (e.g., transmission rate) rather than prediction. Additionally, predictions themselves are only helpful up to a point. Just knowing something will happen is of limited usefulness, in the absence of scenario-based alternatives. From a prescriptive perspective (predictive + actionable) around the COVID-19 disease, it would be helpful to build upon similarities and "local tests" between US counties. By local tests we mean different restrictions and implementation or adherence to these restrictions, and their impact on the disease trajectory.
Working with researchers at the Air Force Research Lab / Autonomous Capabilities Team (ACT3), we apply the above-described methods of LDT-enhanced LSTM modeling on COVID-19 detected infection and death cases [13]. The time interval under study covered March through October 2020. During this time, the global events were such that there was a limited supply of COVID-19 testing resources, a hesitation in applying and adopting Non-Pharmaceutical Interventions (NPIs), and several US counties adopting lockdown procedures. During that same period, there was a single dominantCOVID-19 variant. From this perspective, the disease evolution was not influenced by mixed variants, as became more common in subsequent months. We limit the geographic span and resolution of our study to US County-level. County level data is more likely to contain a systematic definition of a COVID-19 case, following from locally consistent testing approaches and capabilities, consistent NPI measures, and consistent lockdown directives (if any were applied). More importantly, counties in the same state may experience a lag in disease spread among each other. Therefore, known data about the spread of the disease in counties already affected can inform the future state of counties starting to experience their first cases. In a similar fashion, the effect of NPIs (e.g., mask / lock-down mandates), observed in the infection and death timeseries of some counties, can inform the expected effect from similar NPIs in counties which are considering such measures. The value proposition of modeling county level data is thus significant from an operational and NPI implementation point of view.

Description of Data
The 2010 Census demographic datasets [23] for each of the US Counties were used in the analysis. Table 1 lists the data fields that were used in this study. The economic data for each county was sourced from the USDA Economic Research Service [24] and Table 2 lists the data fields that were used in this analysis. Unemployment rate, 2019 Unemployment_rate_2020 Unemployment rate, 2020 In total, 385 socio-economic features from 3142 US Counties were used in the analysis. These constituted the static feature set and were assumed to remain constant during the analysis timeframe.
The COVID-19 infection metrics are aggregated by various entities. Johns Hopkins University [25] is an early and continuing resource for this data. However, they only collate what is reported by local health authorities, which are subject to local delays and constraints in identifying and reporting the disease spread.
For example, it has been observed that reported counts present a periodic dip around weekends. This is simply due to the limitations on scheduled activity for the labs running these tests. Correspondingly, there is a "bump" in counts at the beginning of the week, usually on Mondays.
The COVID-19 datasets from [13] [25] were used to analyze the daily number of infections and deaths. Table 3 lists the data fields that were used in the analysis. The number of cumulative infections and deaths was normalized against county population data. Raw data from [13] was used as is, with the exception of days in which a drop of cumulative infections or cumulative deaths were reported; during those days, the last reported value before the drop was used for all subsequent days until the cumulative values reached that level again.

LSTM Training Regimen
Focusing on prediction, rather than inference, potentially increases the utility of models from other domains. Time series analysis is one such domain, however, the structural constraints on those models are not easily aligned with a disease evolution expectation. An ideal model would "remember" trends and changes over varying time horizons (e.g., the convexity of the infected cases trend changed N days ago, where N could vary with the region under consideration). LSTM models have an established history in natural language processing, where learning the relationships between potentially distanced words help predict the next word. This is predicated on an underlying structure of the language from which samples are drawn (e.g., English), with long term memory keeping track of words earlier in the input. It is this long-term memory we had in mind when testing LSTM as a solution for predicting COVID-19 "trajectories". Given sufficient data, the model should learn to distinguish accelerating spread regions of the timeline from the more linear or saturated growth regions.
In our setup, the LSTM layer is followed by a dense layer with output dependent on the variables predicted. These variables are infected and dead counts, normalized by the population of the county.
LSTM training modules from PyTorch [26] were used and Ray Tune [27] was used to perform hyper-parameter tuning using grid-search. LSTM models were constructed to allow for grid-search across 64, 128, 256, and 512 hidden memory cells and across 1, 2, or 3 network layers. Several loss functions were tested for training: Mean Square Error (MSE), Relative Mean Square Error (RMSE), indexed or scaled versions to account for changing variability in the inputs across time, and versions penalizing for non-monotonic output. Input tensors covering a period between 7 and 30 days were used. Output was compared to actual values on 1, 3 and 5 day sliding intervals (days offset). The models with the top accuracy in prediction (lowest loss vs desired output) were successively retained by Ray Tune within the allocated time/computing resources; these models learned the most accurate representation for that county and point in time. Overall, more than 600 models were trained to extract county level embeddings over time.
For the purpose of consolidating our results, we consider two loss functions: • MSE (abs): the absolute mean square error between the output and expected values • RMSE (rel): the relative difference between output and expected values, with a large penalty imposed for producing non-monotonic sequences. A small quantity (10 −8 ) was added to the denominator to avoid dividing by 0.
The expectation was that RMSE based models will more closely match the disease trends, especially in the earlier stages when their population-normalized values are very small.
The following setup was used for all experiments: • data for all counties covers the interval from first recorded case (for each US county) through 09/18/2020 • there is always a "buffer" of (the last) 30 days which are not "seen" by the trained models (test_days = 30); this means 09/19 through 10/18 is reserved for testing / evaluation of the model • the models are trained for a certain time/computation "budget" using Ray Tune's ASHA Scheduler • individual counties' training epochs consisted of one pass through all (chunked) historical data • mini-batch training was used for all models (3 batches for individual counties models)  The US County embeddings were clustered using the two clustering methods in (6) and (7) with k = 3 clusters at 30, 60 and 90 day points in time. Clusters identified counties with similarity based on a shorter history, then matched other counties with a cluster, then analyzed the evolution of the groups' timeline to inform the new county's evolution.

Analysis of Results
The results presented in this section focus on analyzing 17 US Counties based in the state of Ohio with over 600 embeddings based on various points in time of the trained LSTM models for every county. The following notation is used in this section to represent the data being analyzed: • Data is a vector of observed values over time and can be with or without socio-economic data • Clustering method is either (6) or (7) with (6) represented with the label "k-means" and (7) as "k-medoids" in plots.
• Plot are taken at a Point-in-Time (PIT) reflected in the label of a plot and for a predefined set of clusters identified as cl:n where n is the number of clusters in the plot. • Clustering involves all hidden layer states ("all") or only the last one ("last") Figure 4 provides an example of k-means clustering into 3 clusters of 17 trained LSTM models with 60 days of training data using actual COVID-19 infection counts, relative to the total county population. Figure 4. Example plot for 17 counties in OH clustered using k-means and the actual COVID-19 infections out of total population count. Socio-economic data was not factored in the clustering. Clustering was done using the hidden neural network layers (state) of an LSTM model trained with 60 days of data.
We analyze the alignment (concordance) of the two clustering methods (6) and (7) through two metrics: 1. Accuracy based on a confusion matrix a. Calculated as the percent of the diagonal values present in overall confusion matrix b. Dependent on the cluster order (based on a specific permutation of clusters) c. Has values ranging from 0 to 1, with 1 being more accurate d. Represented as "Acc" in plots 2. Adjusted Rand Index a. Takes into account the random chance "alignment" of clustering methods b. Independent of cluster order c. Has values ranging from -1 to 1, with 1 being more accurate.
d. Represented as "ARI" in plots Figure 5 shows an example of Acc and ARI. Figure 5. Example of Acc and ARI metrics derived from a confusion matrix of a clustering method, clustering data across 3 clusters.
The template representation depicted in Figure 6 shall be used to provide the analysis of the results.     From Figure 7, Figure 8, Figure 9, and Figure 10 it can be concluded that including the socioeconomic factors increases the consistency (overlap) between actual, observation-based clustering, and embedding-based clustering. It is also noticeable, that K-means (6) has an advantage over Kmedoids and that using the last hidden layer of the LSTM as the embedding vector yields better results than using all the layers. The optimum clustering approach for COVID-19 data is to include the socio-economic data and use the last layer of the LSTM as the embedding in a k-means clustering algorithm.
We then proceeded to analyze the change in clustering over time. We define a "cluster stability" metric as follows (using Figure 11 as the example): Figure 11. Using K-means and 3 clusters for the 17 counties in OH with no socio-economic data, this plot shows how the 17 counties changed cluster assignment between PIT=60 and PIT=90 • Cluster 0 (green dots) had initially 10 members o Of these, 3 shifted to cluster 1 and 3 shifted to cluster 2 • Cluster 1 (orange dots) had initially 4 members o Of these, 2 shifted to cluster 0 and 1 shifted to cluster 2 • Cluster 2 (blue dots) had initially 3 members o Of these, 1 shifted to cluster 0 and 1 shifted to cluster 1 • The overall cluster stability metric is defined as the maximal accuracy (for the optimal cluster reordering, yielding the maximum diagonal) o In this case: (4+1+1)/17 = 35.3% Table 4. Cluster stability over time Table 4 shows the cluster stability calculations (last two columns) across two types of embeddings (rows): without socio-economic data and with socio-economic data. The two columns reflect the calculations made using LSTM embeddings derived from the last layer or all layers of the neural network. Figure 12. Cluster Stability plotted over time in comparison with PIT=60 for the cases of (top) without socio-economic data and (bottom) with socio-economic data. Figure 12 plots the cluster stability metrics over a wide range of PIT variations and compares the effect of socio-economic data on the stability of the clusters. Adding the socio-economic data to the LSTM embeddings stabilized the cluster formation over time.

Conclusion
The work presented here demonstrates improved LSTM forecasting through embeddings derived from loosely-decoupled timeseries. We applied this methodology to COVID-19 infection and death at the US county level. Our socio-economic embedding approach demonstrates enhanced 10-day moving average predictions compared to traditional LSTM modeling, especially in conjunction with K-means clustering of the final layer embeddings. Additionally, we demonstrate stability in the clustering of LDTs when combined with socio-economic data, providing increased consistency in predictions. With this approach, US counties later in catching the virus benefit from counties similar in socio-economic demographics, but with an earlier start to their disease propagation, improving the predictive outcome.


extfieldsofstudy:
Computer Science
Engineering

s2fieldsofstudy:
Computer Science
Physics

text:
MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes

We propose a mesh-based neural network (MESH2IR) to generate acoustic impulse responses (IRs) for indoor 3D scenes represented using a mesh. The IRs are used to create a high-quality sound experience in interactive applications and audio processing. Our method can handle input triangular meshes with arbitrary topologies (2K - 3M triangles). We present a novel training technique to train MESH2IR using energy decay relief and highlight its benefits. We also show that training MESH2IR on IRs preprocessed using our proposed technique significantly improves the accuracy of IR generation. We reduce the non-linearity in the mesh space by transforming 3D scene meshes to latent space using a graph convolution network. Our MESH2IR is more than 200 times faster than a geometric acoustic algorithm on a CPU and can generate more than 10,000 IRs per second on an NVIDIA GeForce RTX 2080 Ti GPU for a given furnished indoor 3D scene. The acoustic metrics are used to characterize the acoustic environment. We show that the acoustic metrics of the IRs predicted from our MESH2IR match the ground truth with less than 10% error. We also highlight the benefits of MESH2IR on audio and speech processing applications such as speech dereverberation and speech separation. To the best of our knowledge, ours is the first neural-network-based approach to predict IRs from a given 3D scene mesh in real-time.

INTRODUCTION
Rapid developments in interactive applications (e.g., games, virtual environments, speech recognition, etc.) demand realistic sound effects in complex dynamic indoor environments with multiple sound sources. Generating realistic sound effects is still a challenging problem because of the geometric and aural complexity of the real-world environment. The geometric complexity is a measure of the number of vertices and faces needed to represent the complex environment as a 3D mesh. The aural complexity depends on the number of sound sources, the number of static and dynamic objects in the environment and the acoustic effects (e.g., early reflection, late reverberation, diffraction, scattering, etc.). The way the sound propagates from a sound source to the listener can be modeled as an impulse response (IR) [41]. We can generate sound effects by convolving the IR with a dry sound. The IRs are used to generate plausible sound effects in many interactive applications used for games and AR/VR: Steam Audio [1], Project Acoustics [2], and Oculus Spatializer [3].
Recently, many machine learning-based algorithms are proposed to synthesize the sounds in musical instruments [28,31], estimate the acoustic material properties, and model the acoustic effects from finite objects [32,33,49,57]. Neural-network-based IR generators [59,61] for simple rooms are proposed for speech processing applications (e.g., far-field speech recognition, speech enhancement, speech separation, etc.). FAST-RIR [61] is a GAN-based IR generator that takes shoe-box-shaped room dimensions, listener and source positions, and reverberation time as inputs and generates a large IR dataset on the fly. Gaming applications demand fast IR generators for complex scenes. Complex indoor 3D scenes with furniture can be represented in detail using mesh models. Traditional geometric IR simulators take a 3D scene mesh-model and listener and source positions as inputs and generate realistic environmental sound effects [64]. The complexity of geometric IR simulators increases exponentially with the reflection depth [6] and makes them impractical for interactive applications. No current simulation and learning methods can compute real-time IRs for unseen complex dynamic scenes.
Sound propagation can be accurately simulated using wavebased methods. The wave-based methods solve wave equations using different numerical solvers such as the boundary-element method [79], the finite-difference time-domain simulation [10], etc. The complexity of the wave-based approach grows as the fourth power of the frequency and is limited to static scenes. Geometric acoustic algorithms are a less complex alternative to the wave-based method. Geometric acoustic algorithms can handle complex environments with dynamic objects [43] and multiple sources [66]. However, geometric acoustic methods do not model the low-frequency components of the IRs accurately because of the ray assumption. The sound wave can be treated as a ray when the wavelength of the sound is smaller than the obstacles in the environment. At low frequencies under 500 Hz, the ray assumption is not valid for most scenes and results in significant simulation errors. Hybrid sound propagation algorithms [73,80] combine IRs from wave-based and geometric techniques to generate accurate IRs in the human aural range for complex dynamic scenes. Generating IRs corresponding to thousands of sound sources at an interactive rate is not possible with the hybrid method because of its complexity.
Main Contributions: We propose a novel learning-based IR generator (MESH2IR) to generate realistic IRs for furnished 3D scenes with arbitrary topologies (i.e., meshes with 2000 faces to 3 million faces) not seen during the training. For a given complex scene, MESH2IR can generate IRs for any listener and source positions. We perform mesh simplification to even handle complex 3D scenes represented using a mesh with millions of triangles. Our mesh encoder network significantly reduces the input data size by transforming the 3D scene meshes into low-dimensional vectors of a latent space. Mesh simplification and its encoder network allow us to handle general 3D scenes meshes with a varying number of faces. We present an efficient approach to preprocess the IR training dataset and show that training MESH2IR on preprocessed dataset gives a significant improvement in the accuracy of IR generation. We also evaluate the contribution of energy decay relief in improving the IRs generated from MESH2IR. We train our MESH2IR on an IR dataset computed using a hybrid sound propagation algorithm [73]. Therefore, the predicted IRs from MESH2IR exhibit good accuracy for both low-frequency and high-frequency components. MESH2IR can generate more than 10,000 IRs for a given indoor 3D scene. We show that our MESH2IR can generate IRs 200 times faster than a geometric acoustic simulator [74] on a single CPU. We evaluate the accuracy of the predicted IRs from MESH2IR using power spectrum and acoustic metrics which characterize the acoustic environment. Our MESH2IR can predict the IRs for unseen 3D scenes during training with less than 10% error in three different acoustic metrics. We also show that far-field speech augmented using the IRs generated from MESH2IR significantly improves the performance in speech processing applications.

RELATED WORKS
We first give an overview of IR simulators that can compute IRs at interactive rates in Section 2.1. In Section 2.2, we summarize the deep learning-based algorithms used to predict IRs and describe the prior learning-based IR generator. We highlight various applications in audio and speech processing that can use the predicted IRs in Section 2.3. Finally, we mention different publicly available indoor 3D scenes datasets and give an overview of prior IR datasets.

PHYSICS-BASED IR COMPUTATION
Many wave-based, geometric-based, and hybrid interactive IR simulation algorithms have been proposed to simulate IRs for complex scenes [44]. Wave-based and hybrid algorithms precompute the IRs for a static scene and, at runtime, the IR for an arbitrary listener position is calculated by efficient interpolation techniques [48,58,80]. These precomputation-based interactive IR simulation algorithms can be used only for static scenes [44]. Geometric-based algorithms are proposed for interactive IR simulation in dynamic scenes [43,65,75]. The limitations of geometric-based techniques, such as simulation error at lower frequencies, are inherent in these interactive geometric-based algorithms. There are no general physicsbased algorithms known for computing accurate IRs, including low-frequency components, for general dynamic scenes.

LEARNING-BASED IR COMPUTATION
Recently, neural-network-based methods have been developed to estimate IRs from the reverberant speech signal [70] or from a single image of the environment [38,69], to translate synthetic IRs to real-word IRs [60], to interpolate IRs [62], and to augment the number of IRs [11,59]. FAST-RIR is a learning-based IR generator that is trained to generate specular and diffuse reflections for a given empty shoe-box-shaped room. FAST-RIR may not compute low-frequency components of IRs accurately.

AUDIO PROCESSING USING IR
IRs are used in a wide range of practical applications such as audiovisual navigation [15,16,45], acoustic matching [14], sound rendering [66], sound source localization [25], speech enhancement [54], speech recognition [47,68], and speech separation [7,29]. In most applications, learning-based networks are trained on large, diverse synthetic datasets and tested in real-world environments [74]. The performance of the deep neural network trained for a particular application is depended on the similarity of the synthetic training dataset and the real-world test environment [59]. Synthetic training datasets are created using IR generators. Therefore, the accuracy of the IR generator plays an important role in practical applications that depends on IRs.

INDOOR 3D SCENES & IR DATABASES
The indoor 3D scenes can be either captured using RGB-D videos and reconstructed as meshes (real-world scenes) [8,13,18] or created by humans using professionally designed software (synthetic models) [21,27]. The mesh quality in the real-world scene datasets is not as good as the quality of the synthetic models because 3D scene reconstruction with accurate geometric details is challenging with existing computer vision methods and capturing hardware. Among the synthetic model datasets, 3D-FRONT [21] contains large-scale synthetic furnished indoor 3D scenes with fine geometric and texture details. The 3D-FRONT dataset has 6813 CAD houses, where 18,968 rooms are furnished with 13,151 3D furniture objects. The furniture is placed in varying numbers in meaningful locations in each room (e.g., living room, dining room, kitchen, bedroom, etc.) The publicly available IR datasets are either recorded in a realworld environments (recorded IR) [12,26,40,71] or generated using IR simulation tools (synthetic IR) [15,24,37,73]. The recorded IR datasets are limited in size (fewer than 5000 IRs) and number of environments (fewer than 10 rooms), and no sufficient information on the recording conditions is provided to train a deep learning model. Synthetic IR datasets like SoundSpaces [15] and GWA [73] contain millions of IRs. The IRs in SoundSpaces is simulated using a geometric acoustic method [64] and GWA dataset contains highquality IRs computed using a hybrid method.

Mesh Simplification
Mesh latent vector Figure 1: The architecture of our MESH2IR. Our mesh encoder network encodes a indoor 3D scene mesh to the latent space. The mesh latent vector and the source and listener locations are combined to produce a scene vector embedding ( A ). The generator network generates an IR corresponding to the input scene vector embedding. For the given scene vector embedding, the discriminator network discriminates between the generated IR and the ground truth IR during training.

MESH2IR: OUR APPROACH 3.1 OVERVIEW
Our goal is to predict the IR for the given indoor 3D scene and the source and listener positions (Equation 1). The 3D scenes are represented as triangular meshes and we do not make any assumptions about the topology of the 3D scenes. The mesh format describes the shape of an object using vertices, edges, and faces consisting of triangles represented using 3D Cartesian coordinates (x,y,z). The source positions and listener positions are also represented using 3D Cartesian coordinates. For each scene material (e.g., furniture, floor, wall, ceiling, etc.), we do not explicitly control the characteristics of the scene materials (the amount of absorption or scattering of sound by the scene material). The IRs used to train our MESH2IR is computed by considering the characteristic of scene material in every indoor 3D scene. Therefore, our trained MESH2IR randomly assigns the characteristics of scene materials based on their shape.
The IR is the response of an impulse signal emitted in an environment. IR describes the relationship between a dry sound and the reflected sound from the boundaries in the scene. The reflected sound signal depends on the scene geometry, scene materials, and source and listener positions. IRs can be accurately simulated by solving the wave equation [10]. However, solving a wave equation is not practical for interactive applications because of its complexity. In our work, we propose a learning-based IR generator (MESH2IR) that is capable of approximating thousands of IRs per second for a given complex scene. Our network can be formally described as: where is the predicted IR for the given scene , is the 3D mesh representation of the scene simplified to have around 2000 faces, and are listener location and source location, respectively. and are represented using a 3D vector. We simplify the 3D scene mesh with an arbitrary number of faces to have a constant number of faces (2000 faces). We use a meshbased encoder network N 2 to transform the simplified 3D mesh into a latent vector of a latent space ( Figure 2). N 1 is a generator network, and 1 and 2 are the trained network parameters. Our overall network architecture is shown in Figure 1.
In recent years, cross-modal translation neural networks have gained attention in computer vision. Several algorithms have been proposed for translating video to audio [55], image to audio [69], text to image [81,82], image to mesh [77], etc. In our work, we translate complex scenes represented using meshes to acoustic IRs represented as audio signals.

TRAINING DATASET
In our work, we train MESH2IR on the GWA IR dataset [73] simulated using a hybrid algorithm to generate realistic IRs on unseen complex environments. The IRs in GWA are created by automatically calibrating the ray energies simulated using the geometric acoustic method [74] with the wave effects simulated using finitedifference time-domain wave solver [10] to create high-quality low-frequency and high-frequency wave effects. The GWA dataset consists of 2 million IRs simulated on the indoor 3D environments represented as meshes in the 3D-FRONT dataset. Out of the 2 million IRs, we train MESH2IR on 200,000 IRs simulated in 5,000 different indoor environments from 3D-FRONT dataset [21].

MESH SIMPLIFICATION
The number of faces in the 3D-FRONT dataset varies from about 2000 faces to 3 million faces. We initially simplify the meshes using a quadratic-based edge collapse algorithm in PyMeshLab [51] to have a fewer number of faces (i.e., 2000 faces). The edge collapse algorithm makes sure that the approximation error between the original mesh and the simplified mesh, in terms of Hausdorff distance is small. Therefore the acoustic characteristics do not change due to the simplification step. Figure 3 depicts an example of the original indoor 3D scene mesh and the simplified mesh using the quadratic edge collapse algorithm. We can see that high-level details of the furniture such as bed, pillow, settee, table and cupboard are preserved in the simplified mesh, in addition to the scene geometry. In this example, the simplified mesh has only 2% of faces of the original mesh.

MESH ENCODER
Our goal is to transform the simplified indoor scene meshes into a low-dimensional latent space. The triangular meshes can be represented as graph data. Therefore, we represent the 3D scene meshes as a graph and use a graph network (Figure 2) to reduce the dimension. Our graph network uses graph convolution (GCN) layers [35] Vertex position Edge index Mesh Latent Vector  The indoor 3D scene is represented using an original mesh with around 100,000 faces and a simplified mesh with 2,000 faces. We can see that high-level details of the room geometry and furniture are preserved in the simplified mesh.
to encode the topology information (i.e., edge connectivity) and node features (i.e., vertex coordinates) of the graph. We use graph pooling (gpool) layers [22,36] to reduce the topology information of the graph. The layer-wise propagation rule of multi-layer GCN [35]: where^= ^, and^= + . is the adjacency matrix, is the identity matrix and ( ) is a trainable weight matrix for layer . The feature matrices at layers and + 1 are ( ) and ( +1) , respectively. The topology information is not modified in the GCN layer. The gpool layer [22] creates a new graph with high-level feature encoding by choosing vertices from the original vertex set and discarding the other vertices. Some edges are removed when discarding vertices in the gpool layer. This results in some isolated vertices in the graph. To address this problem, the gpool algorithm increases the graph connectivity by calculating the square of the adjacency matrix ( ) at layer and uses the new adjacency matrix ( ) for gpool computation (Equation 3).
We calculate the channel-wise average (GAP) and the channelwise maximum (GMP) of the node features after each gpool layer and aggregate the GAP and GMP values. We concatenate the aggregated GAP and aggregated GMP values, pass them to linear layers, and get the mesh latent vector ( ) (dimension=8) as the output.

SCENE VECTOR EMBEDDING
We concatenate the mesh latent vector ( ) with the source ( ) and listener positions ( ) in 3D Cartesian space to generate scene vector embedding of dimension 14 (Equation 4). The mesh latent vector will be learned during training.

IR REPRESENTATION & PREPROCESSING
The IRs in the GWA dataset [73] have a sampling rate of 48 kHz. We downsample the IRs in the dataset to 16 kHz and crop the IRs. Downsampling IRs allows us to maintain a longer duration of IRs in a fixed-length input IR vector (3968 samples). The IRs with full duration and the IRs cropped to have a duration of around 0.25 seconds give similar performance in speech applications [61].
IR preprocessing: The IRs in the GWA dataset have large variations in standard deviation (STD) of the magnitude values (10 −12 to 10 −2 ). We noticed that it is hard for MESH2IR to learn from such datasets with high dynamic ranges. To overcome this issue, we divide the IRs by ten times the STD to create preprocessed IRs with the same STD of 0.1 (or any constant STD). We noticed that preprocessing IRs to have constant STD improves the accuracy of MESH2IR (see Section 4.1).
To recover the IR with the original magnitude, we duplicate the STD of the IRs 128 times and concatenate them at the end of the preprocessed IRs. The concatenated IRs will have a length of 4096 (3968+128). The convolution layers in MESH2IR calculate an average of 41 neighboring sample values for each sample of the concatenated IRs and pass them to the next layer. Therefore, concatenated STD values near the end of preprocessed IR magnitudes in a concatenated IR will be corrupted when we calculate the average and pass it to the next layers. We can recover the STD values from the tail-end of the concatenated IR where all the 41 neighboring samples are STD.

ENERGY DECAY RELIEF
The energy decay relief (EDR) obtained from the physics-based algorithm contains enough information to be converted into an "equivalent impulse response" [42]. Therefore, we use EDR in the cost function of our MESH2IR. Constraining the Generator network with additional information (i.e., EDR) helps the model to converge easily.
The energy decay curve (EDC) describes the energy remaining in the IR with respect to time [67]. When compared to the IR itself, the EDC decays more smoothly, and we can use it to measure IR acoustic metrics. The generalized EDC for multiple frequency bands is called EDR [34]. EDR is the total amount of energy remaining in the IR at time seconds in a frequency band centered at Hz: In Equation 5, bin of the short-time Fourier transform at timeframe is denoted by ( , ).
is the total number of time frames.

MODIFIED CONDITIONAL GAN
Our MESH2IR passes the scene vector embedding (Equation 4) to a one-dimensional modified conditional GAN (CGAN) architecture proposed in FAST-RIR [61] to generate a single precise IR for the given indoor 3D scene. CGAN [23,50] is conditioned on a random noise and on a condition to generate multiple different outputs that satisfy the condition . The modified CGAN is only conditioned on to generate a single output.
MESH2IR consists of a generator network ( ) and a discriminator network ( ), which are alternately trained at each iteration. We train and using the IR samples from the data distribution . The objective function of our generator network consists of modified CGAN error, EDR error, and mean square error (MSE). We train the discriminator network with a modified CGAN objective function. For each , we use the corresponding IRs in the GWA dataset [73] as the ground truth when training our network.

Modified CGAN Error (Generator Network):
The CGAN error is used to generate IRs that are hard for the to differentiate from the ground truth IRs.
EDR Error: For each , we calculate the EDR of the generated IR using our MESH2IR ( ) and the ground truth IR ( ). We calculate EDR at a set of frequency bands with center frequencies at 125 Hz, 250 Hz, 500 Hz, 1000 Hz, 2000 Hz, and 4000 Hz ( ). We compare the and for each sample ( ) as follows: The signal energy in high-frequency components of EDR is high when compared with the signal energy in low-frequency components of EDR in the training IR dataset [73]. Therefore, we give more weight to low-frequency components of EDR.
MSE Error: For each , we compare the IR generated from MESH2IR ( ) with the ground truth IR ( ). We calculate the squared difference of each sample ( ) in and as follows: The generator network ( ) and the discriminator network ( ) are trained to compete each other by minimizing the generator objective function L (Equation 9) and maximizing the discriminator objective function L (Equation 10 ). In Equation 9, the relative importance of the EDR error and MSE error are controlled using and respectively.

IMPLEMENTATION DETAILS
Network Architecture: We adapt the graph encoder network in the PyG official repository [4] and modify the network to encode indoor 3D scene meshes to the latent space (Figure 2). Our gpool layer keeps 60% of the original number of vertices in each layer. We use the Generator ( ) network and the Discriminator network ( ) architectures proposed in FAST-RIR [61] and modify their cost functions. We extend the architecture proposed in FAST-RIR by adding our mesh encoder network (Figure 1).
Training: We trained and using the RMSprop optimizer with a batch size of 256. The is iterated 3 times for every iteration. Initially, we start with the learning rate of 8x10 −5 and decay the learning rate by 85% of the original value for every 7 epochs. We trained our network for 150 epochs. We published our code for reproducibility at github [5]. Table 1: The accuracy of IRs generated using MESH2IR on 5 different furnished indoor 3D scenes not used for training. We plot the time-domain representation of ground truth IRs and the predicted IRs, and evaluate the accuracy of predicted IRs using relative reverberation time (T 60 ) error, relative direct-reverberant-ratio (DRR) error, and relative early-decay-time (EDT) error. Our MESH2IR can predict the IRs with less than 10% T 60 and DRR errors, and less than 3% EDT error. We also can see that the envelope and magnitude of the predicted IRs matches the ground truth IRs.

ABLATION EXPERIMENTS
We perform an ablation study to analyze the importance of our IR preprocessing approach (Section 3.6) and to find an efficient way of adding EDR to the cost function. We quantitatively evaluate different variations of our network by calculating the mean absolute difference of different acoustic metrics of the generated IRs and the ground truth IRs. The acoustic characteristics of the 3D indoor environment are described using acoustic metrics [11]. We also measure the mean square error (MSE) of the generated IRs and the ground truth IRs (Equation 8). We train all the variations of our network with 200K IRs from 5000 different indoor 3D scene meshes and generate 11K testing IRs from 600 scene meshes not used during training. We use commonly-used acoustic metrics such as reverberation time ( 60 ), early-decay-time (EDT), and direct-toreverberant ratio (DRR) in our experiment. The time taken to decay the sound pressure by 60 decibels (dB) is called 60 . DRR is the ratio of the sound pressure level of the direct sound to the sound pressure level of reflected sound [53]. EDT is six times the time taken for the sound source to decay by 10 dB, and it depends on the type and location of the sound source.

IR PREPROCESSING
We evaluate the contribution of our IR preprocessing approach discussed in Section 3.6. We train our MESH2IR on IRs without IR preprocessing (MESH2IR-UNPROCESSED) and generate IRs from our trained network. Figure 4 shows a ground truth IR and the IR generated from MESH2IR-UNPROCESSED. We can see that MESH2IR-UNPROCESSED generates noisy output. In Table 1 we can see that MESH2IR estimates the IRs for the given 3D scene to a greater extent.

ENERGY DECAY RELIEF (EDR)
To evaluate the importance and the efficient way of adding EDR to the cost functions, we train and compare two different variations of our MESH2IR network.
Variation 1 (MESH2IR-NO-EDR) : We evaluate the role played by the EDR loss in the generator network by training a variant of MESH2IR without the EDR loss (Equation 7).
Variation 2 (MESH2IR-D-EDR) : Instead of training the Discriminator network to discriminate between generated IRs and ground truth IRs of dimension 3968 x 1, we train the Discriminator network to discriminate between the EDR of the generated IRs and    the ground truth IRs. EDR is calculated at six octave bands with center frequencies at 125 Hz, 250 Hz, 500 Hz, 1000 Hz, 2000 Hz and 4000 Hz. The EDR of an IR has a dimension of 3986x6. We also train the Generator network with the EDR loss.
From Table 2, we can see that adding EDR in the generator loss function improves the 60 , EDT, and MSE. Training the Discriminator network with IRs (MESH2IR) gives a similar performance to passing EDR of the IRs (MESH2IR-D-EDR).

POWER SPECTRUM
The power spectrum describes the energy distribution of the IR in the frequency components that compose the waveform. In Figure 5, we compare the power spectrum of ground truth IRs with the power spectrum of the predicted IR using MESH2IR, MESH2IR-D-EDR, and MESH2IR-NO-EDR. In this example, we placed the listener 1m and 8m away from the source. We can see that in both cases the (a) The source is placed around 1m away from the listener.
(b) The source is placed around 8m away from the listener. Figure 5: The power spectrum of IRs generated using our proposed MESH2IR, MESH2IR-D-EDR, MESH2IR-NO-EDR, and the ground truth IRs from the GWA dataset. We can see that the power spectrum of MESH2IR is closest to the power spectrum of GWA.
power spectrum of MESH2IR is closer to GWA when compared with other variants of our approach.

ACOUSTIC EVALUATION
In this section, we evaluate the accuracy, power spectrum and runtime of our MESH2IR network. We evaluate the accuracy of our MESH2IR network by comparing the relative acoustic metric error values of the IRs predicted from MESH2IR on indoor 3D scenes not used during training and the ground truth IRs from the GWA dataset [73]. We compare the run-time of MESH2IR network with a state-of-the-art geometric acoustic simulator [74].

ACCURACY ANALYSIS
To evaluate the robustness of our MESH2IR, we selected ground truth IRs from the GWA dataset [73] from 5 different indoor 3D scenes with different levels of complexity (the number of faces in a 3D scene mesh). The selected IRs have a large variation in magnitudes and shapes. The distance between the listener and the source position varies from 3.6m to 10.5m. We predicted IRs corresponding to the ground truth IRs in 3D scenes using our MESH2IR. We evaluate the accuracy of our prediction using relative 60 error, relative DRR error, and relative EDT error. Table 1 shows the ground truth IRs, predicted IRs and the accuracy of our prediction. We can see that our approach can predict the IRs with less than 10% 60 and DRR errors, and less than 3% EDT error, which are significantly lower than their just-noticeable-differences (JNDs) [9,19,78].

RUNTIME
We compare the runtime for generating 30,000 IRs from our MESH2IR with the time required to generate these samples from a geometricbased acoustic simulator (GA) [74] and FAST-RIR [61]. The runtime of GA depends on the complexity of the scene, and FAST-RIR is only capable of generating IRs for empty shoe-box-shaped mediumsized rooms (i.e., room dimensions varying from [8m,6m,2.5m] to [11m,8m,3.5m]). We compare the runtime of GA and FAST-RIR given in Ratnarajah et al. [61] with the runtime of MESH2IR for generating IRs for furnished indoor 3D scene meshes in the 3D-Front dataset [21]. We use an Intel(R) Xenon(R) CPU E52699 v4 @ 2.20 GHz and a GeForce RTX 2080 Ti GPU for our evaluation.
For a fair comparison with GA and FAST-RIR methods, which take 3D room dimensions as inputs, we did not consider the time taken for mesh simplification and mesh-to-graph conversion in Table 3. On average, mesh simplification takes around 7.5 seconds and mesh-to-graph conversion takes 0.04 seconds. From Table 3, we can see that MESH2IR is more than 200 times faster than GA on a CPU. MESH2IR is optimized to run on GPUs and supports batch parallelization. MESH2IR is slower than FAST-RIR because we encode a 3D scene mesh represented using a graph to a mesh latent vector using our graph neural network (Figure 2). To generate thousands of IRs for a given furnished indoor 3D scene, we perform mesh simplification, mesh-to-graph conversion, and mesh encoding only once.
We have shown the average time taken for mesh encoding (MESH2IR[Mesh Encoder]) and IR generation using the encoded mesh (MESH2IR[IR Generator]) separately in Table 3. Our MESH2IR can generate more than 10,000 IRs per second on a single GPU for a given furnished indoor 3D scene, and the runtime of MESH2IR is stable irrespective of the scene.

APPLICATIONS
We demonstrate the benefit of our MESH2IR for several downstream speech applications -neural sound rendering, speech dereverberation and reverberant speech separation. For a fair comparison between different methods, we generate 11K IRs from 600 scene meshes not used during training our MESH2IR using GA [74], GWA [73], MESH2IR-D-EDR, and MESH2IR. We generate reverberant speech using the 11K IRs, and train speech dereverberation and speech separation methods. Reverberant speech can be generated from a dry sound source and an IR via a convolution: where ( ) is the reverberant speech signal, ( ) is the IR, ( ) is the dry speech signal.

NEURAL SOUND RENDERING
Our MESH2IR is the first neural-network-based approach to predict IRs from a given 3D scene mesh at interactive rates. Given this advantage, MESH2IR can be applied to general 3D scenes to enable real-time neural sound rendering, without pre-computation on new scenes. For single IR updates in dynamic scenes, MESH2IR can operate at more than 100 frames per second (fps), which is significantly beyond the requirement for interactive applications

SPEECH DEREVERBERATION
Speech dereverberation is the process of obtaining reverb-free speech from reverberant speech. We train speech dereverberation models using data generated from different synthetic IR generation methods and compare their performance on data generated from recorded IRs. We test the models on IRs from the MIT IR dataset [76], the BUTReverb dataset [72], and the RWCP Aachen IR dataset [30,52]. For all experiments, we train the SkipConvNet [39] model with its default parameters. We use the metric speech-toreverberation modulation energy ratio (SRMR) [20] to measure the improvement obtained by the dereverberation process. Higher SRMR implies higher speech quality and lower reverberation effects. We also report the SRMR of the baseline where we do not apply any dereverberation (Reverb).
In Table 4, we test the models on reverberant data generated from recorded IRs. Across all IR datasets, we see that the SRMR of our MESH2IR model is similar to the SRMR of the GWA IRs [73]. We also see that MESH2IR outperforms the GA [74] method, which only generates IRs using geometric simulations (less accurate than IRs generated from the GWA dataset). The SRMR of MESH2IR is closer to GWA IRs when compared with MESH2IR-D-EDR. Therefore, MESH2IR generates more accurate IRs when compared with MESH2IR-D-EDR.

SPEECH SEPARATION
Speech separation, also referred to as the cocktail party problem, is the process of separating a mixture of speech signals into its constituent speech signals. We check the performance of different synthetic IR generation methods on the task of reverberant speech separation -separating a reverberant mixture into its constituent reverberant speech signals. The dry speech signals and mixtures from the Libri2Mix [17] dataset are convolved with IRs to generate the reverberant data. We train the DPRNN-TasNet [46] model for all  speech separation experiments. We utilize the default implementation provided by the Asteroid [56] framework. The model is tested on reverberant mixtures created from real recordings obtained from four different room configurations in the VOiCES [63] dataset. We clearly see that our MESH2IR performs similar to GWA [73], and outperforms GA method [74] and MESH2IR-D-EDR.

CONCLUSION AND FUTURE WORK
We present a novel neural-network-based MESH2IR architecture to generate thousands of IRs for a given furnished indoor 3D scene on the fly. The IR generation speed of our MESH2IR is constant within a given complex 3D scene, irrespective of the complexity of the scene. Our MESH2IR can generate thousands of IRs per second for a given 3D scene. We show that the IRs predicted by our MESH2IR in unseen indoor 3D scenes are highly similar to the ground truth IRs generated from the GWA dataset, which is used to train our MESH2IR. Our work has some limitations. One is we cannot control the characteristics of the scene materials such as the amount of sound absorption and scattering, which can affect the overall amplitude of the IR. Efficiently inputting the characteristics of scene material to our MESH2IR may improve the accuracy of IR generation. In addition, while MESH2IR can handle moving sound by simulating many IRs according to a trajectory, when objects in the scene moves, the scene representation changes, which incurs additional encoding time for MESH2IR, making it less efficient in highly dynamic scenes.
In the future, we would like to integrate our MESH2IR into game engines and other interactive applications, and evaluate its benefits.