Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following. The neural network models that provide the dominant solution to these problems are brittle, especially in low-resource settings: they fail to generalize correctly or systematically from small datasets. Past work has shown that many failures of systematic generalization arise from neural models' inability to disentangle lexical phenomena from syntactic ones. To address this, we augment neural decoders with a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontex-tualized, token-level translation rules. We describe how to initialize this mechanism using a variety of lexicon learning algorithms, and show that it improves systematic generalization on a diverse set of sequence modeling tasks drawn from cognitive science, formal semantics, and machine translation. 1 1 Introduction Humans exhibit a set of structured and remarkably consistent inductive biases when learning from language data. For example, in both natural language acquisition and toy language-learning problems like the one depicted in Fig. 1, human learners exhibit a preference for systematic and compositional interpretation rules (Guasti 2017, Chapter 4; Lake et al. 2019). These inductive biases in turn support behaviors like one-shot learning of new concepts (Carey and Bartlett, 1978). But in natural language processing, recent work has found that state-of-the-art neural models, while highly effective at in-domain prediction, fail to generalize in human-like ways when faced with rare phenomena 1 Our code is released under https://github.com/ ekinakyurek/lexical Train Test dax lug wif zup zup fep zup blicket lug ___ ? and small datasets (Lake and Baroni, 2018), posing a fundamental challenge for NLP tools in the low-data regime. Pause for a moment to fill in the missing labels in Fig. 1. While doing so, which training examples did you pay the most attention to? How many times did you find yourself saying means or maps to ? Explicit representations of lexical items and their meanings play a key role diverse models of syntax and semantics (Joshi and Schabes, 1997; Pollard and Sag, 1994; Bresnan et al., 2015). But one of the main findings in existing work on generalization in neural models is that they fail to cleanly separate lexical phenomena from syntactic ones (Lake and Baroni, 2018). Given a dataset like the one depicted in Fig. 1, models conflate (lexical) information about the correspondence between zup and y with the (syntactic) fact that y appears only in a sequence of length 1 at training time. Longer input sequences containing the word zup in new syntactic contexts cause models to output tokens only seen in longer sequences (Section 5). In this paper, we describe a parameterization for sequence decoders that facilitates (but does not enforce) the learning of context-independent word meanings. Specifically, we augment decoder output layers with a lexical translation mechanism which generalizes neural copy mechanisms (e.g. See et al., 2017) and enables models to generate token-level translations purely attentionally. While the lexical translation mechanism is quite general, we focus here on its ability to improve few-shot learning in sequence-to-sequence models. On a suite of challenging tests of few-shot semantic parsing and instruction following, our model exhibits strong generalization, achieving the highest reported results for neural sequence models on datasets as diverse as COGS (Kim and Linzen 2020, with 24155 training examples) and Colors (Lake et al. 2019, with 14). Our approach also generalizes to real-world tests of few-shot learning, improving BLEU scores (Papineni et al., 2002) by 1.2 on a low-resource EnglishChinese machine translation task (2.2 on test sentences requiring one-shot word learning). In an additional set of experiments, we explore effective procedures for initializing the lexical translation mechanism using lexicon learning algorithms derived from information theory, statistical machine translation, and Bayesian cognitive modeling. We find that both mutual-information-and alignmentbased lexicon initializers perform well across tasks. Surprisingly, however, we show that both approaches can be matched or outperformed by a rule-based initializer that identifies high-precision word-level token translation pairs. We then explore joint learning of the lexicon and decoder, but find (again surprisingly) that this gives only marginal improvements over a fixed initialization of the lexicon. Introduces a new, lexicon-based output mechanism for neural encoderdecoder models. Investigates and improves upon lexicon learning algorithms for initialising this mechanism. Uses it to solve challenging tests of generalization in instruction following, semantic parsing and machine translation. A great deal of past work has suggested that neural models come equipped with an inductive bias that makes them fundamentally ill-suited to human-like generalization about language data, especially in the low-data regime (e.g. Fodor et al., 1988; Marcus, 2018). Our results suggest that the situation is more complicated: by offloading the easier lexicon learning problem to simpler models, neural sequence models are actually quite effective at modeling (and generalizing about) about syntax in synthetic tests of generalization and real translation tasks. Systematic generalization in neural sequence models The desired inductive biases noted above are usually grouped together as systematicity but in fact involve a variety of phenomena: one-shot learning of new concepts and composition rules (Lake and Baroni, 2018), zero-shot interpretation of novel words from context cues (Gandhi and Lake, 2020), and interpretation of known concepts in novel syntactic configurations (Keysers et al., 2020; Kim and Linzen, 2020). What they share is a common expectation that learners should associate specific production or transformation rules with specific input tokens (or phrases), and generalize to use of these tokens in new contexts. Recent years have seen tremendous amount of modeling work aimed at encouraging these generalizations in neural models, primarily by equipping them with symbolic scaffolding in the form of program synthesis engines (Nye et al., 2020), stack machines (Grefenstette et al., 2015; Liu et al., 2020), or symbolic data transformation rules (Gordon et al., 2019; Andreas, 2020). A parallel line of work has investigated the role of continuous representations in systematic generalization, proposing improved methods for pretraining (Furrer et al., 2020) and procedures for removing irrelevant contextual information from word representations (Arthur et al., 2016; Russin et al., 2019; Thrush, 2020). The latter two approaches proceed from similar intuition to ours, aiming to disentangle word meanings from syntax in encoder representations via alternative attention mechanisms and adversarial training. Our approach instead focuses on providing an explicit lexicon to the decoder; as discussed below, this appears to be considerably more effective. Copying and lexicon learning In neural encoderdecoder models, the clearest example of benefits from special treatment of word-level production rules is the copy mechanism . A great deal of past work has found that neural models Inputs Outputs Lexicon Entries A crocodile blessed William . benefit from learning a structural copy operation that selects output tokens directly from the input sequence without requiring token identity to be carried through all neural computation in the encoder and the decoder. These mechanisms are described in detail in Section 3, and are widely used in models for language generation, summarization and semantic parsing. Our work generalizes these models to structural operations on the input that replace copying with general context-independent token-level translation. As will be discussed, the core of our approach is a (non-contextual) lexicon that maps individual input tokens to individual output tokens. Learning lexicons like this is of interest in a number of communities in NLP and language science more broadly. A pair of representative approaches (Brown et al., 1993; Frank et al., 2007) will be discussed in detail below; other work on lexicon learning for semantics and translation includes Liang et al. (2009); Goldwater (2007); Haghighi et al. (2008) among numerous others. Finally, and closest to the modeling contribution in this work, several previous papers have proposed alternative generalized copy mechanisms for tasks other than semantic lexicon learning. Concurrent work by Prabhu and Kann (2020) introduces a similar approach for grapheme-to-phoneme translation (with a fixed functional lexicon rather than a trainable parameter matrix), and Nguyen and Chiang (2018) and Gu et al. (2019) describe less expressive mechanisms that cannot smoothly interpolate between lexical translation and ordinary decoding at the token level. Pham et al. (2018) incorporate lexicon entries by rewriting input sequences prior to ordinary sequence-to-sequence translation. Akyrek et al. (2021) describe a model in which a copy mechanism is combined with a retrieval-based generative model; like the present work, that model effectively disentangles syntactic and lexical information by using training examples as implicit representations of lexical correspondences. We generalize and extend this previous work in a number of ways, providing a new parameterization of attentive token-level translation and a detailed study of initialization and learning. But perhaps the most important contribution of this work is the observation that many of the hard problems studied as compositional generalization have direct analogues in more conventional NLP problems, especially machine translation. Research on system-aticity and generalization would benefit from closer attention to the ingredients of effective translation at scale. This paper focuses on sequence-to-sequence language understanding problems like the ones depicted in Table 1, in which the goal is to map from a natural language input x = [ x 1 , x 2 , . . . , x n ] to a structured output y = [ y 1 , y 2 , . . . , y m ] a logical form, action sequence, or translation. We assume input tokens x i are drawn from a input vocabulary V x , and output tokens from a corresponding output vocabulary V y . Neural encoderdecoders Our approach builds on the standard neural encoderdecoder model with attention (Bahdanau et al., 2014). In this model, an encoder represents the input sequence [ x 1 , . . . , x n ] as a sequence of representations [ e 1 , . . . , e n ] e = encoder ( x ) (1) Next, a decoder generates a distribution over output sequences y according to the sequentially: log p ( y | x ) = y (cid:88) i =1 log p ( y i | y <i , e, x ) (2) Here we specifically consider decoders with attention . 2 When predicting each output token y i , we assign each input token an attention weight  ji as in Eq. (3). Then, we construct a context representation c i as the weighted sum of encoder representations e i :  ji  exp( h (cid:62) i W att e j ) (3) c i = | x | (cid:88) j =1  ji e j (4) The output distribution over V y , which we denote p write ,i , is calculated by a final projection layer: p ( y i = w | x ) = p write i ( w )  exp( W write [ c i , h i ]) (5) Copying A popular extension of the model described above is the copy mechanism , in which output tokens can be copied from the input sequence in addition to being generated directly by the decoder (Jia and Liang, 2016; See et al., 2017). Using the decoder hidden state h i from above, the model first computes a gate probability : p gate =  ( w (cid:62) gate h i ) (6) and then uses this probability to interpolate between the distribution in Eq. (5) and a copy distribution that assigns to each word in the output vocabulary a probability proportional to that word's weight in the attention vector over the input: p copy ( y i = w | x ) = | x | (cid:88) j =1 1 [ x j = w ]   ji (7) p ( y i = w | x ) = p gate  p write ( y i = w | x ) + (1  p gate )  p copy ( y i = w | x ) (8) (note that this implies V y  V x ). Content-independent copying is particularly useful in tasks like summarization and machine translation where rare words (like names) are often reused between the input and output. 2 All experiments in this paper use LSTM encoders and decoders, but it could be easily integrated with CNNs or transformers (Gehring et al. 2017; Vaswani et al. 2017). We only assume access to a final layer h i , and final attention weights  i ; their implementation does not matter. Our model: Lexical translation When the input and output vocabularies are significantly different, copy mechanisms cannot provide further improvements on a sequence-to-sequence model. However, even for disjoint vocabularies as in Fig. 1, there may be strict correspondences between individual words on input and output vocabularies, e.g. zup (cid:55) y in Fig. 1. Following this intuition, the lexical translation mechanism we introduce in this work extends the copy mechanism by introducing an additional layer of indirection between the input sequence x and the output prediction y i as shown in Fig. 2. Specifically, after selecting an input token x j  V x , the decoder can translate it to a context-independent output token  V y prior to the final prediction. We equip the model with an additional lexicon parameter L, a |V x |  |V y | matrix in which (cid:80) w L vw = 1 , and finally define p lex ( y i = w | x ) = | x | (cid:88) j =1 L x j w   ji (9) p ( y i = w | x ) = p gate  p write ( y i = w | x ) + (1  p gate )  p lex ( y i = w | x ) (10) The model is visualized in Fig. 2. Note that when V x = V y and L = I is diagonal, this is identical to the original copy mechanism. However, this approach can in general be used to produce a larger set of tokens. As shown in Table 1, coherent token-level translation rules can be identified for many tasks; the lexical translation mechanism allows them to be stored explicitly, using parameters of the base sequence-to-sequence model to record general structural behavior and more complex, context-dependent translation rules. The lexicon parameter L in the preceding section can be viewed as an ordinary fully-connected layer inside the copy mechanism, and trained end-to-end with the rest of the network. As with other neural network parameters, however, our experiments will show that the initialization of the parameter L significantly impacts downstream model performance, and specifically benefits from initialization with a set of inputoutput mappings learned with an offline lexicon learning step. Indeed, while not widely used in neural sequence models (though c.f. Section 2), lexicon-based initialization was a standard feature of many complex non-neural sequence transduction models, including semantic parsers (Kwiatkowski et al., 2011) and phrase-based machine translation systems (Koehn et al., 2003). But an important distinction between our approach and these others is the fact that we can handle outputs that are not (transparently) compositional. Not every fragment of an input will correspond to a fragment of an output: for example, thrice in SCAN has no corresponding output token and instead describes a structural transformation. Moreover, the lexicon is not the only way to generate: complex mappings can also be learned by p write without going through the lexicon at all. Thus, while most existing work on lexicon learning aims for complete coverage of all word meanings, the model described in Section 3 benefits from a lexicon with high-precision coverage of rare phenomena that will be hard to learn in a normal neural model. Lexicon learning is widely studied in language processing and cognitive modeling, and several approaches with very different inductive biases exist. To determine how to best initialize L , we begin by reviewing three algorithms in Section 4.1, and identify ways in which each of them fail to satisfy the high precision criterion above. In Section 4.2, we introduce a simple new lexicon learning rule that addresses this shortcoming. Statistical alignment In the natural language processing literature, the IBM translation models (Brown et al., 1993) have served as some of the most popular procedures for learning token-level inputoutput mappings. While originally developed for machine translation, they have also been used to initialize semantic lexicons for semantic parsing (Kwiatkowski et al., 2011) and grapheme-to-phoneme conversion (Rama et al., 2009). We initialize the lexicon parameter L using Model 2. Model 2 defines a generative process in which source words y i are generated from target words x j via latent alignments a i . Specifically, given a (source, target) pair with n source words and m target words, the probability that the target word i is aligned to the source word j is: p ( a i = j )  exp (cid:0)  (cid:12)(cid:12)(cid:12) i m  j n (cid:12)(cid:12)(cid:12)(cid:1) (11) Finally, each target word is generated by its aligned source word via a parameter  : p ( y i = w ) =  ( v, x a i ) . Alignments a i and lexical parameters  can be jointly estimated using the expectation maximization algorithm (Dempster et al., 1977). In neural models, rather than initializing lexical parameters L directly with corresponding IBM model parameters  , we run Model 2 in both the forward and reverse directions, then extract counts by intersecting these alignments and applying a softmax with temperature  : L vw  exp (cid:0)   1 (cid:88) ( x,y ) | y | (cid:88) i =1 1 [ x a i = v ] 1 [ y i = w ] (cid:1) (12) For all lexicon methods discussed in this paper, if an input v is not aligned to any output w , we map it to itself if V x  V y . Otherwise we align it uniformly to any unmapped output words (a mutual exclusivity bias , Gandhi and Lake 2020). Mutual information Another, even simpler procedure for building a lexicon is based on identifying pairs that have high pointwise mutual information . We estimate this quantity directly from co-occurrence statistics in the training corpus: pmi ( v ; w ) = log # ( v, w ) # ( v ) # ( w ) + log | D train | (13) where #( w ) is the number of times the word w appears in the training corpus and #( w, v ) is the number of times that w appears in the input and v appears in the output. Finally, we populate the parameter L via a softmax transformation: L vw  exp((1 / ) pmi ( v ; w )) . Bayesian lexicon learning Last, we explore the Bayesian cognitive model of lexicon learning described by Frank et al. (2007). Like IBM model 2, this model is defined by a generative process; here, however, the lexicon itself is part of the generative model. A lexicon (cid:96) is an (unweighted, many-to-many) map defined by a collection of pairs (x, y) with a description length prior: p ( (cid:96) )  e | (cid:96) | (where | (cid:96) | is the number of (input, output) pairs in the lexicon). As in Model 2, given a meaning y and a natural-language description x , each x i is generated independently. We define the probability of a word being used non-referentially as p NR ( x i | (cid:96) )  1 if x i (cid:54) (cid:96) and  otherwise. The probability of being used referentially is: p R ( x j | y i , (cid:96) )  1 ( x j ,y i )  (cid:96) . Finally, p ( x j | y i , (cid:96) ) = (1   ) p NR ( x j | (cid:96) ) +  | y | (cid:88) i =1 p R ( x j | y i , (cid:96) ) (14) To produce a final lexical translation matrix L for use in our experiments, we set L vw  exp((1 / ) p (( v, w )  (cid:96) )) : each entry in L is the posterior probability that the given entry appears in a lexicon under the generative model above. Parameters are estimated using the MetropolisHastings algorithm, with details described in Appendix C. 4.2 A Simpler Lexicon Learning Rule Example lexicons learned by the three models above are depicted in Fig. 3 for the SCAN task shown in Table 1. Lexicons learned for remaining tasks can be found in Appendix B. It can be seen that all three models produce errors: the PMI and Bayesian lexicons contain too many entries (in both cases, numbers are associated with the turn right action and prepositions are associated with the turn left action). For the IBM model, one of the alignments is confident but wrong, because the around preposition is associated with turn left action. In order to understand these errors, and to better characterize the difference between the demands of lexical translation model initializers and past lexicon learning schemes, we explore a simple logical procedure for extracting lexicon entries and thricetwice oppositeafteraroundrightwalkrunleftlookjump IBMModel-2 PMIIRIG HTIWALKIRU NIL EFTILO O KIJUMP and thricetwice oppositeafteraroundrightwalkrunleftlookjump Bayesian IRIG HTIWALKIRU NIL EFTILO O KIJUMP Simple Figure 3: Learned lexicons for the around right split in SCAN (  = 0 . 1 ). that, surprisingly, matchers or outperforms all three baseline methods in most of our experiments. What makes an effective, precise lexicon learning rule? As a first step, consider a maximally restrictive criterion (which we'll call C 1 ) that extracts only pairs ( v, w ) for which the presence of v in the input is a necessary and sufficient condition for the presence of w in the output. nec. ( v, w ) =  xy. ( w  y )  ( v  x ) (15) suff. ( v, w ) =  xy. ( v  x )  ( w  y ) (16) C 1 ( v, w ) = nec. ( v, w )  suff. ( v, w ) (17) C 1 is too restrictive: in many language understanding problems, the mapping from surface forms to meanings is many-to-one (in Table 1, both blessed and bless are associated with the logical form bless ). Such mappings cannot be learned by the algorithm described above. We can relax the necessity condition slightly, requiring either that v is a necessary condition for w , or is part of a group that collectively explains all occurrences of w : no-winner ( w ) = (cid:64) v (cid:48) . C 1 ( v (cid:48) , w ) (18) C 2 ( v, w ) = suff. ( v, w )  ( nec. ( v, w )  no-win. ( w )) (19) As a final refinement, we note that C 2 is likely to capture function words that are present in most sentences, and exclude these by restricting the lexicon to words below a certain frequency threshold: C 3 = C 2  (cid:12)(cid:12) { v (cid:48) : suff. The lexicon matrix L is computed by taking the word co-occurrence matrix, zeroing out all entries where C 3 does not hold, then computing a softmax: L vw  C 3 ( v, w ) exp((1 / ) #( v, w )) . Surprisingly, as shown in Fig. 3 and and evaluated below, this rule (which we label Simple) produces the most effective lexicon initializer for three of the four tasks we study. The simplicity (and extreme conservativity) of this rule highlight the different demands on L made by our model and more conventional (e.g. machine translation) approaches: the lexical translation mechanism benefits from a small number of precise mappings rather than a large number of noisy ones. We investigate the effectiveness of the lexical translation mechanism on sequence-to-sequence models for four tasks, three focused on compositional generalization and one on low-resource machine translation. In all experiments, we use an LSTM encoderdecoder with attention as the base predictor. We compare our approach (and variants) with two other baselines: GECA (Andreas 2020; a data augmentation scheme) and SynAtt (Russin et al. 2019; an alternative seq2seq model parameteriza-tion). Hyper-parameter selection details are given in the Appendix C. Unless otherwise stated, we use  = 0 and do not fine-tune L after initialization. Task The Colors sequence translation task (see Appendix A for full dataset) was developed to measure human inductive biases in sequence-to-sequence learning problems. It poses an extreme test of low-resource learning for neural sequence models: it has only 14 training examples that combine four named colors and three composition operations that perform concatenation, repetition and wrapping. Liu et al. (2020) solve this dataset with a symbolic stack machine; to the best of our knowledge, our approach is the first pure neural sequence model to obtain non-trivial accuracy. Results Both the Simple and IBMM2 initializers produce a lexicon that maps only color words to colors. Both, combined with the lexical translation mechanism, obtain an average test accuracy of 79% across 16 runs, nearly matching the human accuracy of 81% reported by Lake et al. (2019). The two test examples most frequently predicted incorrectly require generalization to longer sequences than seen during training. More details (includ-ing example-level model and human accuracies) are presented in the appendix Appendix A). These results show that LSTMs are quite effective at learning systematic sequence transformation rules from  3 examples per function word when equipped with lexical translations. Generalization to longer sequences remains as an important challenge for future work. Task SCAN (Lake and Baroni, 2018) is a larger collection of tests of systematic generalization that pair synthetic English commands (e.g. turn left twice and jump ) to action sequences (e.g. LTURN LTURN IJUMP ) as shown in Table 1. Following previous work, we focus on the jump and around right splits, each of which features roughly 15,000 training examples, and evaluate models' ability to perform 1-shot learning of new primitives ( jump ) and zero-shot interpretation of composition rules ( around right ). While these tasks are now solved by a number of specialized approaches, they remain a challenge for conventional neural sequence models, and an important benchmark for new models. Results In the jump split, all initializers improve significantly over the base LSTM when combined with lexical translation. Most methods achieve 99% accuracy at least once across seeds. These results are slightly behind GECA (in which all runs succeed) but ahead of SynAtt. 3 Again, they show that lexicon learning is effective for systematic generalization, and that simple initializers (PMI and Simple) outperform complex ones. Task COGS (Compositional Generalization for Semantic Parsing; Kim and Linzen 2020) is an automatically generated English-language semantic parsing dataset that tests systematic generalization in learning language-to-logical-form mappings. It includes 24155 training examples. Compared to the Colors and SCAN datasets, it has a larger vocabulary (876 tokens) and finer-grained inventory of syntactic generalization tests (Table 3). Results Notably, because some tokens appear in both inputs and logical forms in the COGS 3 SynAtt results here are lower than reported in the original paper, which discarded runs with a test accuracy of 0%. task, even a standard sequence-to-sequence model with copying significantly outperforms the baseline models in the original work of Kim and Linzen (2020), solving most tests of generalization over syntactic roles for nouns (but performing worse at generalizations over verbs, including passive and dative alternations). As above, the lexical translation mechanism (with any of the proposed initializers) provides further improvements, mostly for verbs that baselines model incorrectly (Table 3). Task To demonstrate that this approach is useful beyond synthetic tests of generalization, we evaluate it on a low-resource EnglishChinese translation task (the Tatoeba 4 dataset processed by Kelly 2021). For our experiments, we split the data randomly into 19222 training and 2402 test pairs. Results Results are shown in Table 4. Models with a lexical translation mechanism obtain modest improvements (up to 1.5 BLEU) over the baseline. Notably, if we restrict evaluation to test sentences 4 https://tatoeba.org/ ENG-CHN full 1-shot LSTM 24.18  0 . featuring English words that appeared only once in the training set, BLEU improves by more than 2 points, demonstrating that this approach is particularly effective at one-shot word learning (or fast mapping ; Carey and Bartlett 1978). Fig. 2 shows an example from this dataset, in which the model learns to reliably translate Saturn from a single training example. GECA, which makes specific generative assumptions about data distributions, does not generalize to a more realistic low resource MT problem. However, the lexical translation mechanism remains effective in natural tasks with large vocabularies and complex grammars. In all the experiments above, the lexicon was dis-cretized (  = 0 ) and frozen prior to training. In this final section, we revisit that decision, evaluating whether the parameter L can be learned from scratch, or effectively fine-tuned along with decoder parameters. Experiments in this section focus on the COGS dataset. Offline initialization of the lexicon is crucial. Rather than initializing L using any of the algorithms described in Section 3, we initialized L to a uniform distribution for each word and optimized COGS LSTM 0.51  0 . it during training. This improves over the base LSTM ( Uniform in Table 5), but performs significantly worse than pre-learned lexicons. Benefits from fine-tuning are minimal. We first increased the temperature parameter  to 0.1 (pro-viding a soft lexicon); this gave a 1% improvement on COGS (Table 5. Soft ). Finally, we updated this soft initialization via gradient descent; this provided no further improvement (Table 5, Learned ). One important feature of COGS (and other tests of compositional generalization) is perfect training accuracy is easily achieved; thus, there is little pressure on models to learn generalizable lexicons. This pressure must instead come from inductive bias in the initializer. We have described a lexical translation mechanism for representing token-level translation rules in neural sequence models. We have additionally described a simple initialization scheme for this lexicon that outperforms a variety of existing algorithms. Together, lexical translation and proper initialization enable neural sequence models to solve a diverse set of tasksincluding semantic parsing and machine translationthat require 1-shot word learning and 0-shot compositional generalization. Future work might focus on generalization to longer sequences, learning of atomic but non-concatenative translation rules, and online lexicon learning in situated contexts. This work was supported by the Machine-LearningApplications initiative at MIT CSAIL and the MITIBM Watson AI lab. Computing resources were provided by a gift from NVIDIA through the NVAIL program and by the Lincoln Laboratory Supercloud.
abstain abstain abstain objective result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result objective result result objective objective objective objective abstain result other other other other other method other other other other method abstain other other objective other other other objective objective other objective method method method method other other other other other abstain method method other objective other method other other other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other abstain method abstain abstain other abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other other