Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency School of Computer Science, Carnegie Mellon University { pliang,yaochonl,yaohungt,rsalakhu,morency } @cs.cmu.edu Abstract Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various attention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a conditional factorization of the utterance into unimodal factors. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the optimal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer representations, our second baseline extends the first by factorizing into unimodal, bimodal, and trimodal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning. Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations (Streeck and Knapp, 1992). At the heart of many multimodal modeling tasks lies the challenge of learning rich representations of spoken utterances from multiple modalities (Papo et al., 2014). However, learning repre-* authors contributed equally sentations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information (Baltrusaitis et al., 2017). This challenging yet crucial research area has real-world applications in robotics (Mon-talvo et al., 2017; Noda et al., 2014), dialogue systems (Johnston et al., 2002; Rudnicky, 2005), intelligent tutoring systems (Mao and Li, 2012; Banda and Robinson, 2011; Pham and Wang, 2018), and healthcare diagnosis (Wentzel and van der Geest, 2016; Lisetti et al., 2003; Sonntag, 2017). Recent progress on multimodal representation learning has investigated various neural models that utilize one or more of attention, memory and recurrent components (Yang et al., 2017; Liang et al., 2018). There has also been a general trend of building more complicated models for improved performance. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a factorization of the utterance into unimodal factors conditioned on the joint embedding. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the utterance embedding. We derive a coordinate-ascent style algorithm (Wright, 2015) to learn the optimal multimodal embeddings under our model. We show that, under some assumptions, maximum likelihood estimation for the utterance embedding can be derived in closed form and is equivalent to computing a weighted average of the language, visual and acoustic features. Only a few linear transformation parameters need to be learned. In order to capture bimodal and trimodal representations, our second baseline extends the first one by assuming a factorization into unimodal, bimodal, and trimodal factors (Zadeh et al., 2017). To summarize, our simple baselines 1) consist primarily of linear functions, 2) have few parameters, and 3) can be approximately solved in a closed form solution. As a result, they demonstrate simplicity and efficiency during learning and inference. We perform a set of experiments across two tasks and datasets spanning multimodal personality traits recognition (Park et al., 2014) and multimodal sentiment analysis (Zadeh et al., 2016). Our proposed baseline models 1) achieve competitive performance on supervised multimodal learning, 2) improve upon classical deep autoencoders for semi-supervised multimodal learning, and 3) are up to 10 times faster during inference. Overall, we believe that our baseline models offer new benchmarks for future multimodal research. We provide a review of sentence embeddings , multimodal utterance embeddings , and strong baselines Sentence embeddings are crucial for down-stream tasks such as document classification, opinion analysis, and machine translation. With the advent of deep neural networks, multiple network designs such as Recurrent Neural Networks (RNNs) (Rumelhart et al., 1986), Long-Short Term Memory networks (LSTMs) (Hochre-iter and Schmidhuber, 1997), Temporal Convolutional Networks (Bai et al., 2018), and the Transformer (Vaswani et al., 2017) have been proposed and achieve superior performance. However, more training data is required for larger models (Pe-ters et al., 2018). In light of this challenge, researchers have started to leverage unsupervised training objectives to learn sentence embedding which showed state-of-the-art performance across multiple tasks (Devlin et al., 2018). In our paper, we go beyond unimodal language-based sentence embeddings and consider multimodal spoken utterances where additional information from the nonverbal behaviors is crucial to infer speaker intent. Learning multimodal utterance embeddings brings a new level of complexity as it requires modeling both intra-modal and inter-modal interactions (Liang et al., 2018). Previous approaches have explored variants of graphical models and neural networks for multimodal data. RNNs (Elman, 1990; Jain and Medsker, 1999), LSTMs (Hochreiter and Schmidhuber, 1997), and convolutional neural networks (Krizhevsky et al., 2012) have been extended for multimodal settings (Rajagopalan et al., 2016; Lee et al., 2018). Experiments on more advanced networks suggested that encouraging correlation between modalities (Yang et al., 2017), enforcing disentanglement on multimodal representations (Tsai et al., 2018), and using attention to weight modalities (Gulrajani et al., 2017) led to better performing multimodal representations. In our paper, we present a new perspective on learning multimodal utterance embeddings by assuming a conditional factorization over the language, visual and acoustic features. Our simple but strong baseline models offer an alternative approach that is extremely fast and competitive on both supervised and semi-supervised prediction tasks. A recent trend in NLP research has been geared towards building simple but strong baselines (Arora et al., 2017; Shen et al., 2018; Wieting and Kiela, 2019; Denkowski and Neubig, 2017). The effectiveness of these baselines indicate that complicated network components are not always required. For example, Arora et al. (2017) constructed sentence embeddings from weighted combinations of word embeddings which requires no trainable parameters yet generalizes well to down-stream tasks. Shen et al. (2018) proposed parameter-free pooling operations on word embeddings for document classification, text sequence matching, and text tagging. Wieting and Kiela (2019) discovered that random sentence encoders achieve competitive performance as compared to larger models that involve expensive training and tuning. Denkowski and Neubig (2017) emphasized the importance of choosing a basic neural machine translation model and carefully reporting the relative gains achieved by the proposed techniques. Authors in other domains have also highlighted the importance of developing strong baselines (Lakshminarayanan et al., 2017; Sharif Razavian et al., 2014). To the best of our knowledge, our paper is the first to propose and evaluate strong, non-neural baselines for multimodal utterance embeddings. Suppose we are given video data where each utterance segment is denoted as s . Each segment contains individual words w in a sequence w , visual features v in a sequence v , and acoustic features a in a sequence a . We aim to learn a representation It doesn't give any insight or help Language: Visual: Acoustic: Gaussian likelihood Gaussian likelihood Log-linear likelihood =       =       =       MultimodalUtterance MultimodalUtteranceEmbedding PositionalEncodings PositionalEncodings Figure 1: Our baseline model assumes a factorization of the multimodal utterance into unimodal factors conditioned on the joint utterance embedding. Our model is related to the work done by Arora et al. (2016) and Arora et al. (2017). In the following, we first provide a brief review of their method. Given a sentence, Arora et al. (2016) aims to learn a sentence embedding c s . They do so by assuming that the probability of observing a word w t at time t is given by a log-linear word production model (Mnih and Hinton, 2007) with respect to c s : P [ w t  c s ] = exp ( v w t , c s ) Z c s , (1) where c s is the sentence embedding (context), v w t represents the word vector associated with word w t and Z c s =  w  V exp ( v w , c s ) is a normalizing constant over all words in the vocabulary. Given this posterior probability, the desired sentence embedding c s can be obtained by maximizing Equation (1) with respect to c s . Under some assumptions on c s , this maximization yields a closed-form solution which provides an efficient learning algorithm for sentence embeddings. Arora et al. (2017) further extends this model by introducing a smoothing term  to account for the production of frequent stop words or out of context words independent of the discourse vector. Given estimated unigram probabilities p ( w ) , the probability of a word at time t is given by P [ w t  c s ] = p ( w t ) + ( 1   ) exp ( v w t , c s ) Z c s . (2) Under this model with the additional hyperparameter  , we can still obtain a closed-form solution for the optimal c s . In this subsection, we outline our method for learning representations of multimodal utterances. An overview of our proposed baseline model is shown in Figure 1. Our method begins by assuming a factorization of the multimodal utterance into unimodal factors conditioned on the joint utterance embedding. Next, each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the utterance embedding. Finally, we incorporate positional encodings to represent temporal information in the features. We first present the details of our proposed baseline before deriving a coordinate ascent style optimization algorithm to learn utterance embeddings in our model. Unimodal Factorization: We use m s to represent the multimodal utterance embedding. To begin, we simplify the composition of m s by assuming that the segment s can be conditionally factorized into words ( w ), visual features ( v ), and acoustic features ( a ). Each factor is also associated with a temperature hyperparameter (  w ,  v ,  a ) that represents the contribution of each factor towards the multimodal utterance. The likelihood of a segment s given the embedding m s is therefore P [ s  m s ] = P [ w  m s ]  w ( P [ v  m s ])  v P [ a  m s ]  a =  w  w P [ w  m s ]  w  v  v P [ v  m s ]  v  a  a P [ a  m s ]  a . Choice of Likelihood Functions: As suggested by Arora et al. (2017), given m s , we model the probability of a word w using Equation (2). In order to analytically solve for m s , a lemma is introduced by Arora et al. (2016, 2017) which states that the partition function Z m s is concentrated around some constant Z (for all m s ). This lemma is also known as the self-normalizing phenomenon of log-linear models (Andreas and Klein, 2015; Andreas et al., 2015). We use the same assumption and treat Z m s t  Z for all m s . Unlike discrete text tokens, the visual features are continuous. We assume that the visual features are generated from an isotropic Gaussian distribution. In section 5.1, we visually analyze the distribution of the features for real world datasets and show that these likelihood modeling assumptions are indeed justified. The Gaussian distribution is parametrized by simple linear transformations W v , W v  R  v  m s  and b v , b v  R  v  : v  m s  N (  v ,  2 v ) , (4)  v = W v m s + b v , (5)  v = diag ( exp ( W v m s + b v )) . (6) Similarly, we also assume that the continuous acoustic features are generated from a different isotropic Gaussian distribution parametrized as: a  m s  N (  a ,  2 a ) , (7)  a = W  a m s + b  a , (8)  a = diag ( exp ( W a m s + b a )) . (9) Positional Encodings: Finally, we incorporate positional encodings (Vaswani et al., 2017) into the features to represent temporal information. We use d -dimensional positional encodings with entries: P E pos, 2 i = sin ( pos / 10000 2 i / d ) , (10) P E pos, 2 i + 1 = cos ( pos / 10000 2 i / d ) . where pos is the position (time step) and i  [ 1 , d ] indexes the dimension of the positional encodings. We call this resulting model Multimodal Baseline 1 ( MMB1 ). We define our objective function by the log-likelihood of the observed multimodal utterance s . The maximum likelihood estimator of the utterance embedding m s and the linear transformation parameters W and b can then be obtained by maximizing this objective L( m s , W , b ; s ) = log P [ s  m s ; W , b ] , (12) where we use W and b to denote all linear transformation parameters. Coordinate Ascent Style Algorithm: Since the objective (12) is not jointly convex in m s , W and b , we optimize by alternating between: 1) solving for m s given the parameters W and b at the current iterate, and 2) given m s , updating W and b using a gradient-based algorithm. This resembles the coordinate ascent optimization algorithm which maximizes the objective according to each coordinate separately (Tseng, 2001; Wright, 2015). Algorithm 1 presents our method for learning utterance embeddings. In the following sections, we describe how to solve for m s and update W and b . Solving for m s : We first derive an algorithm to solve for the optimal m s given the log likelihood objective in (12), and parameters W and b . m  s =  w  s  w w +  v  s ( W   v  v ( 1 )  ( 1 ) v + W   v  v ( 2 )  ( 2 ) v ) +  a  s ( W   a  a ( 1 )  ( 1 ) a + W   a  a ( 2 )  ( 2 ) a ) . (13) where the shifted visual and acoustic features are:  v ( 1 ) = v  b v ,  v ( 2 ) = ( v  b v )  ( v  b v ) , (14)  a ( 1 ) = a  b a ,  a ( 2 ) = ( a  b a )  ( a  b a ) , (15) where  denotes Hadamard (element-wise) product and the weights  's are given as follows:  w =  w ( 1   )/( Z ) p ( w ) + ( 1   )/( Z ) , (16)  ( 1 ) v = diag (  v exp ( 2 b v )) , (17)  ( 2 ) v = diag (  v exp ( 2 b v )   v ) , (18)  ( 1 ) a = diag (  a exp ( 2 b a )) , (19)  ( 2 ) a = diag (  a exp ( 2 b a )   a ) . (20) Proof. The proof is adapted from Arora et al. (2017) and involves computing the gradients  m s log P [ m s ]   . We express log P [ m s ] via a Taylor expansion approximation and we observe that log P [ m s ]  c +  m s , g  for a constant c and a vector g . Then, we can obtain m  s by computing arg max m  s L( m s , W , b ; s ) which yields a closed-form solution. Please refer to the supplementary material for proof details. Observe that the optimal embedding m  s is a weighted average of the word features w and the (shifted and transformed) visual and acoustic features,  v and  a . Our choice of a Gaussian likelihood for the visual and acoustic features introduces a squared term ( v  b v )( v  b v ) to account for the (cid:96) 2 distance present in the pdf. The transformation matrix W  transforms the visual and acoustic features into the multimodal embedding space. Regarding the weights  , note that: 1) the weights are proportional to the global temperatures  assigned to that modality, 2) the weights  w are inversely proportional to p ( w ) (rare words carry more weight), and 3) the weights  v and  a scale each feature dimension inversely by their magnitude. Updating W and b : To find the optimal linear transformation parameters W and b to maximize the objective in (12), we perform gradient-based optimization on W and b (in Algorithm 1 line 5-8). Proposition 1. [Updating W and b ] The gradients  W L( m s , W , b ) and  b L( m s , W , b ) , in each dimension, are:  W vij L( m s ,W ,b ) =  v tr [(   2 v ( v   v ))  m s j ] , (21)  W vij L( m s ,W ,b ) =   v 2 tr [(   2 v    2 v ( v   v )( v   v )    2 v )   vii m s j ] , (22)  b vi L( m s ,W ,b ) =  v tr [(   2 v ( v   v ))  ] , (23)  b vi L( m s ,W ,b ) =   v 2 tr [(   2 v    2 v ( v   v )( v   v )    2 v )   vii ] . (24) Proof. The proof involves differentiating the log likelihood of a multivariate Gaussian with respect to  and  before applying the chain rule to  = W  m s + b  and  = diag ( exp ( W  m s + b  )) . So far, we have assumed the utterance segment s can be independently factorized into unimodal features. In this subsection, we extend the setting to take account for bimodal and trimodal interactions. We adopt the idea of early-fusion (Srivas-tava and Salakhutdinov, 2012), which means the bimodal and trimodal interactions are captured by the concatenated features from different modalities. Specifically, we define our factorized model as: P [ s  m s ] = P [ w  m s ]  w P [ v  m s ]  v P [ a  m s ]  a P [( w  v )  m s ]  wv P [( w  a )  m s ]  wa P [( v  a )  m s ]  va P [( w  v  a )  m s ]  wva , (25) where  denotes vector concatenation for bimodal and trimodal features. Each of the individual probabilities factorize in the same way as Equation (3) (i.e. P [ a  m s ]  a =  a  a P [ a  m s ]  a ). Similar to baseline 1, we assume a log-linear likelihood (2) for P [ w  m s ] and a Gaussian likelihood (4) for all remaining terms. We call this Multimodal Baseline 2 ( MMB2 ). The optimization algorithm derived in section 3.4 can be easily extended to learn m s , W and b in Baseline 2. We again alternate between the 2 steps of 1) solving for m s given the parameters W and b at the current iterate, and 2) given m s , updating W and b using a gradient-based algorithm. Solving for m s : We state a result that derives the closed-form of m s given W and b : Corollary 1. [Solving for m s ] Assume that the optimal m s lies on the unit sphere (i.e.  m s  22 = 1 ). The closed-form (in Algorithm 1 line 4) for m s is: m  s =  w  w  w w +  v  v ( W   v  v ( 1 )  ( 1 ) v + W   v  v ( 2 )  ( 2 ) v ) +  a  a ( W   a  a ( 1 )  ( 1 ) a + W   a  a ( 2 )  ( 2 ) a ) +  f { w  v , w  a , v  a , w  v  a }  f  f ( W   f  f ( 1 )  ( 1 ) f + W   f  f ( 2 )  ( 2 ) f ) (26) where the shifted (and squared) visual features are:  v ( 1 ) = v  b v ,  v ( 2 ) = ( v  b v )  ( v  b v ) , (27) (and analogously for  f ( 1 ) ,  f ( 2 ) , f  { a, w  v, w  a, v  a, w  v  a } ). The weights  's are:  w =  w ( 1   )/( Z ) p ( w ) + ( 1   )/( Z ) , (28)  ( 1 ) v = diag (  v exp ( 2 b v )) , (29)  ( 2 ) v = diag (  v exp ( 2 b v )   v ) . (30) (and analogously for  ( 1 ) f ,  ( 2 ) f , f  { a, w  v, w  a, v  a, w  v  a } ). Updating W and b : The gradient equations for updating W and b are identical to those derived in Proposition 1, Equations (21-24). Given the optimal embeddings m s , we can now train a classifier from m s to labels y for multimodal prediction. m s can also be fine-tuned on labeled data (i.e. taking gradient descent steps to update m s with respect to the task-specific loss functions) to learn task-specific multimodal utterance representations. In our experiments, we use a fully connected neural network for our classifier. speaker traits recognition and multimodal sentiment analysis. The code for our experiments is released at https://github.com/ yaochie/multimodal-baselines , and all datasets for our experiments can be downloaded at https://github.com/A2Zadeh/ CMU-MultimodalSDK . All datasets consist of monologue videos where the speaker's intentions are conveyed through the language, language, visual and acoustic modalities. The multimodal features are described in the next subsection. Multimodal Speaker Traits Recognition involves recognizing speaker traits based on multimodal utterances. POM (Park et al., 2014) contains 903 videos each annotated for speaker traits: confi-dent (con), voice pleasant (voi), dominance (dom), vivid (viv), reserved (res), trusting (tru), relaxed (rel), outgoing (out), thorough (tho), nervous (ner), and humorous (hum). The abbreviations (inside parentheses) are used in the tables. Multimodal Sentiment Analysis involves analyzing speaker sentiment based on video content. Multimodal sentiment analysis extends conventional language-based sentiment analysis to a multimodal setup where both verbal and non-verbal signals contribute to the expression of sentiment. We use CMU-MOSI (Zadeh et al., 2016) which consists of 2199 opinion segments from online videos each annotated with sentiment from strongly negative ( 3 ) to strongly positive (+ 3 ) . GloVe word embeddings (Pennington et al., 2014), Facet (iMotions, 2017) and COVAREP (Degottex et al., 2014) are extracted for the language, visual and acoustic modalities respectively 1 . Forced alignment is performed using P2FA (Yuan and Liber-man, 2008) to obtain the exact utterance times of each word. The video and audio features are aligned by computing the expectation of their features over each word interval (Liang et al., 2018). For classification, we report multiclass classification accuracy A ( c ) where c denotes the number of classes and F1 score. For regression, we report Mean Absolute Error (MAE) and Pearson's correlation ( r ). For MAE lower values indicate better performance. For all remaining metrics, higher values indicate better performance. Before proceeding to the experimental results, we perform some sanity checks on our modeling assumptions. We plotted histograms of the visual and acoustic features in CMU-MOSI utterances to visually determine if they resemble a Gaussian distribution. From the plots in Figure 2, we observe that many of the features indeed converge approximately to a Gaussian distribution across the time 1 Details on feature extraction are in supplementary. steps in the utterance, justifying the parametrization for the visual and acoustic likelihood functions in our model. Our first set of experiments evaluates the performance of our baselines on two multimodal prediction tasks: multimodal sentiment analysis on CMU-MOSI and multimodal speaker traits recognition on POM. On CMU-MOSI (right side of Table 1), our model MMB2 performs competitively against many neural models including early fusion deep neural networks (Nojavanasghari et al., 2016), several variants of LSTMs (stacked, bidirectional etc.) (Hochreiter and Schmidhuber, 1997; Schuster and Paliwal, 1997), Multi-view LSTMs (Ra-jagopalan et al., 2016), and tensor product recurrent models (TFN) (Zadeh et al., 2017). For multimodal personality traits recognition on POM (left side of Table 1), our baseline is able to additionally outperform more complicated memory-based recurrent models such as MFN (Zadeh et al., 2018) on several metrics. We view this as an impressive achievement considering the simplicity of our model and the significantly fewer parameters that our model contains. As we will later show, our model's strong performance comes with the additional benefit of being significantly faster than the existing models. Our next set of experiments evaluates the performance of our proposed baseline models when there is limited labeled data. Intuitively, we expect our model to have a lower sample complexity since training our model involves learning fewer parameters. As a result, we hypothesize that our model will generalize better when there is limited amounts of labeled data as compared to larger neural models with a greater number of parameters. We test this hypothesis by evaluating the performance of our model on the CMU-MOSI dataset with only 40%, 60%, 80%, and 100% of the training labels. The remainder of the train set now consists of unlabeled data which is also used during training but in a semi-supervised fashion. We use the entire train set (both labeled and unlabeled data) for unsupervised learning of our multimodal embeddings before the embeddings are fine-tuned to predict the label using limited labeled data. A comparison is performed with two models that also learn embeddings from unlabeled multimodal utterances: 1) deep averaging autoencoder ( AE ) (Iyyer et al., 2015; Hinton and Salakhutdinov, 2006) which averages the temporal dimension before using a fully connected autoencoder to learn a latent embedding, and 2) sequence to sequence autoencoder ( seq2seq ) (Sutskever et al., 2014) which captures temporal information using a recurrent neural network encoder and decoder. For each of these models, an autoencoding model is used to learn embeddings on the entire training set (both labeled and unlabeled data) before the embeddings are fine-tuned to predict the label using limited la-Method Average Time (s) Inferences Per Second (IPS) DF 0.305 1850 EF-LSTM 0.022 31200 MV-LSTM 0.490 1400 BC-LSTM 0.210 3270 TFN 2.058 333 MFN 0.144 4760 MMB1 0.00163 421000 MMB2 0.00219 313000 Table 3: Average time taken for inference on the CMU-MOSI test set and Inferences Per Second (IPS) on a single Nvidia GeForce GTX 1080 Ti GPU, averaged over 5 trials. beled data. The validation and test sets remains unchanged for fair comparison. Under this semi-supervised setting, we show prediction results on the CMU-MOSI test set in Table 2. Empirically, we find that our model is able to outperform deep autoencoders and their recurrent variant. Our model remains strong and only suffers a drop in performance of about 3% (75.1%  72.9% binary accuracy) despite having access to only 40% of the labeled training data. To demonstrate another strength of our model, we compare the inference times of our model with existing baselines in Table 3. Our model achieves an inference per second (IPS) of more than 10 times the closest neural model (EF-LSTM). We attribute this speedup to our (approximate) closed form solution for m s as derived in Theorem 1 and Corollary 1, the small size of our model, as well as the fewer number of parameters (linear transformation parameters and classifier parameters) involved. To further motivate our design decisions, we test some ablations of our model: 1) we remove the modeling capabilities of the visual and acoustic modalities, instead modeling only the language modality, 2) we remove the positional encodings, and 3) we remove the fine tuning step. We provide these results in Table 4 and observe that each component is indeed important for our model. Although the text only model performs decently, incorporating visual and acoustic features under our modeling assumption improves performance. Our results also demonstrate the effectiveness of positional encodings and fine tuning without having to incorporate any additional learnable parameters. This paper proposed two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a factorization of the utterance into unimodal factors conditioned on the joint embedding while the second baseline extends the first by assuming a factorization into unimodal, bimodal, and trimodal factors. Both proposed models retain simplicity and efficiency during both learning and inference. From experiments across multimodal tasks and datasets, we show that our proposed baseline models: 1) display competitive performance on supervised multimodal prediction, 2) outperform classical deep autoencoders for semi-supervised multimodal prediction and 3) attain significant (10 times) speedup during inference. Overall, we believe that our strong baseline models provide new benchmarks for future research in multimodal learning. PPL and LM were partially supported by Sam-sung and NSF (Award 1750439). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Sam-sung and NSF, and no official endorsement should be inferred. YHT and RS were supported in part by the NSF IIS1763562, Office of Naval Research N000141812861, and Google focused award. We would also like to acknowledge NVIDIA's GPU support and the anonymous reviewers for their constructive comments on this paper.
abstain abstain abstain abstain objective abstain abstain result objective result objective abstain abstain abstain other abstain abstain objective abstain abstain abstain result abstain abstain objective abstain method objective objective method abstain abstain abstain abstain method abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective other other other other other other abstain abstain abstain method abstain abstain abstain method method abstain other method abstain abstain method abstain method abstain other method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method objective method method abstain method abstain abstain abstain abstain abstain other abstain abstain abstain method abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain result abstain result abstain abstain other result abstain result abstain abstain abstain abstain abstain abstain other other abstain abstain result result abstain abstain result abstain abstain abstain result result abstain objective abstain abstain objective objective other other other other