We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research. Where existing work often compares against random or majority class baselines, we argue that unimodal approaches better capture and reflect dataset biases and therefore provide an important comparison when assessing the performance of multimodal techniques. We present unimodal ablations on three recent datasets in visual navigation and QA, seeing an up to 29% absolute gain in performance over published baselines. All datasets have biases. Baselines should capture these regularities so that outperforming them indicates a model is actually solving a task. In multimodal domains, bias can occur in any subset of the modalities. To address this, we argue it is not sufficient for researchers to provide random or majority class baselines; instead we recommend presenting results for unimodal models. We investigate visual navigation and question answering tasks, where agents move through simulated environments using egocentric (first person) vision. We find that unimodal ablations (e.g., language only) in these seemingly multimodal tasks can outperform corresponding full models (  4.1). This work extends observations made in both the Computer Vision (Goyal et al., 2018; Cirik et al., 2018) and Natural Language (Mudrakarta et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Gururangan et al., 2018; Kaushik and Lipton, 2018) communities that complex models often perform well by fitting to simple, unintended correlations in the data, bypassing the complex grounding and reasoning that experimenters hoped was necessary for their tasks. We ablate models from three recent papers: (1) navigation (Figure 1) using images of real homes paired with crowdsourced language descriptions (Anderson et al., 2018); and (2, 3) navigation and egocentric question answering (Gordon et al., 2018; Das et al., 2018a) in simulation with synthetic questions. We find that unimodal ablations often outperform the baselines that accompany these tasks. Recommendation for Best Practices: Our find-ings show that in the new space of visual navigation and egocentric QA, all modalities, even an agent's action history, are strongly informative. Therefore, while many papers ablate either language or vision, new results should ablate both . Such baselines expose possible gains from unimodal biases in multimodal datasets irrespective of training and architecture details. In the visual navigation and egocentric question answering tasks, at each timestep an agent receives an observation and produces an action. Actions can move the agent to a new location or heading forward turn left turn right tilt up tilt down end start f o r w a r d t u r n l e f t t u r n r i g h t t i l t u p t i l t d o w n e n d s t a r t C o n d i t i o n a l M a r g i n a l .36 .44 .43 .54 .54 .393 .255 .257 .22 .22 .16 .071 .012 .012 .001 .00 .00 .00 .00 .00 .00 .00 .02 .02 .01 .01 .01 .01 Figure 2: P ( act = col | prev = row ) and marginal action distributions in Matterport training. Peaked distributions enable agents to memorize simple rules like not turning left immediately after turning right, or moving forward an average number of steps. (e.g., turn left ), or answer questions (e.g., answer brown' ). At timestep t , a multimodal model M takes in a visual input V t and language question or navigation command L to predict the next action a t . The navigation models we examine also take in their action from the previous timestep, a t  1 , and minimally sensed' world information W specifying which actions are available (e.g., that forward is unavailable if the agent is facing a wall). In each benchmark, M corresponds to the au-thor's released code and training paradigm. In addition to their full model, we evaluate the role of each input modality by removing those inputs and replacing them with zero vectors. Formally, we define the full model and three ablations: Full Model is M ( V t , L , a t  1 ; W ) (2) A is M ( (cid:126) 0 , (cid:126) 0 , a t  1 ; W ) (3) A + V is M ( V t , (cid:126) 0 , a t  1 ; W ) (4) A + L is M ( (cid:126) 0 , L , a t  1 ; W ) (5) corresponding to models with access to A ction inputs, V ision inputs, and L anguage inputs. These ablations preserve the architecture and number of parameters of M by changing only its inputs. We evaluate on navigation and question answering tasks across three benchmark datasets: Matterport Room-to-Room (no question answering compo-nent), and IQUAD V1 and EQA (question answering that requires navigating to the relevant scene in the environment) (Anderson et al., 2018; Gordon et al., 2018; Das et al., 2018a). We divide the latter two into separate navigation and question answering components. We then train and evaluate models separately per subtask to analyze accuracy. An agent is given a route in English and navigates through a discretized map to the specified destination (Anderson et al., 2018). This task includes high fidelity visual inputs and crowdsourced natural language routes. Published Full Model: At each timestep an LSTM decoder uses a ResNet-encoded image V t and previous action a t  1 to attend over the states of an LSTM language encoder ( L ) to predict navigation action a t (seen in Figure 2). Published Baseline: The agent chooses a random direction and takes up to five forward actions, turning right when no forward action is available. IQUAD V1 (Gordon et al., 2018) contains three question types: existence (e.g., Is there a ...? ), counting (e.g., How many ...? ) where the answer ranges from 0 to 3, and spatial relation: (e.g., Is there a ... in the ...? ). The data was constructed via randomly generated configurations to weaken majority class baselines (Figure 3). To evaluate the navigation subtask, we introduce a new THOR-Nav benchmark. 1 The agent is placed in a random location in the room and must approach one of fridge, garbage can, or microwave in response to a natural language question. Although we use the same full model as Gordon et al. (2018), our QA results are not directly comparable. In particular, Gordon et al. (2018) do not quantify the effectiveness of the QA component independent of the scene exploration (i.e. navigation and interaction). To remove the scene explo-1 Formed from a subset of IQUAD V1 questions. ration steps of Gordon et al. (2018), we provide a complete ground-truth view of the environment. 2 We use ground-truth rather than YOLO (Redmon et al., 2016) due to speed constraints. Nav Full Model: The image and ground-truth semantic segmentation mask V t , tiled question L , and previous action a t  1 are encoded via a CNN which outputs a distribution over actions. Optimal actions are learned via teacher forcing. Nav Baseline: The agent executes 100 randomly chosen navigation actions then terminates. In AI2THOR (Kolve et al., 2017), none of the kitchens span more than 5 meters. With a step-size of 0.25 meters, we observed that 100 actions was significantly shorter than the shortest path length. Published QA Full Model: The question encoding L is tiled and concatenated with a top-down view of the ground truth location of all objects in the scene V . This is fed into several convolutions, a spatial sum, and a final fully connected layer which outputs a likelihood for each answer. EQA (Das et al., 2018a) questions are programmatically generated to refer to a single, unambiguous object for a specific environment, and are fil-tered to avoid easy questions (e.g., What room is the bathtub in? ). At evaluation, an agent is placed a fixed number of actions away from the object. Published Nav Full Model: At each timestep, a planner LSTM takes in a CNN encoded image V t , LSTM encoded question L , and the previous action a t  1 and emits an action a t . The action is executed in the environment, and then a lower-level controller LSTM continues to take in new vision observations and a t , either repeating a t again or returning control to the planner. Published Nav Baseline: This baseline model is trained and evaluated with the same inputs as the full model, but does not pass control to a lower-level controller, instead predicting a new action using the planner LSTM at each timestep (i.e., no hierarchical control). Das et al. (2018a) name this baseline LSTM+Question . 2 This approximates the agent having visited every possible location, interacted with all possible objects, and looked in all possible directions before answering. Published QA Full Model: Given the last five image encodings along the gold standard navigation trajectory, V t  4 . . . V t , and the question encoding L , image-question similarities are calculated via a dot product and converted via attention weights to a summary weight  V , which is concatenated with L and used to predict the answer. Das et al. (2018a) name this oracle-navigation model ShortestPath+VQA . QA Baseline: Das et al. (2018a) provide no explicit baseline for the VQA component alone. We use a majority class baseline inspired by the data's entropy based filtering. Across all benchmarks, unimodal baselines outperform baseline models used in or derived from the original works. Navigating unseen environments, these unimodal ablations outperform their corresponding full models on the Matterport (ab-solute  2 . 5 % success rate) and EQA (  0 . 06 m distance to target). We evaluate our ablation baselines on Matterport, 3 THOR-Nav, and EQA (Table 1), 4 and discover that some unimodal ablations outperform their corresponding full models. For Matterport and THOR-Nav, success rate is defined by proximity to the target. For EQA, we measure absolute distance from the target in meters. Unimodal Performance: Across Matterport, THOR-Nav, and EQA, either A + V or A + L 3 We report on Matterport-validation since this allows comparing Seen versus Unseen house performance. 4 For consistency with THOR-Nav and EQA, we here evaluate Matterport using teacher forcing . achieves better performance than existing baselines. In Matterport, the A + L ablation performs better than the Full Model in unseen environments. The diverse scenes in this simulator may render the vision signal more noisy than helpful in previously unseen environments. The A + V model in THOR-Nav and EQA is able to latch onto dataset biases in scene structure to navigate better than chance (for IQA), and the nonhierarchical baseline (in EQA). In EQA, A + V also outperforms the Full Model; 5 the latent information about navigation from questions may be too distant for the model to infer. The agent with access only to its action history ( A ) outperforms the baseline agent in Matterport and THOR-Nav environments, suggesting it learns navigation correlations that are not captured by simple random actions (THOR-Nav) or programmatic walks away from the starting position (Mat-terport). Minimal sensing (which actions are available, W ) coupled with the topological biases in trajectories (Figure 2), help this nearly zero-input agent outperform existing baselines. 6 Matterport Teacher vs Student forcing With teacher forcing, at each timestep the navigation agent takes the gold-standard action regardless of what action it predicted, meaning it only sees steps along gold-standard trajectories. This paradigm is used to train the navigation agent in THOR-Nav and EQA. Under student forcing, the agent samples the action to take from its predictions, and loss is computed at each time step against the action that would have put the agent on the shortest 5 EQA full & baseline model performances do not exactly match those in Das et al. (2018a) because we use the expanded data updated by the authors https://github. com/facebookresearch/EmbodiedQA/ . 6 This learned agent begins to relate to work in minimally sensing robotics (O'Kane and LaValle, 2006). path to the goal. Thus, the agent sees more of the scene, but can take more training iterations to learn to move to the goal. Table 2 gives the highest validation success rates across all epochs achieved in Matterport by models trained using student forcing. The unimodal ablations show that the Full Model, possibly because with more exploration and more training episodes, is better able to align the vision and language signals, enabling generalization in unseen environments that fails with teacher forcing. EQA Navigation Variants Table 3 gives the average final distance from the target ( d T , used as the metric in Table 1) and average minimum distance from target achieved along the path ( d min ) during EQA episodes for agents starting 10, 30, and 50 actions away from the target in the EQA navigation task. At 10 actions away, the unimodal ablations tend to outperform the full model on both metrics, possibly due to the shorter length of the episodes (less data to train the joint parame-ters). The A + V ablation performs best among the ablations, and ties with or outperforms the Full Model in all but one setting, suggesting that the EQA Full Model is not taking advantage of language information under any variant. We evaluate our ablation baselines on IQUAD V1 and EQA, reporting top-1 QA accuracy (Table 4) given gold standard navigation information as V . These decoupled QA models do not take in a previous action, so we do not consider AONLY ablations for this task. lations perform nearly at chance. 7 The VONLY model with access to the locations of all scene objects only improves by 2% over random guessing. For EQA, single modality models perform significantly better than the majority class baseline. The vision-only model is able to identify salient colors and basic room features that allow it to reduce the likely set of answers to an unknown question. The language only models achieve nearly 50%, suggesting that despite the entropy filtering in Das et al. (2018a) each question has one answer that is as likely as all other answers combined (e.g. 50% of the answers for What color is the bathtub? are grey , and other examples in Figure 4). Historically, semantic parsing was used to map natural language instructions to visual navigation in simulation environments (Chen and Mooney, 2011; MacMahon et al., 2006). Modern approaches use neural architectures to map natural language to the (simulated) world and execute actions (Paxton et al., 2019; Chen et al., 2018; Nguyen et al., 2018; Blukis et al., 2018; Fried et al., 2018; Mei et al., 2016). In visual question answering (VQA) (Antol et al., 2015; Hudson and Manning, 2019) and visual commonsense reasoning (VCR) (Zellers et al., 2019), input images are accompanied with natural language questions. Given the question, egocentric QA requires an agent to navigate and interact with the world to gather the relevant information to answer the question. In both cases, end-to-end neural architectures make progress on these tasks. For language annotations, task design, diffi-culty, and annotator pay can introduce unintended artifacts which can be exploited by models to cheat on otherwise complex tasks (Glockner 7 Majority class and chance for IQUAD V1 both achieve 50%, 50%, 25% when conditioned on question type; our Baseline model achieves the average of these. F i n a l I m a g e Question V Only L Only Maj Class Full Model Answer What room is the iron located in? What color is the dresser? What color is the loudspeaker ? What room is the fruit bowl located in? Brown Kitchen Brown Kitchen Brown Green Living Room Kitchen Brown Bathroom Brown Kitchen Brown Brown Brown Brown Brown Bathroom Brown Kitchen What color is the bathtub  ? Grey Bathroom Grey Brown Grey Figure 4: Qualitative results on the EQA task. The language only model can pick out the most likely answer for a question. The vision only model finds salient color and room features, but is unaware of the question. et al., 2018; Poliak et al., 2018). Such issues also occur in multimodal data like VQA (Goyal et al., 2018), where models can answer correctly without looking at the image. In image captioning, work has shown competitive models relying only on nearest-neighbor lookups (Devlin et al., 2015) as well as exposed misalignment between caption relevance and text-based metrics (Rohrbach et al., 2018). Our unimodal ablations of visual navigation and QA benchmarks uncover similar biases, which deep architectures are quick to exploit. In this work, we introduce an evaluation framework and perform the missing analysis from several new datasets. While new state-of-the-art models are being introduced for several of these domains (e.g., Matterport: (Ma et al., 2019a; Ke et al., 2019; Wang et al., 2019; Ma et al., 2019b; Tan et al., 2019; Fried et al., 2018), and EQA: (Das et al., 2018b)), they lack informative, individual unimodal ablations (i.e., ablating both language and vision) of the proposed models. We find a performance gap between baselines used in or derived from the benchmarks examined in this paper and unimodal models, with unimodal models outperforming those baselines across all benchmarks. These unimodal models can even outperform their multimodal counterparts. In light of this, we recommend all future work include unimodal ablations of proposed multimodal models to vet and highlight their learned representations. This work was supported by NSF IIS-1524371, 1703166, NRI-1637479, IIS-1338054, 1652052, ONR N00014-13-1-0720, and the DARPA CwC program through ARO (W911NF-15-1-0543).
objective method result abstain abstain abstain result objective result other abstain abstain result objective abstain abstain abstain other abstain abstain abstain method abstain method method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other objective objective other result abstain objective other