Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors. Recent advances in reinforcement learning and optimal control make it possible to design control algorithms on these latent representations, but the field still lacks a large-scale standard dataset for unified comparison. In this work, we present a large-scale dataset and evaluation framework for representation learning for the complex task of landing an airplane. We implement and compare several approaches to representation learning on this dataset in terms of the quality of simple supervised learning tasks and disentanglement scores. The resulting representations can be used for further tasks such as anomaly detection, optimal control, model-based reinforcement learning, and other applications.
Overview and contributions:
The authors present a newly collected dataset and evaluation framework for learning representations for landing an airplane. The dataset is collected from the X-Plane simulation environment and consists of 8011 landings, each landing consists of time series data from 1090 sensors. Their evaluation metric is a combination of disentanglement score, regression tasks, and failure classification. The authors test a combination of baseline models from basic autoencoders to dynamic actions-aware encoders. The writing is generally clear but I have doubts about the proposed evaluation metrics, experiments, and significance of the dataset (details below).
Strengths:
1. The task seems to be novel and complex. The authors have done a good job of collecting the dataset and ensuring that the data is clean and comprehensive.
2. The authors have performed a comprehensive job of evaluating the combinations of baseline models for their proposed task.
Weaknesses:
1. Table 4 on evaluation results, while comprehensive, lacks some explanation. The issue with MSE is that it is hard to interpret what these values mean. Specifically, how difficult is this task? How well can a human perform on this task? How well are the baselines doing relative to human-level performance, and is there room for improvement? The answers to these questions are important towards whether this new dataset will be a strong benchmark for representation learning.
2. There is less novelty in terms of the models presented for evaluation since they are composed of existing models. What are some state-of-the-art models for similar tasks, and do they constitute fair comparison?
Questions to authors:
1. Refer to weakness points 1 and 2.
2. What biases do you think might exist in the dataset during the collection process? How might these biases affect what the models learn, and how can they be mitigated?
3. How do you ensure that all sensors are active at all times and that all sensors provide useful information for predicting the label? Are there cases where the multisensor data is noisy in certain modalities or missing in other modalities? If so, what are some models that can remain robustness to noisy or missing modalities?
4. Why do you think disentangled representations will help? Sure, they have been generally shown to help learn more interpretable representations, and help in flexible generation from disentangled factors. But in terms of discriminative or generative performance on your newly proposed dataset, does learning disentangled representations help? What are some models that can learn effectively learn such disentangled representations?
Presentation improvements, typos, edits, style, missing references:
Section 3, line 7, 'with with a frequency' -> 'with'
I would suggest referring to some recent work on multimodal temporal fusion, such as "Memory Fusion Network for Multi-view Sequential Learning. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-Philippe Morency, AAAI 2018"