  As a vital step toward the intelligent agent, Action understanding matters
for intelligent agents and has attracted long-term attention. It can be formed
as the mapping from the action physical space to the semantic space. Typically,
researchers built action datasets according to idiosyncratic choices to define
classes and push the envelope of benchmarks respectively. Thus, datasets are
incompatible with each other like "Isolated Islands" due to semantic gaps and
various class granularities, e.g., do housework in dataset A and wash plate in
dataset B. We argue that a more principled semantic space is an urgent need to
concentrate the community efforts and enable us to use all datasets together to
pursue generalizable action learning. To this end, we design a structured
action semantic space in view of verb taxonomy hierarchy and covering massive
actions. By aligning the classes of previous datasets to our semantic space, we
gather (image/video/skeleton/MoCap) datasets into a unified database in a
unified label system, i.e., bridging ``isolated islands'' into a "Pangea".
Accordingly, we propose a novel model mapping from the physical space to
semantic space to fully use Pangea. In extensive experiments, our new system
shows significant superiority, especially in transfer learning. Code and data
will be made publicly available.

Computer Vision and Pattern Recognition;Artificial Intelligence;Machine Learning