  Drones equipped with cameras can significantly enhance human ability to
perceive the world because of their remarkable maneuverability in 3D space.
Ironically, object detection for drones has always been conducted in the 2D
image space, which fundamentally limits their ability to understand 3D scenes.
Furthermore, existing 3D object detection methods developed for autonomous
driving cannot be directly applied to drones due to the lack of deformation
modeling, which is essential for the distant aerial perspective with sensitive
distortion and small objects. To fill the gap, this work proposes a dual-view
detection system named DVDET to achieve aerial monocular object detection in
both the 2D image space and the 3D physical space. To address the severe view
deformation issue, we propose a novel trainable geo-deformable transformation
module that can properly warp information from the drone's perspective to the
BEV. Compared to the monocular methods for cars, our transformation includes a
learnable deformable network for explicitly revising the severe deviation. To
address the dataset challenge, we propose a new large-scale simulation dataset
named AM3D-Sim, generated by the co-simulation of AirSIM and CARLA, and a new
real-world aerial dataset named AM3D-Real, collected by DJI Matrice 300 RTK, in
both datasets, high-quality annotations for 3D object detection are provided.
Extensive experiments show that i) aerial monocular 3D object detection is
feasible; ii) the model pre-trained on the simulation dataset benefits
real-world performance, and iii) DVDET also benefits monocular 3D object
detection for cars. To encourage more researchers to investigate this area, we
will release the dataset and related code in
https://sjtu-magic.github.io/dataset/AM3D/.

Computer Vision and Pattern Recognition;Artificial Intelligence