STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenes

IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, pp. 19576–19585

Abstract

Accurately detecting and tracking pedestrians in 3D space is challenging due to large variations in rotations, poses and scales. The situation becomes even worse for dense crowds with severe occlusions. However, existing benchmarks either only provide 2D annotations, or have limited 3D annotations with low-density pedestrian distribution, making it difficult to build a reliable pedestrian perception system especially in crowded scenes. To better evaluate pedestrian perception algorithms in crowded scenarios, we introduce a large-scale multimodal dataset, STCrowd. Specifically, in STCrowd, there are a total of 219 K pedestrian instances and 20 persons per frame on average, with various levels of occlusion. We provide synchronized LiDAR point clouds and camera images as well as their corresponding 3D labels and joint IDs. STCrowd can be used for various tasks, including LiDAR-only, image-only, and sensor-fusion based pedestrian detection and tracking. We provide baselines for most of the tasks. In addition, considering the property of sparse global distribution and density-varying local distribution of pedestrians, we further propose a novel method, Density-aware Hierarchical heatmap Aggregation (DHA), to enhance pedestrian perception in crowded scenes. Extensive experiments show that our new method achieves state-of-the-art performance for pedestrian detection on various datasets. https://github.com/4DVLab/STCrowd.git.

Authors 10

1. Peishan Cong

   ShanghaiTech University

   Affiliation as printed

   ShanghaiTech University

2. Xinge Zhu

   Chinese University of Hong Kong

   Affiliation as printed

   The Chinese University of Hong Kong

3. Feng Qiao Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

4. Yiming Ren

   ShanghaiTech University

   Affiliation as printed

   ShanghaiTech University

5. Xidong Peng

   ShanghaiTech University

   Affiliation as printed

   ShanghaiTech University

6. Yuenan Hou

   Shanghai Artificial Intelligence Laboratory

   Affiliation as printed

   Shanghai AI Laboratory

7. Lan Xu

   ShanghaiTech University

   Affiliation as printed

   ShanghaiTech University

   Shanghai Engineering Research Center of Intelligent Vision and Imaging

8. Ruigang Yang

   University of Kentucky

   Affiliation as printed

   University of Kentucky

9. Dinesh Manocha

   University of Maryland, College Park

   Affiliation as printed

   University of Maryland at College Park

10. Yuexin Ma

    ShanghaiTech University

    Affiliation as printed

    ShanghaiTech University

    Shanghai Engineering Research Center of Intelligent Vision and Imaging