Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, the authors first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. They then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, the authors provide tables that summarize topics and methods. They also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

• Record URL:
  http://dx.doi.org/10.1109/TITS.2020.2972974
• Availability:
  • Find a library where document is available. Order URL: http://worldcat.org/oclc/41297384
• Supplemental Notes:
  • Copyright © 2021, IEEE.
• Authors:
  • Feng, Di
  • Haase-Schütz, Christian
  • Rosenbaum, Lars
  • Hertlein, Heinz
  • Gläser, Claudius
  • Timm, Fabian
  • Wiesbeck, Werner
  • Dietmayer, Klaus
• Publication Date: 2021-3

Language

• English

Media Info

• Media Type: Digital/other
• Features: Figures; References; Tables;
• Pagination: pp 1341-1360
• Serial:
  • IEEE Transactions on Intelligent Transportation Systems
  • Volume: 22
  • Issue Number: 3
  • Publisher: Institute of Electrical and Electronics Engineers (IEEE)
  • ISSN: 1524-9050
  • Serial URL: http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=6979

Subject/Index Terms

• TRT Terms: Automated vehicle control; Autonomous vehicles; Cameras; Data files; Data fusion; Detection and identification systems; Laser radar; Machine learning; Streetscape; Visual texture recognition
• Subject Areas: Data and Information Technology; Highways; Vehicles and Equipment;

Filing Info

• Accession Number: 01770475
• Record Type: Publication
• Files: TLIB, TRIS
• Created Date: Apr 26 2021 3:14PM