Interactive, Web-based Platform for “Big” Transportation Data Integration and Analytics

An exponential growth in various transportation data streams has brought new opportunities and challenges in the realm of transportation data warehousing. Increased data has the potential to improve planning, monitoring, prediction, and management of transportation systems but only if the manipulation of such gigantic datasets could be automated efficiently. With increasing demand for modern data warehousing, there has been a significant growth in commercial and open-source tools. The current paper presents a completely open-sourced, web-based platform that leverages recent advances in big data to efficiently process multiple streams of transportation data and deploying a variety of applications that will enable transportation agencies to make practical, data-driven decisions. Using a Hadoop and Spark cluster, and the generation of Graphical Processing Units (GPUs), the developed platform is able to generate responses to different forms of user queries at a much faster rate compared to traditional data warehouses. A CPU-GPU architecture is proposed to enable large datasets to enable both visualization and analytics to be carried out seamlessly on a web browser. The platform has two main components: a data center that provides the capacity of storing large, heterogenous datasets, and an applications development center that enables users to visualize and analyze large datasets on a web browser. The developed platform is fast, taking approximately fractions of a seconds to run complex queries on probe, crash, detector and transit datasets. Its interactive visualization applications can render responses to visual queries at a rate of about 600 milliseconds per 10 million rows.

• Record URL:
  https://annualmeeting.mytrb.org/OnlineProgram/Details/15463
• Supplemental Notes:
  • This paper was sponsored by TRB committee AED20 Standing Committee on Urban Transportation Data and Information Systems.
• Corporate Authors:

  Transportation Research Board

  ,    
• Authors:
  • Shu, Xiaofan
  • Adu-Gyamfi, Yaw
  • Sun, Carlos
  • 0000-0002-8857-9648
  • Edara, Praveen
  • 0000-0003-2707-642X
• Conference:
  • Transportation Research Board 100th Annual Meeting
  • Location: Washington DC, United States
  • Date: 2021-1-5 to 2021-1-29
• Date: 2021

Language

• English

Media Info

• Media Type: Digital/other
• Features: Figures; References; Tables;
• Pagination: 19p

Subject/Index Terms

• TRT Terms: Data analysis; Data files; Data warehouses; Interactive computer systems; Transportation; Web applications
• Subject Areas: Data and Information Technology; Transportation (General);

Filing Info

• Accession Number: 01764139
• Record Type: Publication
• Report/Paper Numbers: TRBAM-21-01195
• Files: TRIS, TRB, ATRI
• Created Date: Feb 4 2021 11:00AM