Analysis of Big Transportation Data for Better Infrastructure Management: A Case Study Using Very Large Weigh-in-Motion Data

With the advance of technology, the size and availability of data increase for all sectors including transportation. Similarly, availability and size of Weigh-in-Motion (WIM) data increase every day, which help the professionals to monitor the trucks in the network and observe trends in the weights and configurations of the vehicles. However, truck monitoring does not occur in near real-time due to difficulties in data collection and processing the data especially if it is from large number of stations. Moreover, the “traditional” database systems are either too slow in analyzing these big datasets or charge substantial licensing fees per utilized CPU core. Apache Spark is an open-source big data analytics and distributed computing framework and it can offer a solution to aforementioned problems. In this study, the authors demonstrate the potential benefits of using Spark and cloud computing in analyzing WIM data using one of the largest datasets in the transportation literature containing over 400 million vehicle records from the statewide WIM sites in New Jersey. First, a set of queries are chosen, which are mostly essential for exploratory analyzing of WIM data.  Utilizing another data source for overweight truck permits, more complex queries running on multiple datasets are also developed. Then, Spark is benchmarked against traditional databases using these queries. The performance of Spark on a computer cluster is also investigated with varying resource configurations. The results show that considerable computation time gains are possible in analyzing this large data using big data tools, especially on a computer cluster.

• Record URL:
  • https://annualmeeting.mytrb.org/OnlineProgram/Details/15491
• Supplemental Notes:
  • This paper was sponsored by TRB committee ACP70 Standing Committee on Highway Traffic Monitoring.
• Corporate Authors:

  Transportation Research Board

  ,    
• Authors:
  • Demiroluk, Sami
  • Ozbay, Kaan
  • Nassif, Hani
• Conference:
  • Transportation Research Board 100th Annual Meeting
  • Location: Washington DC, United States
  • Date: 2021-1-5 to 2021-1-29
• Date: 2021

Language

• English

Media Info

• Media Type: Web
• Features: Figures; References; Tables;
• Pagination: 17p

Subject/Index Terms

• TRT Terms: Case studies; Data analysis; Data collection; Infrastructure; Overweight loads; Trucks; Weigh in motion
• Subject Areas: Data and Information Technology; Highways;

Filing Info

• Accession Number: 01764249
• Record Type: Publication
• Report/Paper Numbers: TRBAM-21-03177
• Files: TRIS, TRB, ATRI
• Created Date: Feb 4 2021 11:00AM