Integrated Modeling for Road Condition Prediction — Phase 3 Evaluation Report
Chapter 4. Findings
The purpose of the evaluation is to explore whether Integrated Modeling for Road Condition Prediction (IMRCP) had an impact on the Kansas City Scout (KC Scout) operations and to assess whether the information was useful to the KC Scout operators and supervisors. The key questions guiding the evaluation data collection and analyses are: (1) Did IMRCP have an operational impact? and (2) Did the users consider the IMRCP information useful? To explore whether IMRCP had an impact, the evaluation investigated the accuracy of IMRCP speeds and speed forecasts. To investigate whether IMRCP information was useful, KC Scout operators and supervisors were interviewed to obtain their insights and perspectives. The findings in this section describe the outcomes of investigating speed and speed forecast accuracy and operator and supervisor perceptions of IMRCP’s operational impact and usefulness.
Did Integrated Modeling for Road Condition Prediction Have an Operational Impact?
This evaluation found that IMRCP had minimal operational impact during the 2018–2019 and 2019–2020 winter driving seasons. However, operators did report that they often referred to IMRCP for weather forecast information prior to weather events. They stated that during normal day operations, they relied on existing KC Scout tools, applications, and information sources to monitor real-time traffic operations. Consequently, IMRCP speeds and speed forecasts were basically unused by the operators.
Analysis of the Traffic Estimation and Prediction System (TrEPS) and machine learning-based prediction (MLP) speeds and speed forecasts at three locations during seven winter days revealed that the two IMRCP models produced speeds and speed forecasts of varying accuracy. Three types of analyses for each location and date were completed:
Looking at the accuracy of historical predicted speeds, TrEPS speed data were found to be problematic on several dates, sometimes with several hours of predicted speeds considerably different (over 20 miles per hour [mph]) from the detector speeds. The TrEPS model predicts speeds using several input parameters, such as changes to posted speed limit, roadway construction (or blockage) information, and detector data. If any of these inputs are inaccurate or missing, it may affect TrEPS ability to predict accurate speeds. For example, it appeared that speeds reported by TrEPS at Interstate 435 (I–435) eastbound (EB) at State Line Road were affected by a construction work zone that was not reported by KC Scout. The MLP speed data often appeared to be within about 5 mph of the detector speeds and were much less affected by the I––435 at State Line Road construction work zone changes, but occasionally had periods with large deviations (up to 20 mph) above or below the detector speeds.
Analysis of forecast speeds found that MLP and TrEPS speed forecasts both showed periods where the forecasts appeared to have large deviations (up to 20 mph) above and below the detector speeds. The deviations appeared to grow larger and for longer periods of time with longer-term forecasts. The MLP forecast speeds tended to more closely match the detector speeds than the TrEPS forecasts and the errors were examined in the forecast speed error analysis.
The analysis of speed forecast errors used three formulas to calculate and compare the speed forecast errors: absolute, relative absolute, and root mean square error.
The absolute error analysis measured the difference between the forecast speed and the actual detector speed. In general, when examining the 15–115-min speed forecasts:
The relative absolute error measured how large the absolute error was compared to the actual speed and provided the percent size of the error. In general, when examining the 15–115-min speed forecasts:
The root mean square error measured the spread (or concentration) between forecasts and actual speeds. In general, when examining the 15–115-min forecasts:
Did Users Consider IMRCP Information as Useful?
This evaluation found that KC Scout operators and supervisors liked the weather-related prediction components of IMRCP, but preferred to use existing tools for obtaining information about real-time traffic conditions and incidents. Although road and weather information were sometimes helpful, the operators did not rely on IMRCP to select more relevant operational strategies. A KC Scout representative stated that IMRCP might be a helpful tool for planning and assessing winter maintenance response efforts and emergency operation staffing decisions.
The operators stated they accessed IMRCP when weather events were approaching, and found the weather information helpful. They mentioned that IMRCP was useful for both rain and winter weather, but more so for when winter weather was expected. The operators used it less during heavy rain or flooding events. Otherwise, the operators tended to not use IMRCP as part of their normal daily routines.
United States Department of Transportation - Federal Highway Administration