CHAPTER 3. CHANGES IN PERFORMANCE MEASURES

This chapter describes the changes to the performance measures reported in 2012. These changes are necessary to address gaps in performance measurement due to current program objectives and activities and recent advances in capability and technology for road weather management.

Gaps In Performance Measurement

Based on the current inventory of program activities, the team identified gaps in the previous performance measurement framework. Some of these gaps are described below.

Extent of Use and Adoption of Mobile Data-based Applications

Since the previous update, the Road Weather Management Program (RWMP), as part of the Connected Vehicle Research track, has been supporting the development of Vehicle to Infrastructure (V2I) applications that leverage mobile data and vehicle to infrastructure connectivity to support road weather management. Currently, the RWMP is supporting several development efforts for V2I applications including the Motorist Advisories and Warnings (MAW), Spot Weather Impact Warning (SWIW) applications for commercial vehicles, Enhanced Maintenance Decision Support Systems application (EMDSS). The WRTM program is also supporting several applications that use mobile data and remote connectivity in Wyoming, South Dakota, and Michigan. The aforementioned V2I applications are in research phases, and agencies' involvements in these efforts are captured through the research and development (R&D) performance measure. However, a gap remains in understanding how V2I and mobile data are being used by the States.

Climate Change/Extreme Weather/Resilience

The stakeholder community has expressed a great interest in managing extreme weather and improving the resiliency of operations, and the program has supported various stakeholder engagement activities on this topic through the American Association of State Highway and Transportation Officials (AASHTO) and Transportation Research Board (TRB). Currently, the program has not identified performance measures that effectively assess growth in this area at the State departments of transportation (DOT) and local levels. Research and guidance development is ongoing in this area but is still in early stages.

Section 1201 Rule Compliance

In 2015, State DOTs began reporting road condition data in compliance with the Section 1201 rule for appropriate areas and segments. While the rule does not specify how compliance should be achieved or the methods to report data, monitoring the self-reported rate of compliance with the rule requirements is a good measure to track timeliness of road condition availability in the nation.

Meteorological Assimilation Data Ingest System Transition

With the sunset of the Clarus program, the RWMP has been supporting the transition of the fixed and mobile Environmental Sensor Stations (ESS) data maintained by State DOTs to Meteorological Assimilation Data Ingest System (MADIS). The program is supporting the signing of data sharing agreements between State DOTs and National Oceanic and Atmospheric Administration (NOAA) and the integration of Clarus quality-checking algorithms into MADIS. To ensure the broad national scope of Clarus is transitioned, it is important for the program to track how many States are now connected to MADIS.

Expanding Partnerships

Starting with primarily the maintenance groups within State DOTs and a select group of weather/meteorological experts, the program has expanded its reach over the past decade to operations, technology, private sector, and the broader weather enterprise. While participation in stakeholder meetings is tracked, there is no performance measure that tracks the growth in partnerships as evidenced by the new groups involved in road weather management. For example, representatives from universities, private sector information service providers (e.g., Inrix, Waze), and connected vehicle experts have participated in recent stakeholder meetings.

Mainstreaming of Road Weather Management Programs

Throughout the past decade, an emphasis of the program has been to mainstream road weather management as a core function of State DOTs. Supporting this desire is the recent engagement in the institutional capability maturity framework development and deployment for road weather management.

Performance Measurement/Continuous Improvement of Road Weather Management Programs

The role of performance measurement continues to grow through both Federal and State initiatives. The Moving Ahead for Progress in the 21st Century Act (MAP-21) requires a greater consideration of performance in transportation investments, and States are looking at ways to maximize the return on investment of their limited resources. Higher traveler expectations are also fueling the increased use of performance measures to gauge agency performance. Currently, there are no measures that document the extent of use in collecting and reporting performance measures by the State DOTs.

Updated Performance Measures for 2015

Table 8 lists the measures that were identified as candidates for inclusion in the 2015 performance measures update. A total of 27 measures are tracked. Seven new measures have been added since the last update in 2012. One measure has been deleted, and the wording/definitions of four measures have been revised. With the exception of Objective 4, there are multiple performance measures associated with all objectives of the program. New measures are highlighted in bold within the table.

Quantifying Measures

Each of the 27 measures was quantified during this update using a variety of data sources. Conducting the performance measure update requires collecting data available in 2015 from multiple sources on the specific RWMP activities and the broader impact of road weather management efforts.⁽⁹⁾ There are four main sources that provide data for the RWMP performance measures:

• RWMP Records - The depth and breadth of the RWMP's research, training, and engagement activities are documented in RWMP records—these data demonstrate the reach and impact of the RWMP.
• State DOT Survey - One of the key data sources used in the previous updates was a targeted survey of State DOTs, which compiled data on the current levels of RWM deployment and capabilities around the country. For the 2015 update, a brief online survey was distributed to representatives at 49 State DOTs (all States except Hawaii). The survey was completed by 52 respondents from 40 states (an 82% response rate), comprised of almost all the winter-weather states. Figure 2 below illustrates the distribution of the survey respondents.
• Literature Reviews and Internet Searches - Peer-reviewed literature and Pooled Fund Studies (PFS) provide additional data for the performance measure update especially as it pertains to data regarding system outcomes and specific case studies/evaluation of road weather management strategies.
• Additional Data Sources - Other data resources are used to supplement the primary sources listed above to meet the data requirements for the performance measurement update. These include the Intelligent Transportation Systems (ITS) Deployment Statistics, ITS Benefit-Cost Database, and the FHWA Operations Efficiency Index (OEI). In many cases, these data elements will be used to support the findings for the performance measures.

Figure 2. Chart. Map of the 2015 State Survey Respondents (shown in blue).

Assumptions, Challenges, and Constraints

The performance measures are crafted to reflect the changes to the current program and broader road weather management context, yet limitations arise particularly relating to data availability and the ability to isolate independent impacts. The following assumptions and limitations should be noted:

• The main assumption underlying the use of these performance measures is the State DOT as the unit of measure for data collection. While State DOTs represent the primary stakeholders for the RWMP, there are other entities involved in the implementation of road weather management. The involvement of these other agencies in the data collection is limited, but will be highlighted and quantified where possible.
• Additionally, while some significant impacts are highly attributable to the RWMP programs, projects, and activities, some aspects of the program's goal attainment may result from indirect impact channeled through other national efforts (e.g., AASHTO, Pooled Funds, etc.) that operate within the realm of road weather management. Therefore, it is a challenge for performance measurement to entirely isolate and measure the independent impacts attributable to the RWMP from aggregate impacts that are contributing to goal attainment.
• The State DOT survey is developed to maximize comparability for the measures used in previous updates by replicating wording used in prior surveys. However, changes in survey methodology and reporting that occur with external data sources make comparability a challenge—this is particularly true with other survey sources such as ITS Deployment Statistics and the OEI.
• The lack of widely accepted performance measures and methods for evaluating winter maintenance activities across the nation makes regional comparisons difficult especially at the outcome level.

Performance Results

The following sections provide results for each of the twenty-seven measures organized by the eight objectives. A short summary across the objective is also provided.