Improving Transportation Systems Management and Operations – Capability Maturity Model Workshop White Paper – Performance Measurement

Executive Summary

Background

Research done through the Second Strategic Highway Research Program (SHRP 2) determined that agencies with the most effective transportation systems management and operations (TSM&O) activities were differentiated not by budgets or technical skills alone, but by the existence of critical processes and institutional arrangements tailored to the unique features of TSM&O applications. The significance of this finding has been validated in 40 State and regional self-assessment workshops using the Capability Maturity Model (CMM) and its six dimensions of organizational capabilities. This White Paper focuses on Performance Measurement as one of the central dimensions of capability needed to support effective transportation systems management and operations (TSM&O – including collaboration with public safety agencies. MPOs, local government, and public-private partnerships The Paper summarizes the TSM&O state-of-the-practice based on the Workshops and subsequent implementation plans developed at 23 sites selected by FHWA and the American Association of State Highway and Transportation Officials (AASHTO) as part of SHRP 2 Implementation.

Scope

The paper includes the following material:

• A description of the SHRP 2 research and Workshop process related to the institutional and process aspects of TSM&O including a description of the CMM Self-Assessment Framework and its application to the Performance Measurement dimension.
• A discussion of the state-of-the-practice regarding Performance Measurement in terms of their key elements including self-assessed capability level.
• A description of key synergies between Performance Measurement and the other dimensions of capability – and managers span of control to affect improvement.
• Best practice examples and references.
• Suggested actions to address Performance Measurement needs on a national level.
• An Appendix presenting the Common Implementation Plan Priority Actions for the Performance Measurement dimension.

State of the Practice Findings for TSM&O Performance Measurement

Key findings from the workshops included:

Measures Definition

• Policy visibility of performance. Most states/regions are conscious of the impending requirements of MAP‑21, and performance measures are much discussed in professional circles. All locations were at least in the stage of developing operations performance measures and most had started to compile them. Several agencies cited the need for guidance and standardization in performance measure development.
• Performance measure definition. Lack of performance measure definitions for weather, work zones, and signalized arterials was frequently mentioned as a problem. Performance measures for programs where multiple agencies are involved – such as incident management – is sometimes problematic.
• Input, output, and outcome measures The agencies that defined outcome measures reported having reviewed the literature and observed what other agencies were doing, but a single reference for guidance was not used. As with output measures, the need for guidance and standardization of outcome was cited by several agencies.
• Resources for Performance Measurement. Obtaining funding for Performance Measurement is a challenge for some agencies. In some cases, upper management is not convinced of the need for it, and the funding must come from existing budgets. One promising trend observed in several State DOTs (and discussed in others) involved active use of an existing agencywide Performance Measurement office/unit or an intention to establish such a unit in response to the Performance Measurement requirements of MAP‑21.

Data Acquisition

• Existing data availability. The availability of data for incident management activities varies among agencies. Some TSM&O units collect and “own” TIM data. In other agencies, TSM&O units are dependent on emergency responder CAD systems for TIM data. Freeway detector data also are widely available but not all agencies use them to develop congestion statistics (outcome measures). Work zone data are difficult to obtain. Work zones are usually overseen by other units within the agency (e.g., construction, capital projects) and might not be connected to other operations activities, even during implementation. As a result, they have their own processes.
• Outsourcing. Several agencies mentioned MAP‑21 as a driving force behind travel-time/speed data acquisition. Private vendor vehicle probe data are becoming more widely available. Many suggested that they were looking into probe data not only to meet MAP‑21 requirements but also to fill in gaps where detectors do not exist. Several agencies have existing contracts with traffic information providers, while others are investigating it, especially in response to meeting MAP‑21 requirements.

Measures Utilization

• Internal utilization. Incident management and snow and ice control are the two areas where performance data are used for operational management. The high public visibility of road clearance conditions and operations has led many snow belt states to track and report clearance in real time. Some agencies conduct after-action review of incidents that are supported by the data, although the reviews drill down deeper into what caused problems and what worked well with the management of the incident. Traveler information program performance (e.g., web site hits and VMS messages) also was noted by several areas: usage statistics and trends were monitored, and in some cases influenced operational decisions in terms of system enhancements or upgrades.
• External reporting. Production of periodic performance reports was the most common use of performance measures, although not all agencies produced them. A few States included TSM&O-related activity measures – largely output data on external (web site) dashboards. Because of data availability and the ease of summarizing them, incident characteristics were by far the most frequent subject of performance reports. Travel time (congestion) based reports based on measured data were far more rare.
• Management accountability. Accountability for TSM&O program performance is in the early stages. Several States have incident clearance targets but conduct reviews only when the target (often 90 minutes) is exceeded. There were no instances described in workshops where DOT units were subject to performance reviews in this regard.
• Comprehensive performance management program. No agency has achieved a fully integrated Performance Measurement system that links inputs, outputs, outcomes, and targets into a formal TSM&O performance management process. Agency staff are aware of the importance of outcome measures to making the business case for TSM&O to decision makers and the public, but they have made very limited progress in considering the data and analytics related to outcome measures such as travel time, reliability, and safety.
• Outsourcing of outcome measures. Private sector probe data is seen by many States as a way of obtaining useful performance analyses. Several States are in the early stages of identifying outcome measures and acquiring probe data to support them. DOTs with extensive toll operations are capitalizing on tags as probes. A number of States and regions recognize the need to focus on Performance Measurement for arterial operations, although data availability is an obstacle.
• Use of performance measures in business case materials. Only a few agencies have prepared a TSM&O strategic plan that identifies TSM&O goals and objectives and develops performance measures that track progress towards them. Few agencies had any guiding documents of any kind (e.g., operations data business plan, Performance Measurement plan) to guide the development of the Performance Measurement that was in place; most were done with minimal advance planning. Several agencies cited a need for guidance on conducting before/after evaluations of operations projects.

Synergism

Performance Measurement is especially interactive with the Business Processes and Collaboration dimensions. The Business Processes dimension should be used to define the Performance Measurement framework. This should be an ongoing process, not a single undertaking or a one-way link. Performance Measurement itself should evolve along with the other dimensions as more is learned about what types of measurement are needed. The Collaboration dimension is significant in that Performance Measurement needs to be consistent across departments and agencies. Collaboration is important to Performance Measurement in that it can “break down silos” of related but uncoordinated activities.

State DOT and Regional Implementation Plan Priorities

The leading participant-suggested actions for Performance Measurement include:

• Creating a comprehensive performance measurement system. This includes: linking performance measures to TSM&O goals; establishing common performance measure definitions; defining performance measures for all aspects of operations; linking output measures to outcome measures for system performance; specifying target setting procedures; identifying data sources to support performance measures; and software specifications for a formal monitoring system.
• Promoting operations in traditional planning and programming processes. This includes: integrating operations into planning documents; applying a common set of performance measures for all phases of the project development process; developing a modeling plan and tools for supporting TSM&O analysis; and specifying evaluation procedures for completed TSM&O projects.
• Developing a communication strategy for describing the benefits of TSM&O to upper management and the public.

Best Practices and National Needs

This white paper describes example best practices and reference material regarding the implementation plan priority needs noted above. The paper also suggests supportive national actions to improve TSM&O Performance Measurement including: developing standard definitions and procedures for plan development, measures, data processing, target setting, and reporting; syntheses of best practices in performance measurement and management; a field test of a comprehensive operations performance management system. Important roles were seen for FHWA, AASHTO and the National Operations Center of Excellence in supporting these efforts.