Organizing for Reliability – Capability Maturity Model Assessment and Implementation Plans
Executive Summary

1.0 State DOT and Regional Capability Maturity Self-Assessment Program: General Background

Many State DOTs and regions have recognized the importance of more effective TSM&O to improving customer service and system performance. TSM&O is an integrated program to optimize the performance of existing multimodal infrastructure through implementation of specific systems, services, and projects designed to preserve capacity and improve the security, safety, and reliability of the transportation system.

1.1 SHRP 2 Research on Reliability and the TSM&O Capability Maturity Model

SHRP 2 included a Reliability Focus Area that produced research and products on many important data, analytic, and design issues, as well as process and applications improvements. In addition, the SHRP 2 Reliability Focus Area included a project to identify the institutional characteristics of the agencies with the more effective TSM&O activities. (Institutional Architectures to Improve Systems Operations and Management, SHRP 2 L06, 2012.) This research determined that agencies with the most effective TSM&O activities were differentiated not by budgets or technical skills alone, but by the existence of critical processes and institutional arrangements tailored to the unique features of TSM&O applications. These processes and institutional arrangements are organized into six dimensions: business processes; systems and technology; performance measurement; agency culture, organization and staffing; and collaboration.

Using these critical dimensions, the research project adapted concepts from the CMM process – widely used in the Information Technology industry – to create a self-assessment framework designed to help transportation agencies identify their current strengths, weaknesses, and related actions needed to improve their capabilities for effective TSM&O – in effect, a roadmap for “getting better at getting better.”

A detailed version of the CMM framework was further developed into a web-based self-assessment tool: AASHTO Guide to Transportation Systems Management and Operations. (National Cooperative Highway Research Program Project 03-94 (2011).) The framework has been used in the National Operations Academy™ and the Regional Operations Forum programs, as well as by FHWA in specific applications. (National Operations Academy™ and Regional Operations Forums.)

The TSM&O CMM framework was adapted for use in the facilitated one-day self-assessment workshop process for State DOTs and regions. As part of the SHRP 2 research project, the process was validated in five State and regional workshops in 2010 and 2011. Following validation, FHWA sponsored 11 additional workshops from 2011 through 2013.

1.2 CMM Workshop Program

Based on this experience, FHWA expanded the workshop concept through its SHRP 2 Implementation Assistance Program by adding a pre-workshop senior leadership meeting and post-workshop implementation plan development based on the priority actions established in the self-assessment process. In 2013, FHWA and AASHTO solicited State DOTs and regional agencies and selected 27 host entities to implement this full process. Three of these 27 previously had conducted a self-assessment workshop and were selected to develop an implementation plan.

In total, 40 TSM&O CMM workshops have been conducted through December 2014 as shown in Figure 1.1.

Figure 1.1 Map. Self-Assessment CMM Workshop Locations

(Source: Cambridge Systematics, Inc.)

This report synthesizes findings, as of December 2014, from 23 of the sites selected by FHWA and AASHTO in 2013 as part of the SHRP 2 Implementation Assistance Program. These 23, listed in Table 1.1, included consultant workshop development and facilitation. Twenty of these sites, indicated in bold, also received consultant assistance for creation of their implementation plans. Of the 23 workshops, 19 were hosted by State DOTs, of which 17 were statewide in focus. Six workshops had a regional focus (indicated in italics), of which two were hosted by State DOT districts and four were hosted by regional entities, including metropolitan planning organizations (MPOs).(The three sites not bolded in Table 1.1 completed an implementation plan without consultant assistance, along with four other sites (out of 27) not listed in the table for which no consultant assistance was supplied for a workshop or an implementation plan.)

Table 1.1 Self-Assessment CMM Workshop Locations Analyzed in this Final Report

Arizona	NOACA (Cleveland, Ohio)
California	Ohio
Colorado	Oregon
Florida District 5 (Orlando)	Pennsylvania
Georgia	Rhode Island
Iowa	South Dakota
Kansas District 5 (Wichita)	Tennessee
Maryland	Utah
New Jersey	Washington, D.C.
Michigan	Washington State
Missouri	Whatcom (Whatcom County, Washington)
NITTEC (Buffalo, New York)

1.3 The CMM Self-Assessment Workshop Objectives and Stakeholders

The CMM workshops are intended to improve the effectiveness of TSM&O applications and activities by assisting staff and management who are central players in statewide and regional TSM&O agencies. The workshop framework provides a structured focus on the six dimensions of capability, together with a facilitated self-assessment process in which participants evaluate their current activities and arrangements according to criteria from the CMM framework defining levels of capability. The current challenges and problems identified by workshop participants are used to identify actions needed to improve capability and are subsequently embodied in an implementation plan to improve the effectiveness of TSM&O.

Given the self-assessment focus, workshop participation includes agency middle management: the unit managers and key technical staff with day-to-day oversight of TSM&O-related activities. Senior management are involved in a pre-workshop briefing, referred to as the Senior Leadership meeting, and their approval of the implementation plans are required as a precondition of Federal financial assistance. Workshops are limited to 20 to 25 participants selected by the host agency to represent the most important players at central office and district/regional units, as well as their partners in the delivery of TSM&O, including public safety agencies, MPOs, local governments, and the private sector.

1.4 The Capability Maturity Self-Assessment Framework

The CMM self-assessment framework is structured in terms of six dimensions of capability. Three dimensions are process oriented:

1. Business Processes, including planning, programming, and budgeting (resources);
2. Systems and Technology, including use of systems engineering, systems architecture standards, interoperability, and standardization; and
3. Performance Measurement, including measures definition, data acquisition, and utilization.

Three dimensions are institutional:

1. Culture, including technical understanding, leadership, outreach, and program legal authority;
2. Organization and Staffing, including programmatic status, organizational structure, staff development, and recruitment and retention; and
3. Collaboration, including relationships with public safety agencies, local governments, MPOs, and the private sector.

For each of these six dimensions, the self-assessment utilizes four criteria-based “levels” of capability maturity that indicate the direction of managed changes required to improve TSM&O effectiveness:

• Level 1 – “Performed.” Activities and relationships largely ad hoc, informal, and champion driven, substantially outside the mainstream of other DOT activities.
• Level 2 – “Managed.” Basic strategy applications understood; key processes support requirements identified and key technology and core capacities under development, but limited internal accountability and uneven alignment with external partners.
• Level 3 – “Integrated.” Standardized strategy applications implemented in priority contexts and managed for performance; TSM&O technical and business processes developed, documented, and integrated into DOT; partnerships aligned.
• Level 4 – “Optimizing.” TSM&O as full, sustainable core DOT program priority, established on the basis of continuous improvement with top-level management status and formal partnerships.

Table 1.2 illustrates the basic CMM self-assessment framework of dimensions and levels of capability.

Table 1.2 CMM Self-Assessment Framework

Dimensions	Capability Level Criteria – Level 1 Performed	Capability Level Criteria – Level 2 Managed	Capability Level Criteria – Level 3 Integrated	Capability Level Criteria – Level 4 Optimizing
Business Processes (Planning and Programming)	Each jurisdiction doing its own thing according to individual priorities and capabilities	Consensus regional approach developed regarding TSM&O goals, deficiencies, B/C, networks, strategies and common priorities	Regional program integrated into jurisdictions’ overall multimodal transportation plans with related staged program	TSM&O integrated into jurisdictions’ multi-sectoral plans and programs, based on formal continuing planning processes
Systems and Technology	Ad hoc approaches to system implementation without consideration of systems engineering and appropriate procurement processes	Regional ConOps and architectures developed and documented with costs included; appropriate procurement process employed	Systems and technology standardized and integrated on a regional basis (including arterial focus) with other related processes and training as appropriate	Architectures and technology routinely upgraded to improve performance; systems integration interoperability maintained on continuing basis
Performance Measurement	Some outputs measured and reported by some jurisdictions	Output data used directly for after-action debriefings and improvements; data easily available and dashboarded	Outcome measures identified (networks, modes, impacts) and routinely utilized for objective-based program improvements	Performance measures reported internally for utilization and externally for accountability and program justification
Culture	Individual staff member champions promote TSM&O, varying among jurisdictions	Jurisdictions’ senior management understands TSM&O business case and educates decision makers/public	Jurisdictions’ mission identifies TSM&O and benefits with formal program and achieves wide public visibility/understanding	Customer mobility service commitment accountability accepted as formal, top level core program of all jurisdictions
Organization/Staffing	TSM&O added on to units within existing structure and staffing – dependent on technical champions	TSM&O-specific organizational concept developed within among jurisdictions with core capacity needs identified, collaboration takes place	TSM&O Managers have direct report to top management; Job specs, certification and training for core positions	TSM&O senior managers at equivalent level with other jurisdiction services and staff professionalized
Collaboration	Relationships ad hoc and personal (public-public, public-private)	Objectives, strategies, and performance measures aligned among major players (transportation and public safety agencies (PSAs)) with after-action debriefing	Rationalization/sharing/formalization of responsibilities among key players through co-training, formal agreements, and incentives	High level of TSM&O coordination among owner/operators (State, local, private)

Levels cannot be skipped as an agency improves its capability since each level builds on the technical and/or organizational readiness of the previous level. The dimension at the lowest level of capability is usually the principal constraint to improvement in program effectiveness and therefore the highest priority focus for capability improvement.

1.5 The CMM Workshop Process

The full CMM self-assessment workshop sequence conducted for the 23 agencies summarized in this report consists of four steps. The first step involves pre-workshop preparation by the facilitators to review key documents provided by the State or region documenting current TSM&O initiatives and activities, programs, and priorities in the host area. Facilitators review items such as ITS strategic plans, concepts of operations, long-range transportation plans, agency organization charts, and other relevant plans and documents. A pre-workshop conference call with the lead and/or core team from the host State or region also helps to identify current issues and challenges.

The second step is a senior leadership meeting held a day in advance of the workshop at which senior agency leaders are briefed to explain the process and its value and to obtain their perspectives on important issues.

The third step is the facilitated self-assessment workshop involving the essential TSM&O-related managers from State DOTs and their partners. This workshop starts with a facilitated self-identification of current strengths and weaknesses in each dimension and a consensus determination regarding their current levels of capability. These assessments serve as the basis for participants’ identification of the actions needed to improve to the next level of capability in each dimension and the priorities on which to focus.

The fourth step consists of implementation plan development. Based on workshop results, the facilitators suggest work program tasks for the priority dimensions, starting with those at the lowest level of capability. The host agency then produces a detailed implementation work program to improve capabilities in these priority dimensions and makes adjustments to the recommendations as they see fit. Upon FHWA and host agency management approval, the implementation plan is eligible for both financial and technical resource support from FHWA and AASHTO under the SHRP 2 Implementation Assistance Program. The support includes training, workshops, peer exchanges, technical assistance, collaboration activities, and additional progress assessments and meetings.

The CMM concept including the dimensions and levels utilized in the TSM&O program level assessments presented in this report have also been adopted by FHWA for utilization in developing application-specific self-assessment tools and activities.