Freeway Management and Operations Handbook

Chapter 4 – Performance Monitoring and Evaluation
Page 1 of 3

4.1 Introduction

Freeway management programs – consisting of operational strategies, low-cost roadway improvements, and ITS-based systems – are implemented as a means to enhance safety, preserve mobility, improve reliability, increase productivity, and meet the public's expectations for efficient and effective travel. Moreover, freeway management initiatives are planned, designed, deployed, operated, and maintained with public funding. It is therefore important to ensure that these funds are spent wisely, that the agency makes the best use of its available resources, and that the full potential of past and current investments is realized. This, in turn, requires that the performance of the freeway be continuously monitored and evaluated, and that appropriate analytical tools and measures be used to identify and quantify problems, to evaluate alternative solutions, and to assess the extent to which the implemented solutions achieve their objectives.

Another consideration is that freeway management and operations – particularly ITS-based improvements – are increasingly funded through the use of regular sources (i.e., not specific to ITS or operations). The move to "mainstream" funding mechanisms necessitates the integration of freeway management and operations into the established transportation planning process, where freeway management strategies and systems can be evaluated both against, and in combination with, conventional transportation components such as major road widening and new facility construction. It is critical that the associated benefits and costs are known and compared in an equitable manner (i.e., using the same set of performance measures and criteria), thereby providing an economic justification for the implementation of freeway management systems and operational strategies.

Increased customer expectation and public sector accountability have helped to focus attention on performance measurement as one of the essential tools at the practitioner's disposal. To be held accountable, one needs a clear understanding of what they are trying to accomplish and how to assess the results in such a way that they can continue to improve. Indeed, this is why performance measurement in government has become such a hot topic. Osborne and Gaebler (1992) summed it up well in their landmark work, Reinventing Government:

• If you don't measure results, you can't tell success from failure.
• If you can't see success, you can't reward it.
• If you can't see failure, you can't correct it.

Freeway practitioners work to achieve results. Performance measures are indicators of work performed and results achieved (1).

4.1.1 Chapter Scope & Objectives

The use of performance measures in transportation planning and investment decision-making processes of public agencies has increased significantly. This demand has lead to the need for information and guidance on how to integrate the consideration of freeway performance into these processes. Moreover, the day-to-day operation and management of the freeway requires real-time knowledge how well the freeway is performing and the existence of any problems. There is general agreement among transportation practitioners that freeway system performance monitoring, evaluation, and reporting should be performed and continuously supported by operating agencies.

Performance measures are the primary focus of this chapter, including discussions of why they are important, their relationship to the decision-making process, and important considerations when selecting performance measures. Several examples of performance measures that may be utilized for freeway management and operations are then provided. The section on performance measurement concludes with discussions on information gathering, data archiving, and reporting.

There are several attributes of a freeway management program – such as how well the operations process is organized and administered, and how well it interacts with other agencies and affected stakeholders – that are difficult to quantify in terms of a performance measure. Several self-assessment tools have been developed by FHWA for this purpose, and are discussed herein.

Evaluation of a freeway management and operations program (and other transportation improvements) must occur throughout the life cycle of the facility, including identifying problems and segments with less-than-desired performance, analyzing and prioritizing alternative solutions for correcting these problems, estimating the associated benefits and costs, and determining the actual improvement in performance and its cost effectiveness. An overview of such methods and analytical tools (e.g., Highway Capacity Manual, simulation, before-and-after studies, estimating costs and benefits) is also provided in this chapter.

4.1.2 Relation to Other Freeway Management Activities

Performance monitoring and evaluation is a continuous process that occurs throughout the life cycle of the freeway facility. Moreover, as shown in the "funnel diagram" in the previous chapter (Figure 3-1), determining performance measures" is one of the key activities when establishing, enhancing, and managing a freeway operations program. Reiterating the description of the performance measure "step" from Chapter 2:

"The performance measures provide the basis for evaluating the transportation system operating conditions and identifying the location and severity of congestion and other problems. The performance measures provide the mechanism for quantifying the operation of the network, and should also be used to evaluate the effectiveness of implemented freeway management strategies and to identify additional improvements. Another aspect of performance measurement is sharing and providing managers and users with access to real-time and archived system performance data."

Performance measures and analytical tools need to be considered and/or utilized in other freeway management activities, including:

• Stakeholders – Stakeholders are interest groups who benefit from, or are otherwise impacted by, freeway management and operations (e.g., the various transportation providers, transportation system users, and other persons or organizations with a strong material interest in success or failure of freeway management). The stakeholders should be involved in the processes to define performance measures and how they are used.
• Needs – This is an initial assessment of how the freeway network should operate relative to how it operates today. Needs are embodied in the vision, goals, and overall public policy. They may be further defined in a variety of ways, including performance evaluations (e.g., comparing actual operational measures to performance criteria) and analytical assessments of freeway performance (e.g., before and after studies).
• Implementation – Using the broad description of this step from Chapter 3, this activity includes design – deciding "how" each need and the corresponding requirements will be satisfied. Performance measures and analytical tools identified in this chapter may be useful in evaluating alternatives and selecting the most cost-effective one (e.g., simulation, estimate benefits or utilities for each alternative, estimate life-cycle costs of each alternative, perform comparative analysis). A type of performance measures is also used during the actual implementation of freeway improvements – criteria for the various component and system tests.
• Evaluation involves the routine collection and analysis of appropriate data, comparing the results with the previously-established performance measures, and evaluating the performance of the strategies, policies, systems, and operator procedures that comprise the program. This "feedback" element of the process allows practitioners to assess the effectiveness of their efforts, to identify areas for improvement, to justify these improvements (e.g., configuration management process), to demonstrate the benefits provided by the program, and to support requests for additional resources.

The collection of data is an important element of a performance monitoring and evaluation process. An ITS-based Freeway Management System (FMS) represents a potentially valuable data source in this regard. Accordingly, in addition to the other functions of a surveillance subsystem (discussed in Chapter 15), it should be developed and deployed to automatically collect, store and analyze data associated with performance measures to the greatest extent possible. FMS-generated data will not only benefit the transportation operations and planning communities by allowing them to access more and better data. It will also enhance the appeal of FMS deployment by significantly broadening its originally intended benefits.

4.2 Performance Measures

4.2.1 Overview

Performance measurement may be defined as follows (from Reference 2):

"Performance measurement is a process of assessing progress toward achieving predetermined goals, including information on the efficiency with which resources are transformed into goods and services (outputs), the quality of those outputs (how well they are delivered to clients and the extent to which clients are satisfied) and outcomes (the results of the program activity compared to its intended purposes), and the effectiveness of government operations on terms of their specific contributions to program objectives"

Performance measures provide the basis for identifying the location and severity of problems (such as congestion and high accident rates), and for evaluating the effectiveness of the implemented freeway management strategies. This monitoring information can be used to track changes in system performance over time, identify systems or corridors with poor performance, identify the degree to which the freeway facilities are meeting goals and objectives established for those facilities, identify potential causes and associated remedies, identify specific areas of a freeway management program or system that requires improvement / enhancements, and provide information to decision-makers and the public. In essence, performance measures are used to measure how the transportation system performs with respect to the overall vision and adopted policies, both for the ongoing management and operations of the system, and the evaluation of future options.

Agencies have instituted performance measures and the associated monitoring, evaluation, and reporting processes for a variety of reasons – to provide better information about the transportation system to the public and decision makers (in part due, no doubt, to a greater expectation for accountability of all government agencies); to improve management access to relevant performance data; and to generally improve agency efficiency and effectiveness, particularly where demands on the transportation agency have increased while the available resources have become more limited.

A rather succinct view of performance measures is provided by Wolf (Reference 3) in a paper prepared for the 4th Integrated Transportation Management System (ITMS) conference held in 2001. The author describes the California performance measurement effort, stressing that it "was critical throughout the process to remind partners what performance measurement was and still is:

• A standard management function to help understand accomplishments
• A planning tool to improve investment analysis
• Customer-oriented as opposed to service provider-driven
• A genuine system perspective, as modally blind as possible
• A lengthy, evolving process
• Very effective if there is a clear purpose and simple set of metrics based on readily obtainable data;

And what performance measurement isn't:

• A panacea
• An isolated exercise
• A magical "Black Box"
• Naive over-simplification
• Usurpation of regional authority"

Finally, it should be emphasized that performance measures for transportation operations are not a fleeting trend; but a permanent way of doing business that eventually will be used at all levels of transportation agencies.

4.2.2 Performance Measures and Decision Making

Chapter 2 describes several "tiers" amongst which the authority for transportation decision-making is dispersed. Performance measures are necessary at each of these tiers. However, as Meyer (Reference 4) points out, "at each level, there could be measures desired by the corresponding decision makers that are specific to that decision context. At the very highest level, this could imply that decision makers might be interested in issues and performance measures not directly linked to information surfacing from the other levels."

A Transportation Management Center (TMC) requires an accurate real-time monitoring of the freeway's performance, and how that performance compares to "normal" (using performance measures over time to define "normal"). The TMC manager and operators monitor the performance of the facility to assess existing conditions for short-term non-recurring events and for longer term recurring congestion, determine and implement operational plans, and inform freeway users of existing and predicted near-term conditions. The freeway manager also uses the results of the performance monitoring to identify deficiencies in the physical freeway system, and provides planners and designers with the necessary information and input to incorporate into the planning and design of future improvements to the facility.

Similarly, an Integrated Transportation Management System (ITMS) also requires real-time monitoring information, aggregated over the entire region, to address the performance of the entire surface transportation network (with data obtained from multiple TMCs and other sources). The real time information may be used to implement and monitor region-wide response plans. The data may also be archived and evaluated later to either modify existing response plans or create new ones.

The Agency tier requires performance measures for resource allocation and programming (i.e., making choices among alternatives) and for trade-off analyses – for example, setting appropriate performance targets for a policy or system plan when the trade-offs involve different objectives (safety and system preservation). The use of performance measures to help define the implications of these choices and trade-offs can be one of the most powerful ways to use performance measures to influence decisions.

Another common application of performance measures is in long-range planning at the Regional / Statewide tier. As noted in Chapter 2, State DOT's and regional agencies (metropolitan planning organizations – MPOs) maintain long-range planning activities to determine how to build and manage the transportation system to meet the stated needs and goals of the relevant customer group. In this context, performance measures must be sufficiently specific to permit distinguishing the effect of investment in one modal system or program of activities versus another. The objective is to give decision makers better information about the likely impact and outcome of different mixes of investment (or budget) among different programs. In a broader context, performance measures are needed at the statewide / regional level to help drive policies, goals, and objectives. They may also be useful for identifying the need for increased revenue and influencing the associated legislation (e.g., increased fuel tax).

Performance measures can also be used at the National tier to assist with policy making, goal setting, developing and justifying legislation, and developing reports for Congress. Even at this high level, measures can be identified that are consistent with broad policy and goals and that specify the desired outcome in unambiguous, quantifiable terms. The actual measures selected must sum up the net effect at the agency, ITMS, and TMC levels of many smaller, discrete actions. The time frame of the effect of such actions may be relatively long – some measures might not show marked change until a given policy has been implemented for several years.

Another important consideration is the need to improve the links between resource allocation decisions, system conditions, and performance results. Pickrell and Neuman (Reference 5) identify the following factors as contributing to the desire to link performance data to decisions about system investment:

• Accountability: Publicly funded agencies have come under increasing pressure to be accountable to ''customers.'' Performance measurement provides a means of determining if resources are being allocated to the priority needs, as identified through performance monitoring and reported to external or higher-level entities.
• Efficiency: Setting performance targets that are aligned with an agency's goals and mission help staff, management, and decision-makers stay focused on the priorities, thereby increasing efficiency. It improves internal management and the ability to direct resources where needed, to track results, and to make adjustments with greater confidence that the changes will have the desired effect.
• Effectiveness: Performance measurement may help an agency to better achieve objectives that have been identified through the planning process, and to improve the correlation between agency objectives and those of the system users or the general public. It reflects a shift in agency thinking away from simply output (e.g., ''tons of salt applied'') to outcome (e.g., ''reduction in ice-related fatalities'') and allows progress to be tracked explicitly.
• Communications: As an adjunct to accountability, a good performance measuring program cannot help but improve communications with an agency's customer base and constituency, including other agencies and entities that are involved with the operation and management of the surface transportation network.
• Clarity: Performance measurement can lend clarity of purpose to an agency's actions and expenditures, forcing clear thinking about the purposes of planning, programming actions, and investments in the transportation system.

One of the key discussion points from Chapter 2 is that freeway practitioners must become more involved and "provide substantive input to the Statewide and/or regional transportation planning process on necessary investments to improve system performance". Developing performance measures and collecting the associated data represent a potential approach for increasing one's involvement in the transportation planning and resource allocation process.

Performance measures should be viewed as a tool to improve, guide, and enhance the decision-making process; not as the means to replace or ''automate'' it. As noted in Reference 5, "much has been said about the undesirability of creating a ''black box'' approach to planning or decision making. Practitioners have commented that in some cases, decision makers tend to overapply performance data and absolve themselves of the responsibility to apply professional judgment or take responsibility for decisions. The emphasis should be on improving the transparency of the planning and programming processes rather than further cloaking them in quantitative language understandable to only a few. It should encourage participants to be clear about their objectives and more explicit about how they will work to achieve those objectives."

4.2.3 Developing Performance Measures

The development of performance measurement systems is a dynamic and incremental process. There is neither one right set of performance measures nor one right process to develop a performance measurement system. References 1, 2, 3, 5, 6, 7 and 8 provide guidelines for developing performance measures and the attributes of good performance measures as summarized below:

• Goals and objectives – Performance measures should be identified to reflect goals and objectives, rather than the other way around. This approach helps to ensure that an agency is measuring the right parameters and that "measured success" will in fact correspond with actual success in terms of goals and objectives. Measures that are unfocused and have little impact on performance are less effective tools in managing the agency. Moreover, just as there can be conflicting goals, reasonable performance measures can also be divergent (i.e., actions that move a particular measure toward one objective may move a second measure away from another objective). Such conflicts may be unavoidable, but they should be explicitly recognized, and techniques for balancing these interests should be available.
• Data needs – Performance measures should not be solely defined by what data are readily available. Difficult-to-measure items, such as quality of life, are important to the community. Data needs and the methods for analyzing the data should be determined by what it will take to create or ''populate'' the desired measures. At the same time, some sort of "reality check" is necessary – for example: are the costs to collect, validate, and update the underlying data within reason, particularly when weighed against the value of the results; can easier, less costly measures satisfy the purpose – perhaps not as elegantly, but in a way that does the job. Ideally, agencies will define and, over time, implement the necessary programs and infrastructure (e.g., detection and surveillance subsystems) for data collection and analysis that will support a more robust and descriptive set of performance measures.
• Decision-making process – Performance measures must be integrated into the decision-making process; otherwise, performance measurement will be simply an add-on activity that does not affect the agency's operation. Performance measures should be based on the information needs of decision makers, with the level of detail and the reporting cycle of the performance measures matching the needs of the decision makers. As previously noted, different decision making tiers will likely have different requirements for performance measures. One successful design is a set of nested performance measures such that the structure is tiered from broader to more detailed measures for use at different decision-making levels.
• Facilitate Improvement – The ultimate purpose of performance measures must clearly be to improve the products and services of an agency. If not, they will be seen as mere "report cards", and games may be played simply to get a good grade. Performance measures must therefore provide the ability to diagnose problems and to assess outcomes that reveal actual operational results (as compared to outputs that measure level of effort, which may not be the best indicator of results).
• Stakeholder Involvement – Performance should be reported in stakeholder terms; and the objectives against which performance is measured should reflect the interests and desires of a diverse population, including customers, decision makers, and agency employees. Buy-in from the various stakeholders is critical for initial acceptance and continued success of the performance measures. If these groups do not consider the measures appropriate, it will be impossible to use the results of the analysis process to report performance and negotiate the changes needed to improve it. Those who are expected to use the process to shape and make decisions should be allowed to influence the design of the program from the beginning. Similarly, those who will be held accountable for results (who are not always the same as the decision makers) and /or will be responsible for collecting the data should be involved early on to ensure that they will support rather than circumvent the process or its intended outcome. The selected performance measures should also reflect the point of view of the customer or system user. An agency must think about who its customers are, what the customers actually see of the department's activities and results, and how to define measures that describe that view.
• Other Attributes – Good performance measures possess several attributes that cut across all of the "process" issues noted above. These include:
  • Limited number of measures – All other things being equal, fewer rather than more measures is better, particularly when initiating a program. Data collection and analytical requirements can quickly overwhelm an agency's resources. Similarly, too much information, too many kinds of information, or information presented at too fine a level of disaggregation can overwhelm decision makers. The corollary is to avoid a performance measure that reflects an impact already measured by other measures. Performance measures can be likened to the gauges of a dashboard – several gauges are essential, but a vehicle with too many gauges is distracting to drive.
  • Easy to measure – The data required for performance measures should be easy to collect and analyze, preferably directly and automatically from a freeway management (or other) system.
  • Simple and understandable – Within the constraints of required precision, accuracy, and facilitating improvement, performance measures should prove simple in application with consistent definitions and interpretations. Any presentation of performance measures data must be carefully designed such that it is easy for the audience to understand the information, and that the data analysis provides the information necessary to improve decision making.
  • Time frame – The decision-making "tiers" can have significantly different time frames, both for the making of the decision and for the effect of that decision to take place. Using performance measures to monitor the effectiveness of a policy plan requires measures that can reflect long-term changes in system usage or condition. Similarly, performance measures for the operation of a TMC should reflect changes within a "real-time" context. Once established, performance measures should be in place long enough to provide consistent guidance in terms of improvements and monitoring to determine whether the objectives are being met.
  • Sensitivity – Performance measurement must be designed in such a way that change is measured at the same order-of-magnitude as will likely result from the implemented actions.
  • Geographically appropriate – The geographic area covered by a measure varies depending on the decision-making context in which it is used. The scope of measures used to evaluate progress on broad policies and long-range planning goals and objectives often are region-wide, statewide, and even nation-wide. To be effective in an operations context, measures may need to be focused on a specific geographic area (e.g., corridor, system).

4.2.4 Examples of Performance Measures

This section identifies several potential performance measures from a number of different references. It is not the intent of this section to suggest that the practitioner should utilize all of these performance measures (several of which are repeated between different references). Quite the opposite. The number of performance measures should be kept to a manageable minimum number, provided that they conform to the attributes discussed in the previous section, and answer the following key questions regarding the freeway network:

• How many people/vehicles are using the system?
• Where and when are they being delayed and / or subject to unsafe conditions?
• How frequently do those delays / unsafe conditions occur?
• How bad are the delays / unsafe conditions?
• What are the reasons for these delays / unsafe conditions?
• Can I measure the effect of operational improvements on the delays / unsafe conditions?

4.2.4.1 Overview

Performance measures are often described as input, output, or outcome measures. Input measures look at the resources dedicated to a program; output measures look at the products produced; and outcome measures look at the impact of the products on the goals of the agency. For example, with respect to increasing roadway capacity, an input measure might be materials consumed; output measures could include lane-miles added; while an outcome measure might include the reduction in hours of user delay, resulting from the increased capacity. Outcome measures are preferred because they directly relate the agency's strategic goals to the results of the activities undertaken to achieve them. Outcome measures are also generally more difficult to define and measure. In deciding which measures to use, the agency needs to consider whether data can be collected to allow a measure to be calculated accurately and with sufficient frequency for it to be a useful tool in guiding decisions (7).

4.2.4.2 Background

A paper by Meyer (Reference 4) developed for the October 2000 Conference on Performance Measures and Performance Based Planning and Programming, provides the following short history of the use of performance measures.

"The primary developmental period for the systematic approach toward transportation planning that characterizes much of current practice occurred in the 1960s and 1970s. Transportation planning then was concerned with many issues, but primarily the focus was on system expansion to meet the growing demands for automobile travel and the corresponding characteristics of high speed and safe use of the road systems. Average vehicular speed, estimated usage of the system or network links (such as volume to capacity), number of crashes, and costs became the most used criteria for evaluating alternative transportation system plans. Because these were the criteria used for plan evaluation, they also tended to be the measures used in monitoring the ''effectiveness'' of transportation system performance.

As the nation's urban road system expanded in response to unprecedented population and employment growth, congestion on this system and the concomitant effects on the environment and on people's daily lives became important issues to system users, decision makers, and analysts. Congestion, the effects of congestion, and measuring congestion levels were thus some of the major system performance issues that drew the interest of transportation professionals in the 1980s and 1990s. However, much of this professional interest focused on measures that had been developed in the mid-1950s by engineers and planners who were interested in the impacts of congestion on vehicle flow. Suggested measures of congestion during this earlier period focused on three major factors:

• Operational characteristics of traffic flow, which included speed, delays, and overall travel times;
• Volume-to-capacity characteristics, which required a comparison of actual volumes with road capacity; and
• Freedom of movement characteristics, which required a determination of the percentage of vehicles restricted from free movement and the durations of such restrictions."

Meyers concludes this brief review of the background on performance measurement with the observation that "many of the measures proposed today to monitor system performance are similar to those proposed 50 years ago at the beginning of comprehensive transportation planning in the United States. In many ways, these measures carry a value judgment about what the system user, or perhaps society in general, perceives as acceptable or desirable performance. The measures have become entrenched as current and accepted practice for the monitoring of system performance, even though they were originally used for alternatives evaluation or design standards. For the road users, however, there may be different measures that reflect actual trip patterns and trip characteristics. If transportation is one of the empowering factors that allows economic development, affects environmental quality, and influences perceptions of quality of life, then decision makers will presumably want to know how system performance over time relates to these purposes."

4.2.4.3 Examples – Performance Based Planning

Table 4-1 illustrates the types of performance measures that have been proposed as part of the performance-based transportation planning. (Note: Table 4-1 is from Reference 4, edited to reflect those measures most applicable to freeway operations. Reference 4 itself is a summary of performance measures developed by Cambridge Systematics.) These measures are linked to the types of goals that are often part of the transportation planning process; although not that all of these measures will necessarily be part of the process. As previously discussed, the more measures there are, the more likely it is that their use for decision making will be confusing and ineffective. Rather, Table 4-1 is simply an illustration of the different types of measures that could be considered for each goal.

4.2.4.4 Examples – Mobility Measures

Providing individual mobility and accessibility to urban activities is an important goal for transportation planning, and therefore a critical precursor to the types of societal outcomes desired. Several efforts have been made to develop system level mobility indices. The Texas Transportation Institute (as reported in Reference 4) has developed several mobility measures that could be applied at the metropolitan level. Travel time plays a leading role in almost all of these measures, including the following:

• Travel Rate (minutes per mile) = travel time (in minutes) / segment length (miles)
• Delay Rate (minutes per mile) = actual travel rate – acceptable travel rate
• Reliability Factor = percentage of time that a person's travel time is no more than 10% higher than average
• Total Delay (vehicle-minutes) = [actual travel time (min.) – acceptable travel time (min.)] x vehicle volume.

The "acceptable travel time" is the total time it would take to travel a segment during expected conditions. This travel time is generally calculated assuming travel at the posted speed limit, although it may also be calculated using a congestion threshold speed established from local performance goals for mobility.

4.2.4.5 Examples – FHWA Mobility Monitoring Program

The Mobility Monitoring Program, a performance monitoring application sponsored by the Federal Highway Administration, attempts to quantify two key performance attributes of the transportation system – mobility and reliability. In non-technical terms, the mobility measures attempt to answer the question "how easy is it to move around?" and the reliability measures attempt to answer the question "how much does that 'ease of movement' vary?" For both mobility and reliability concepts, the monitoring approach is built upon travel-time based measures. Travel time concepts are well understood and used daily by non-technical audiences (e.g., commuters, travelers, passengers) and private sector transportation businesses (References 9 & 10)

The Program reports several measures each for mobility and reliability. Each of the measures attempts to quantify slightly different components of mobility and reliability. The primary mobility measures included in the program reports (9, 10) are:

• Travel time index – a ratio of travel conditions in the peak period to a target or acceptable travel condition (typically free-flow conditions are used). The travel time index indicates how much longer a trip will take during a peak time. For example, a travel time index of 1.3 indicates that the trip will take 30 percent longer (1.3 times longer).
• Percent of congested travel – this is primarily a system measure that quantifies the extent of congestion. A free-flow speed is used as a congestion "benchmark" and any travel on a road section for a time period that is less than the free-flow speed is determined to be congested. The congested travel is summed and then divided by total travel estimates.
• Delay per person – expressed in person-hours per year, this measure is used to reduce the total travel delay value to a figure that is more relatable to user experience. It also normalizes the impact of mobility projects that handle much higher demand than other alternatives.

These mobility performance measures reflect the average level of congestion and mobility. However, a number of empirical studies have demonstrated that travelers value not only the time it usually takes to complete a trip but also the reliability in travel times. For example, many commuters will plan their departure times based on an assumed travel time that is greater than the average to account for a lack of reliability.

During the first year of the Program, three reliability performance measures were tracked:

• Buffer index – this measure expresses the amount of extra "buffer" time needed to be on-time 95 percent of the time (late one day per month). Travelers could multiply their average trip time by the buffer index, then add that buffer time to their trip to ensure they will be on-time 95 percent of all trips. An advantage of expressing the reliability (or lack thereof) in this way is that a percent value is distance and time neutral.
• Percent variation – also known as the coefficient of variation, this is the amount of variability in relation to average travel conditions. It is calculated as the standard deviation divided by the mean. A traveler could multiply their average travel time by the percent variation, then add that product to their average trip time to get the time needed to be on-time about 85 percent of the time (one standard deviation above the mean). Higher values indicate less reliability.
• Misery index – this measure attempts to quantify the intensity of delay for only the worst trips. The average travel rate is subtracted from the upper 20 percent of travel rates to get the amount of time beyond the average for some amount of the slowest trips.

Of the three, the buffer index rose above others as the preferred measure, and it seemed to resonate with most audiences. There is no single agreed-upon reliability measure, and no customer/user market research has been performed. Even for these measures, it is not certain what level of reliability or variability (e.g., 85 percent, 90 percent, 95 percent, a combination) should be examined.

Data from transportation operations centers in 10 cities were used to develop and test the procedures and the performance measures. Individual city reports are available on the study website: http://mobility.tamu.edu/mmp.

4.2.4.6 Examples – NCHRP Synthesis 311

This Synthesis (Reference 2) examined the use of performance measures for the monitoring and operational management of highway segments and systems. More than 70 performance measures were identified. These were evaluated against the following criteria (adapted from many of the same references identified in Section 4.2.3 herein, and paralleling the attributes discussed in Section 4.2.3):

• Clarity and simplicity (e.g., simple to present and interpret, unambiguous, quantifiable units, professional credibility)
• Descriptive and predictive ability (e.g., describes existing conditions, can be used to identify problems and to predict changes)
• Analysis capability (e.g., can be calculated easily and with existing field data, techniques available for estimating the measure, achieves consistent results)
• Accuracy and precision (e.g., sensitive to significant changes in assumptions, precision is consistent with planning applications and with an operation analysis)
• Flexibility (e.g., applies to multiple modes, meaningful at varying scales and settings)

Table 4-2 lists those measures that received the highest scores (at least 75% of the possible maximum points) and were consistently reported in the synthesis of practice. These measures were also recommended based on "their ability to serve as a foundation for other commonly reported measures, such as congestion index".

4.2.4.7 System and Maintenance Measures of Performance

In addition to measuring the performance of the transportation network (including freeways), managers of freeway management systems will likely need additional performance measures as related to the performance of the FMS itself and its components. As discussed in Reference 11 ("Guidelines for transportation Management Systems Maintenance Concepts and Plans") the following parameters are useful data when evaluating products, "however, the reader of product specifications should be warned about hyperbole":

• Mean time between failures (MTBF) – defined in Reference 14 as the average time between hours of exposure for all like products divided by the number of failures.
• Mean time to repair – number of hours to make good the failed item
• Average cost to repair
• Design life

Reference 11 emphasizes that "design life and MTBF is not the same thing for all ITS devices. In some cases equipment can last decades if it is well maintained and necessary repairs are made. A hard drive, that may have a MTBF of 50 years, a design life of 5 years and a warranty for 2 years will cause an ITS system to crash and usually cannot be repaired. When considering the spares and replacements of ITS devices the developer of the plan needs to consider the most appropriate measure for that device on their facility."

Measuring the performance of the system maintenance program provides information both on organization and management issues in addition to the reliability of various FMS devices. Having metrics of the system provides the system with continual feedback on how well the system and its individual components are operating. The metrics associated with the structure of the plan could include:

• Down time of the entire system (e.g., aggregated over a specified period of time)
• Number of times the system is down
• Time to detect failure
• Time to handle responsive maintenance
• Time to handle emergency maintenance
• Time to bring system back on-line
• Negative calls from the public
• Adverse press