Evaluation Methods and Techniques | Advanced Transportation and Congestion Management Technologies Deployment Program

Chapter 3: Performance Measures

This chapter provides a set of recommended performance measures (PMs) to assist Advanced Transportation Congestion Management Technologies Deployment (ATCMTD) grantees in meeting the reporting requirements of the FAST (Fixing America's Surface Transportation) Act. As outlined in 23 U.S.C. 503(c)(4)(F), grantees must produce annual reports that describe the findings from their deployments, including data on benefits, costs, effectiveness, and lessons learned, among other data (see Develop Annual Report for specific FAST Act reporting requirements).

In addition, 23 U.S.C. 503(c)(4)(G) requires the Secretary of Transportation to submit a Program Level Report (not later than 3 years after the date of the first grant award and each year thereafter) that describes how the program has:

• Reduced traffic-related fatalities and injuries;
• Reduced traffic congestion and improved travel time reliability;
• Reduced transportation-related emissions;
• Optimized multimodal system performance;
• Improved access to transportation alternatives;
• Provided the public with access to real-time integrated traffic, transit, and multimodal transportation information to make informed travel decisions;
• Provided cost savings to transportation agencies, businesses, and the traveling public; or
• Provided other benefits to transportation users and the general public.

The PMs presented below are intended to provide ATCMTD grantees a core set of measures. In developing the set of recommended PMs, several key criteria were utilized, to the extent possible. Namely, the measures should be:

• Based on USDOT or other Federal guidance (as available),¹
• Appropriate for a broad range of technologies, and
• Able to be monetized for the purposes of benefit-cost analysis (BCA).

While the measures tend to be quantitative and outcome-based, measures that rely on qualitative data are also presented, as ATCMTD grantees will want to include performance measures that reflect a mix of both quantitative and qualitative data. In designing their evaluations, the ATCMTD grantees should start with the performance measures described below; however, the list is by no means exhaustive. Grantees may want to include additional performance measures that are tailored to their specific deployments and that provide insight on the safety, mobility, agency efficiency, and other impacts of their technology deployments. It should be noted that projects will not necessarily address all of the performance areas. PMs should be selected based on the technology being deployed, the anticipated impacts, and data availability.

The remainder of this chapter presents performance measures for each of the key performance areas outlined in the FAST Act:

1. Improve Safety
2. Improve Mobility/Reduce Traffic Congestion
3. Reduce Environmental Impacts
4. Optimize Multimodal Performance
5. Improve Access to Transportation Alternatives
6. Effectiveness of providing real-time integrated multimodal transportation information to the public to make informed travel decisions
7. Cost Savings and Improved Return on Investment
8. Other Benefits/Lessons Learned

The references at the end of this chapter list a number of useful resources, such as FHWA's Transportation Performance Management (TPM) Toolbox, which includes the TPM Guidebook and Resources (see https://www.tpmtools.org/about/). TPM measures and targets may provide grantees with a source of data to meet the grant performance measurement requirements.

Improve Safety

Table 7 presents a number of safety-related performance measures, organized by mode of transportation. While they are generally prioritized within each mode, grantees must consider the measures that are most relevant to their specific deployments. That is, the selection of performance measures will depend on the technologies being deployed and what problem(s) they are trying to solve. Careful thought should be given to the specific type of safety benefits that are anticipated from the technology deployment.

Nearly all of the PMs involve a measure of change (e.g., in crashes, fatalities or injuries), which is based on a comparison of data between a baseline (pre-deployment) period and a post-deployment period. The preferred type of measure is a rate, because it adjusts for the level of exposure; however, there may be cases where counts are the only data available (e.g., for bicycle or pedestrian measures).

FHWA adopted five safety-related performance measures as part of the TPM program. These include total counts for fatalities, serious injuries, and (as a separate category) fatalities and serious injuries to non-motorized road users, and rates per 100 million vehicle miles traveled (VMT) for fatalities and serious injuries. These categorizations are covered within the more detailed list of performance measures listed below. The Safety Performance Management Final Rule also established methodological guidelines for reporting these measures, which grantees may find useful.²

Grantees should consider the use of multiple measures to understand the safety impacts of their technologies. In addition to crash records or field test data on crash precursors, survey data can provide a complement (but not a substitute) to these other data sources, providing useful data on user (e.g., drivers, transit operators, etc.) experience or attitudes.

It is also important to consider the geographic scope when developing PMs. The measures included in Table 7 can be used at any geographic level (intersection, corridor, or region). However, it is important to note that as geographic scope decreases, random variation tends to increase, and thus intersection or even corridor-level analysis can be highly variable year to year. Any comparisons at these lower levels should be made with care. When reporting the performance measurement findings, grantees should clearly convey the geographic scope of the measures.

Table 7. Performance Measures for Improving Safety.

-	Performance Measures
-	Vehicle
1	(Rate) Crashes per Vehicle Miles Traveled (VMT)
2	(Rate) Fatalities per VMT
3	(Rate) Injuries per VMT
4	(Count) Number of crashes
5	(Count) Number of fatalities
6	(Count) Number of injuries
7	(Rate) Secondary crashes3 per VMT
8	(Count) Number of secondary crashes
9	Crash precursors (e.g., Time to Collision; Hard Braking) Refer to the literature on the relationship between the precursor event and actual safety outcomes (e.g., number or severity of crash)
10	Percent of drivers who feel more safe (i.e. from crashes) while driving [along X corridor] [survey/interview]
11	Percent of drivers who indicate that [X warning/feature etc.] is very or somewhat helpful. [survey/interview]
-	Transit
12	(Rate) Transit crashes per Vehicle Revenue Miles (or Passenger Miles Traveled) (PMT)
13	(Rate) Fatalities or injuries per Passenger Miles Traveled (PMT)
14	(Count) Number of transit passenger fatalities and/or injuries
15	Percent of transit vehicle operators who indicate that [X warning/feature etc.] is very or somewhat helpful [survey/interview]
-	Non-motorized
16	(Count) Number bicycle crashes, injuries, and/or fatalities4
17	(Count) Number of pedestrian crashes, injuries, and/or fatalities
18	Percent of bicyclists [or pedestrians] who feel more safe (i.e., from crashes) [crossing at X intersection/traveling along Y corridor] [survey/interview]

Improve Mobility/Reduce Congestion

This section highlights mobility and congestion related performance measures. The measures are organized by transportation mode, and are generally prioritized within mode. Grantees' selection of performance measures, however, will depend on the technologies being deployed and what problem(s) they are trying to solve. Careful thought should be given to the specific type of mobility benefits that are anticipated from the technology deployment.

Preferred measures include travel time, average speed, and travel time reliability (TTR). While TTR is important to travelers, there is no consensus within USDOT on how to measure it, so this document does not recommend a specific measure. Standard deviation of travel time (or travel time index) is the most common method for measuring TTR, but variance or other measures may also be used. The least preferred measure is vehicle volume or throughput as it does not directly measure mobility benefits.

In developing the list of suggested PMs for measuring ATCMTD mobility impacts (see Table 8), the TPM measures described in the National Performance Management Measures: Assessing Performance of the National Highway System, Freight Movement on the Interstate System, and Congestion Management and Air Quality Improvement (CMAQ) Program rule were incorporated.⁵

It is anticipated that grantees will be collecting the data to measure mobility/congestion benefits through field tests (i.e., new data collection), and possibly through modeling or simulation. Surveys may provide a complementary source of data on user experience or satisfaction, but surveys should not be a substitute for field test data.

In most cases, the performance measures can be used at the intersection, corridor, or regional level, and it is important to consider geographic scope when developing performance measures. For technologies deployed at intersections, grantees should consider measuring impacts both at the intersection AND the corridor or regional level, as the impacts may differ (i.e., the problem may have shifted from one intersection to another location).

Time of day should also be taken into account. In cases where mobility impacts are anticipated to be greatest during peak hours, the performance measures should focus on those peak hours.

Table 8. Performance Measures to Improve Mobility/Reduce Congestion.

-	Performance Measures
-	Vehicle
1	Travel time: (Rate) Vehicle hours traveled (VHT) per vehicle miles traveled (VMT)
2	(Count) Average speed
3	(Rate) Travel Time Reliability 23 CFR 490.507: Percent of person-miles traveled on the Interstate [non-interstate national highway system] that are reliable (ratio of the 80th percentile travel time to a "normal" travel time (50th percentile)). FHWA's National Performance Management Research Data Set (NPMRDS) is a potential data source for TTR, but grantees will need to assess the appropriateness of this data in meeting their evaluation needs.
4	Delay per trip (travel time) (Rate) per vehicle or per person (Count) average or total CMAQ Rule: Annual hours of peak hour excessive delay per capita (person-hours) Note: Delay accounts for difference between actual and free-flow travel time
5	(Rate) Vehicle volume/throughput (vehicles/hour)
6	Percent of travelers who report being very satisfied or somewhat satisfied with: Their travel experience [along X corridor] Their travel time [along X corridor] The reliability of their travel time [along X corridor] Travel speed [along X corridor] [survey/interview]
7	Percent of travelers who report that their travel experience along X corridor is (select appropriate measure): Less congested More reliable Takes less time [survey/interview]
-	Transit
8	Average run time for transit Note: Breaking data out by route can highlight particular locations with positive or negative impacts
9	On-time performance (% trips)
10	Total passenger delay or average passenger delay
11	Completion Rate for transit service
12	Percent of riders who are very satisfied or somewhat satisfied with the following aspects of service: Travel time On-time performance Frequency of service Location of stops Wait times [survey/interview]
13	Percent of transit vehicle operators who report that [travel time/travel time reliability/average speeds] has improved along the route [survey/interview]
14	Percent of transit vehicle operators who are very satisfied with [travel time/travel time reliability/average speed] along their route [survey/interview]
-	Pedestrian/Bicycle/Rideshare
15	Percent of pedestrian/riders who feel [travel time/on-time performance/etc.] has improved [survey/interview]
16	Percent of riders who perceive that the rideshare time estimates are very accurate or somewhat accurate [survey/interview]
-	Freight6
17	Port Turn Time, including: Reduced wait time (to enter terminal) Reduced terminal time (time in terminal)
-	Other
18	Incident Clearance Time7 (minutes)
19	23 CFR 490.707: Mode Share of Non-SOV modes (including telework)

For evaluations related to signalized control (including adaptive signal systems), specific performance measures which capture the ability of the control mechanism to respond to traffic and improve mobility should be considered.

Table 9. Additional Performance Measures Related to Signalized Control.

-	Performance Measures
-	Volume and Capacity
20	Saturation (by lane, approach, movement, or intersection)
21	Phase Termination Percent of terminations due to gap out, max out, etc.
22	Number of Stops
23	% Arrival on Green, % Arrival on Red
24	Purdue Coordination Diagram (qualitative)

Many of these performance measures (and delay and speed measures) can be automatically produced using automated traffic signal performance measure (ATSPM) software. Data from modern traffic controllers can be analyzed using ATSPMs, significantly easing the burden of analysis and visualization for some studies. FHWA promoted ATSPMs as part of the fourth iteration of Every Day Counts (EDC-4). Through a pooled-fund effort, open-source software was developed which can take controller log information and automatically produce a wide variety of performance measures and create visualizations and statistics using those data. Several States have implemented these systems, with Utah DOT among the early adopting agencies (see Utah DOT's ATSPM website: https://udottraffic.utah.gov/atspm/).

Reduce Environmental Impacts

When evaluating environmental impacts, the Program Level Report objectives include reducing transportation-related emissions. Analysis should include applicable mobile-source emissions of regulated pollutants that are known to have adverse public health effects, namely ozone precursors—volatile organic compounds and nitrogen oxides—as well as carbon monoxide, and particulate matter (both PM10 and PM2.5) and the applicable precursors from transportation sources. Reductions in energy consumption and carbon dioxide equivalent could also be reported.

Chapter 4 provides information about models and tools that can be used for emissions and energy measurement. Additionally, the References section on Emissions and Energy Measurement provides links and useful resources.

Table 10. Performance Measures for Reduced Environmental Impacts.

-	Performance Measures
-	Emissions
1	Net Project Emissions in kilograms per day (kg/day)8
-	Energy9
2	Energy Reduction in British Thermal Units (Btu)
3	Energy Reduction in Kilojoules (kJ)
4	Energy Reduction in gallons of fuel saved (gallons)

Optimize Multimodal Performance

Given the complex nature of our transportation systems, it is challenging to define and measure optimized multimodal performance. Below are a few suggested performance measures, including both quantitative and qualitative measures that provide insight on whether the system is progressing towards more optimal multimodal performance.

• Travel time, indexed by mode
• VMT avoided through transit or other modes
• Bike ridership
• Use of carpool/vanpool/rideshare
• Percent of riders who feel [travel time/on-time performance/etc.] has improved
• Interagency or inter-operator coordination - for example:
  • Number of meetings or other interactions;
  • Number/development of Memorandums of Understandings
  • Development and/or use of common strategies, response plans, etc.
  • Level of automation for common strategies or response plans
• Project team and/or other stakeholder feedback on how the deployment has optimized multimodal performance

Improve Access to Transportation Options

Accessibility (or access) can have multiple meanings. While the FAST Act does not explicitly define what it means by access to transportation options, this is typically interpreted as the existence of physical access to goods, services, and destinations (i.e. transportation) and/or the ease of reaching goods, services, activities, and destinations. Access can be measured from the supply side (does the system provide access) as well as the demand side (do users have access (or ease of access) to transportation options?).

Table 11 presents a range of measures related to improved access to transportation options, as defined above. The selection of performance measures will depend on the technologies being deployed and what problem(s) they are trying to solve. A number of the measures are specific to transit; however, others may apply across a range of transportation options, so the evaluation team will need to tailor the performance measure to their specific deployment.

Table 11. Performance Measures for Improved Access to Transportation Options.

-	Performance Measures
1	Number of households within ¼ mile of a public transit stop (or ½ mile of transit station)
2	Ridership (transit, ridesharing, bicycle, etc., as appropriate)
3	Number of (new) bicycle share/carshare programs OR Number of new partnerships/(memorandum of understanding) MOUs between transit agencies and transportation network companies (TNCs), bikesharing or other Mobility on Demand services
4	Number of new riders (people who have not previously used the mode) - either total over a period, or per unit of time (transit, ridesharing, bicycle, etc.)
5	Percent aware of different transportation options (or change in awareness) [survey/interview]
6	Percent reporting [X mode] improved their [travel experience/commute] [survey/interview]
7	Percent reporting it was very easy or somewhat easy to book/pay a ride [survey/interview]
8	Percent reporting it was very easy or somewhat easy to find the pick-up location for the [vehicle/rideshare/bicycle share/shuttle] [survey/interview]
9	Percent reporting the drop-off location (e.g., for bus/rideshare/shuttle) was very convenient or somewhat convenient to their final destination [survey/interview]
10	Percent of riders who found the [transit/rideshare/bikeshare, etc.] service affordable [survey/interview]

Effectiveness of Providing Real-Time Integrated Transportation Information to the Public to Make Informed Decisions

While there has been quite a bit of research conducted on advanced traveler information systems (ATISs), there is no standard set of performance measures that is used to measure the effectiveness of these information systems. Typically, research has relied on counting the number of users and/or surveying users to understand the characteristics of their use (e.g., when, how often, types of information sought, etc.), their satisfaction with the system, and the impacts of the ATIS on their travel behavior.

For projects that are providing the public with real-time integrated traffic, transit, and multimodal transportation information, use of the ATIS should be measured for all platforms (apps, website, kiosk, etc.). If possible, the types of information that users are accessing should be automatically recorded, along with other aspects of use, such as time of day and amount of time spent accessing the information. These data will provide useful insights; however, they will need to be supplemented with user surveys to understand the effectiveness of the ATIS. The table below provides suggested performance measures.

Table 12. Performance Measures for Effectiveness of Providing Real-Time Integrated Traveler Information.

-	Performance Measures
1	Percent using ATIS
2	Percent of users who used the ATIS to plan a multimodal trip [survey/interview]
3	Percent of ATIS users very satisfied or somewhat satisfied with the [accuracy OR reliability] of the real-time traffic, transit, and/or multimodal information. OR Percent very or somewhat satisfied with the accuracy (or reliability) of specific types of information (as appropriate): Incident information Construction information Road weather condition information Transit arrival times Parking availability information Terminal turn times [survey/interview]
4	Percent of ATIS users reporting that the real-time information has improved (select as appropriate): Their overall travel experience Their commute AND/OR Percent of ATIS users who feel the real-time traffic and/or transit information was useful [survey/interview]
5	Percent of ATIS users who made a change in travel (either before or during their trip) based on the real-time information provided: Percent who switched departure time Percent who switched their route Percent who canceled a trip [survey/interview]
6	Percent of users very satisfied or somewhat satisfied with: Location of kiosks Ease of using the kiosk [survey/interview]
7	Percent of transit vehicle operators who are very satisfied or somewhat satisfied with the real-time information: Re-routing information Special event [survey/interview]

Cost Savings and Improved Return on Investment

Cost savings may be measured in a variety of ways and the measures depend on the technology being deployed. This may be measured directly in dollars; if measured in time (e.g., staff time) it can be converted to dollar savings. Return on investment can be measured through a benefit-cost analysis (see Chapter 4 for more information).

Table 13. Performance Measures for Cost Savings and Return on Investment.

-	Performance Measures
-	Agency
1	Decreased operating expenses, such as: Decreased staff time for X activity (i.e., efficiency savings)
2	Decreased maintenance costs (e.g., due to improved asset management strategies)
3	Transit agencies may consider: Decreased costs per passenger (or per unit of time) Increased fare revenues earned Increased fare revenues per total operating expenses (recovery ratio) Vehicle revenue miles or hours
-	Public
4	Benefit-Cost Ratio or Net Present Value

Other Benefits/Lessons Learned

As needed, ATCMTD grantees should develop additional PMs that measure anticipated benefits that are not captured in the PMs presented in this chapter. Measures of other benefits may be quantitative or qualitative in nature. At a minimum, any surveys or interviews that are conducted should include an open-ended question that asks if there are "any other benefits" of the deployment (e.g., in addition to the safety and/or mobility benefits).

In addition, grantees should measure "lessons learned" from their deployments. While surveys may be used for this purpose, it is recommended that evaluation teams conduct at least a few interviews with key project stakeholders to gather lessons learned data. Interviews provide rich, qualitative data, and allow the interviewer to probe for more detailed information.

Finally, for new and emerging technologies, there may be additional measures that are not captured in the performance areas described above, but that are nonetheless important to measure—for example, user experience and/or acceptance.

A few example PMs for automated vehicle technologies are provided below (separately for riders and onboard controllers or maintenance staff):

Riders:

• Assessment of ride comfort (jerkiness, acceleration)
• Comfort level with AV technology and/or unstaffed operation
• Recommendations for improvements

Onboard controllers or Maintenance staff:

• Observations on passenger experiences/needs
• Issues or challenges with the technology
• Recommendations for improvement

Performance Measure References

Beaulieu, M. et al. (2014). WSDOT's Handbook for Corridor Capacity Evaluation, Olympia, WA: Washington State Department of Transportation, obtained from: http://wsdot.wa.gov/publications/fulltext/graynotebook/CCR14_methodology.pdf

Easley, R. et al. (2017). Freight Performance Measure Primer, Report No. FHWA-HOP-16-089 Washington, D.C., obtained from: https://ops.fhwa.dot.gov/publications/fhwahop16089/index.htm

Federal Highway Administration. (2017). National Performance Management Measures to Assess System Performance, Freight Movement, and CMAQ Program, Washington, D.C., obtained from: https://www.federalregister.gov/documents/2017/01/18/2017-00681/national-performance-management-measures-assessing-performance-of-the-national-highway-system

Federal Highway Administration. (2019). Transportation Performance Management, obtained from: https://www.fhwa.dot.gov/tpm/, last accessed March 2019.

Krugler, P. et al. (2006). Performance Measurement Tool Box and Reporting System for Research Programs and Projects, College Station, TX: National Cooperative Highway Research Program obtained from: http://www.trb.org/Publications/Blurbs/159957.aspx

National Cooperative Highway Research Program. (2008). Cost-Effective Performance Measures for Travel Time Delay, Variation, and Reliability, Washington, D.C., obtained from: https://www.nap.edu/catalog/14167/cost-effective-performance-measures-for-travel-time-delay-variation-and-reliability

National Transportation Operations Coalition. (2005). Performance Measurement Initiative, Final Report, Washington, D.C., obtained from: https://transportationops.org/publications/performance-measurement-initiative-final-report

Research and Technology Coordinating Committee. (2012). Identifying Potential Performance Measures for FHWA RD&T, Washington, D.C., obtained from: https://www.nap.edu/read/22816/chapter/3

Shaheen, S., Cohen, A., Yelchuru, B. Sarkhili, S. (2017). Mobility on Demand Operational Concept Report, Report No. FHWA-JPO-18-611, Washington, D.C., obtained from: http://innovativemobility.org/wp-content/uploads/Mobility-on-Demand-Operational-Concept-Report-2017.pdf

Smith, S., Bellone, J., Bransfield, S., Ingles, A., Noel, G., Reed, E., & Yanagisawa, M. (2015). Benefits Estimation Framework for Automated Vehicle Operations, Report No. FHWA-JPO-16-229, Cambridge, MA, obtained from: https://rosap.ntl.bts.gov/view/dot/4298

Taylor, R. (2010). PennDOT ITS Evaluations and Activities Final Report, Report No. FHWA-PA-2010-001-060908, Camp Hill, PA, obtained from: http://www.dot7.state.pa.us/BPR_PDF_FILES/Documents/Research/Complete%20Projects/Planning/ITS%20Evaluations%20and%20Activities.pdf

Vasconez, K. (2010). Traffic Incident Management (TIM) Performance Measurement Knowledge Management System, Report No. FHWA-HOP-10-011, Washington, D.C., obtained from: https://ops.fhwa.dot.gov/publications/fhwahop10011/tim_kms.pdf