Evaluation Methods and Techniques: Advanced Transportation and Congestion Management Technologies Deployment Program

Chapter 1: Introduction

The Fixing America's Surface Transportation Act, Pub. L. 114-94, (FAST Act) established the Advanced Transportation Congestion Management Technologies Deployment (ATCMTD) Program to make competitive grants for the development of model deployment sites for large scale installation and operation of advanced transportation technologies.¹ The primary goals of these demonstrations are to improve safety, efficiency, system performance, and infrastructure return on investment. The FAST Act requires the Secretary of the U.S. Department of Transportation (USDOT) to request applications each fiscal year (2016 through 2020) and to award grants to at least five and not more than ten eligible entities. The legislation also mandates reporting requirements for the grantees, including Annual Reports that describe how the project costs compare to the benefits and that provide data on the benefits and effectiveness of the deployments, as well as lessons learned and recommendations for future deployment strategies.² This Evaluation Methods and Techniques document is designed to assist grantees in designing and executing robust evaluations that enable them to fulfill the reporting requirements of 23 U.S.C. 503(c)(4)(F).

In addition to the Grantee Annual Reports, beginning three years after the first grant award, and annually thereafter, the Secretary must post on the USDOT website a report on the effectiveness of the grant recipients in meeting their projected deployment plans, including data on safety, mobility, environmental quality, system performance, as well as other outcomes.³

This Evaluation Methods and Techniques document seeks to provide grantees with a recommended set of performance measures that address these reporting requirements. To the extent that grantees use the same or similar measures to assess deployment outcomes, it will be possible to synthesize findings across projects and obtain a better understanding of the impacts of their deployments across multiple technologies and sites.

How to Use this Document

The Evaluation Methods and Techniques document contains information on a range of topics. In order to better guide readers to the information that will be most useful to them, a brief description of each chapter and the target audience is provided below.

Evaluation Overview (Chapter 2): This chapter provides a framework for designing evaluations, based on evaluation best practices. An emphasis is placed on performance measurement and linking performance measures to goals and objectives. While evaluation team members are likely familiar with the information, they may find the templates useful. In particular, both evaluation team members and project team members should consult the section on Annual Reports, which provides information for completing reporting requirements.

Performance Measures (PM) (Chapter 3): This chapter is critical to both the project team and evaluation team members. It presents performance measures for each of the goal areas of the ATCMTD program, as described in 23 U.S.C. 503(c)(4)(F,G). Where available, particularly for the areas of safety, congestion/mobility, and environmental impacts, the PMs are drawn from USDOT guidance. While the list is not exhaustive and use of these specific PMs is not required, it is highly recommended that grantees utilize the PMs presented in this chapter to satisfy statutory reporting requirements. This will enable USDOT to more easily synthesize findings across project sites for the Program Level Report.

Methods and Analytics (Chapter 4): This chapter includes three key topics: Benefit-Cost Analysis (BCA); Survey and Interview Methods, and Emissions and Energy Measurement. These sub-chapters are designed primarily for evaluation team members, as the information is presented at some level of detail. However, project team members may also find these chapters informative. Each of the following sub-sections also includes a set of references for more detailed information.

• BCA: This analytic method is the recommended approach for comparing project benefits and costs. The sub-chapter is largely based on USDOT guidance and provides best practices in performing BCA.
• Survey and Interview Methods: Best practice information is provided on a range of survey related topics, including sampling, sample size, recruitment, and questionnaire design, among others.
• Emissions and Energy Measurement: This chapter presents different methods and resources for addressing the measurement of emissions and energy and is geared toward projects that have identified this as one of their goal areas.

Technology Specific Best Practices (Chapter 5): This chapter outlines considerations and lessons learned with respect to evaluating specific technologies, namely: Adaptive Signal Control, Connected Vehicle Technologies, and Automated Vehicle Technologies. For projects that are deploying any of these technologies, the project team members as well as evaluation team members may obtain some useful insights regarding the evaluation of these technologies. A list of references for each sub-section is also included.

This document will be updated as appropriate, based on the needs of the grantees.