Evaluation Methods and Techniques: Advanced Transportation and Congestion Management Technologies Deployment Program

Appendix A: Evaluation Plan Checklist

This checklist is designed to assist ATCMTD grantees in developing their Evaluation Plans. Example templates that grantees can include in their Evaluation Plans are also provided. The templates are a tool for summarizing key evaluation-related information at a high level; the Evaluation Plans should include more detailed descriptions of measures, methods, and procedures.

If you have any questions about the checklist, please contact Margaret Petrella (Social Scientist, The Volpe Center) at Margaret.Petrella@dot.gov.

Evaluation Plan Topic

Project Overview

Describe the project and highlight the technologies being deployed
List the project stakeholders (project team, partners, evaluation team) and describe roles and responsibilities, particularly with respect to completing the evaluation
Summarize what constitutes end-of-project successes
Provide a deployment and evaluation schedule in terms of months and years; include project milestones

Evaluation Goals/Objectives and Evaluation Questions¹ (see Table 20 below)

Describe project evaluation goals and/or objectives and associated evaluation questions
Develop at least one evaluation question for each goal or objective; multiple specific, evaluation questions are better than a few general ones

Performance Measures (PMs) (see Table 21 below)

Identify one or more performance measure(s) for each evaluation question
Ensure (describe how) you are meeting the performance measures prescribed in the FAST Act
Develop system performance measures that measure whether the technology is functioning as intended (i.e. to verify the functionality of the technology)
Ensure (describe how) your PMs are measurable within the scope of the evaluation. If targets are described, ensure they are appropriate
Think about the unit of analysis (metric) needed for your analysis
Describe the data sources for each PM (include existing data sources as well as primary data collection). If your agency is uncertain about the data sources or elements, indicate what data you would need to measure the PMs and note that updates to the plan will include more details on “X”

Evaluation Methodology (see Table 21 below)

Describe the method(s) that will be used to address each evaluation question (likely a mix of quantitative and qualitative methods)
- Describe the experimental design, as appropriate (before-after; treatment-control)
- Describe potential confounding factors
Ensure the evaluation design enables the measurement of the proposed PMs; identify the specific data elements that are required
Describe any limitations or risks associated with the method or the data elements
If multiple technologies are deployed, be clear how the different technologies will be evaluated

Data Collection Procedures and Data Management

Describe how the data will be collected, including any plans for a pilot
- For example, for surveys, plan should include: general method of recruitment; sample size; potential survey topics
- For field studies, plan should include: location, data collection frequency, data collection period
Address data management (e.g., data logging and transmission to the evaluation team (if applicable); data storage; data access and privacy protection; data fusion (if applicable), data quality checks, etc.) [note: for existing systems, less detail may be needed]

Table 20: Example TEMPLATE for Evaluation Goals, Objectives, and Evaluation Questions
Goal Area	Objective	Evaluation Questions
Improve Safety	Reduce Traffic crashes	To what extent has Connected Vehicle (CV) application X reduced traffic crashes along corridor X?
Improve Safety	Reduce Traffic crashes	What proportion of drivers using CV application X rated the safety warnings as helpful?
Reduce Congestion/Improve Mobility	Improve travel times	What impact did adaptive signal control have on travel times along corridor Y?
Improve Effectiveness of Real-Time Integrated Transportation Information	Provide the public with access to real-time integrated traffic, transit, and multimodal transportation information to make more informed travel decisions	Did a majority of application users indicate that the travel time information helped improve their commute decision making?
Cost Savings and Improved Return on Investment	Provide cost savings to transportation agencies	What was the benefit-cost ratio of the adaptive signal control deployment?
Share Institutional Insights	Lesson Learned	What lessons learned did project managers identify to facilitate future successful deployments of CV?
[ADD]	[ADD]	[ADD]

Note: Examples are included for illustrative purposes only. Additionally, there may be multiple performance measures per Evaluation question.

Table 21: Example TEMPLATE for Evaluation Performance Measures and Methodology
Evaluation Questions/Hypotheses	Performance Measure	Information Source/Method	Data Element	Limitations/Constraints
What proportion of drivers using CV application X rated the safety warnings as helpful?	% of respondents who feel safety warning was helpful	Survey	Survey response in post-survey	Low response rate may be an issue
What impact did adaptive signal control have on travel times along corridor Y?	% change in average travel times	Field test (vehicle probe data)	Pre-post comparison of vehicle probe data	Weather, incidents may affect measurement
What lessons learned did project managers identify to facilitate future successful deployments of CV?	Lessons Learned	Interviews	Responses to questions about lessons learned	Findings for one project may not generalize to other locations
What was the benefit cost ratio of the adaptive signal control deployment?	Net present value	Benefit-cost analysis	Monetized estimates of project impacts	Incomplete data Some impacts are difficult to quantify
[ADD]	[ADD]	[ADD]	[ADD]	[ADD]

Note: Examples are included for illustrative purposes only. Additionally, there may be multiple performance measures per Evaluation question.

¹ Evaluation questions may take the form of hypotheses. [ Return to Note 1 ]

Previous | Next