Measures of Effectiveness and Validation Guidance for Adaptive Signal Control Technologies

Chapter 1. Introduction

Purpose

The intent of this project is to identify, develop, demonstrate and document measures of effectiveness that can be applied to validate the achievement of traffic signal operations objectives. Adaptive Signal Control Technology (ASCT) is an operational strategy that applies tactics aimed at achieving singular or multiple operations objectives. Validation, in the context of this project, is a component of the systems engineering process that evaluates if the system implemented addresses the operations objectives that were articulated as needs during the development of the concept of operations for the system.

Figure 1. Diagram. Systems Engineering Process with Needs and Validation Processes Highlighted.

(Source: Federal Highway Administration.)

This report documents the tools and methodology developed for validation and summarizes the testing of this approach and measures on a field site in Mesa, Arizona where an ASCT system has been deployed for over one year. The intent of the field study was not to evaluate the Mesa ASCT system specifically but rather to demonstrate the application of the validation measures and methodology to a real world implementation of ASCT. The City of Mesa allowed the test phase to include approximately 30 days during which the ASCT was turned off and background coordination patterns were used instead. These 30 days were randomly distributed over the course of two months. Tube counters and Bluetooth detectors were deployed temporarily for volume and travel time data collection. GPS probe data and phase timing and detector status data were collected during the test period. Green occupancy ratio, percent arrivals on green, platoon ratio, and route travel times and reliability metrics were calculated for five intersections in the test area.

Background

The term Adaptive Signal Control Technology (ASCT), describes any system that collects data, evaluates traffic signal performance on the basis of one or more of the system’s functional objectives and then updates signal timing in response to that evaluation. Adaptive systems were initially developed in the United Kingdom and Australia in the late 1970s and introduced to the United States in the mid 1980s. Early deployments of Adaptive control systems in the United States were conducted as research or demonstration tests to evaluate their effectiveness and showed promising results. Compared to traditional systems, these ASCT were considered complex and difficult to maintain and operate due to communications and detection requirements. The FHWA sponsored the development of four adaptive control algorithms with the objective of reducing implementation cost and creating systems that were more consistent with US style traffic signal infrastructure.

ASCTs have not historically been widely implemented due their cost and complexity (perceived or real) compounded by a lack of clear documentation of benefits. In particular, the wide variation in cost, complexity, and performance reporting methodologies of evaluation studies for ASCT contribute to misunderstanding of the value and capabilities of available systems. Systems that were compared against poor existing timings get high marks and systems that were compared against excellent existing timings are perceived as having little value. Similarly, heavy oversaturated traffic conditions make challenging work for any signal timing approach, and failures of ASCT in such situations, in particular when they have been oversold to eliminate such congestion, tend to be notable.

As part of the Every Day Counts initiative, FHWA actively promoted the implementation of Adaptive Signal Control Technology (ASCT) to improve traffic signal system operations. A model system engineering process was tailored to facilitate the implementation of ASCT with the intent of addressing the risks that, when unchecked, have resulted in the failure or discontinued use of many ASCT deployments in the US. The model systems engineering documents discuss the development of agency objectives, a concept of operations, and system requirements to guide the design and implementation of ASCT. The second major component of the systems engineering process is then to verify that the selected system has met its stated requirements after system implementation and validate the system against the system’s performance objectives and goals.

Verification is largely an inspection process by which the agency observes that the procured system includes the features that were requested in the requirements. Validation is the process by which the features and capabilities of the system are measured against the stated operational objectives of the agency in procuring and deploying the technology. The intent of this project is to develop a generic validation process and tools for agencies to use to validate that selected ASCT meet their performance objectives.

Table 1 summarizes the connection between agency objectives, measures of effectiveness, and field data used to calculate those measures.

Table 1. Mapping of MOEs to Objectives and Data Sources.

MOEs	Data Sources	Operational Objectives (FHWA-HOP-11-27, PG 94, References 3.4.4)
Route travel time Route travel delay Route average speed Route travel time reliability	Import travel time data from Bluetooth scanner Import trajectory data from GPS probe	Smooth Flow Multiple objectives by TOD
Link travel time, delay Number of stops per mile on route	Import trajectory data from GPS probe	Smooth flow Manage queues Multiple objectives by TOD
Traffic volume on route Time to process equivalent volume	Import count data from tube counter file	Throughput Manage queues Multiple objectives by TOD
Percent arrivals on green, by link V/C ratio by movement Platoon ratio, by link Phase green to occupancy ratio by movement Reliability of phase metrics	Import high-resolution signal timing and detector data	Smooth flow Access equity Multiple objectives by TOD

This document is organized as follows:

• Introduction (Chapter 1).
• The role of validation in implementation of ASCT (Chapter 1).
• Operational Objectives of ASCT and signal operations (Chapter 2).
• Description of generic measures of effectiveness that can validate system objectives (Chapters 3 and 4).
• Validation Guidance (Chapter 5).
• Future Research and Development (Chapter 6).
• Literature review of validation and evaluation studies in ASCT (Chapters 7 and 8).
• Overview of MOE tools and processes (Appendix A and B).
• Field testing of the validation methodology (Appendix C).
• MOE evaluations and findings (Appendix D).
• Validation results (Appendix E).
• Supporting results and additional details from the field deployment (Appendix F).