Measures of Effectiveness and Validation Guidance for Adaptive Signal Control Technologies

Chapter 3. Methods for Validation of ASCT Deployments

Over the years there have many attempts at measuring performance associated with different adaptive signal control technologies. Most of the literature has been produced by agencies, consultants, and academics. A summary of representative studies done by academics, consultants, and agency staff is presented in this section. A more detailed literature review is provided in Chapters 7 and 8.

Most studies spend a substantial amount of time describing the corridor characteristics and much less time describing existing signal timings and common areas of substandard performance. Furthermore, most studies do not articulate or analyze the specific modifications made by the ASCT that resulted in the improvements to the conditions as reported by the MOEs.

Most studies have focused on the key metrics associated with standard traffic operations – stops, delay, and travel times. Some studies have also tried to include safety effects including crashes. Collecting route travel time data is the most common evaluation approach for traffic studies. In most studies, multiple “probes” travel the corridor collecting start time, time at which each signalized intersection is encountered, and the time to reach the stopping point of the corridor. The collection of travel time data using probes requires a careful consideration of the number of travel runs necessary to be considered statistically significant, but many studies estimate the runs based on reasonableness and cost. These relatively low numbers of runs can be used to compare averages, but are not effective in assessing improvements to travel time reliability that may result from application of ASCT. Further, when budgets are limited travel time runs are most commonly collected only during peak periods which may not reflect the capability of the ASCT to improve conditions during the shoulder and off-peak times, where ASCT may actually have more impact when demands are more variable.

In addition, travel-time varies with traffic volume. Some studies have considered volumes in validating that two conditions are statistically similar, but volume is not typically used to augment the travel data to compare how before and after treatments vary with volume. Newer approaches such as (Fehon et al., 2010) use volume data explicitly. An estimate of total stops can be also be made using the probe data and the route volume.

Many studies report only arterial end-to-end travel times and neglect collection of data on routes that have different origins and destinations in the system and combinations of turning movements. Only a limited number of studies have considered multiple paths through an ASCT network (Hunter et al., 2006). When vehicle re-identification technologies are used instead of probe vehicles, stops cannot be computed, so reductions or changes in delay and travel time are only typically reported.

Most studies augment route travel times with collection of a limited amount of side-street performance data using traditional manual observation techniques – counting vehicle queues and estimating delays. Due to the manual labor involved, all studies are limited in the duration of the data collection due to project budget. Pedestrian delay is typically measured using a stopwatch technique.

These simple techniques are effective, but just cannot be used for long periods of time due to cost and the need of humans to take breaks. Videotaping can alleviate some of the need for on-site observers and allow some “fast-forward” time savings, but still cannot be used to evaluate a system for extended periods. High-resolution phase timing and detection data that is now becoming available from controllers, ASCT, and other signal systems can be used to reduce the manual effort to collect such measures. Video-analytic methods are emerging but have yet been proven reliable.

Studies that report queue lengths as performance measures are almost always counted manually with observers. NCHRP 3-79 recommends use of videotaping and manual post processing. A few vendor technologies are emerging that claim the capability to count turning movements and queue lengths automatically from video cameras images. Such systems have not been evaluated and validated extensively enough to consider such automated methods as part of this process at this time.

Almost all studies approach the data collection efforts in a “before” and “after” format and the data collected is quite limited in duration. In particular, studies neglect the collection of measures that reflect the ASCT capabilities to modify its operation to efficiently accommodate variations. It is also not uncommon to collect the “before” conditions and “after” conditions with several months of time in between the two studies. Over this time, travel demand can, and typically does, change due to a variety of reasons, such as site development and seasonal changes. It has also been estimated that signal timing performance degrades approximately 3 percent per year, so waiting longer to collect “after” data will typically show more substantial improvements.

To get around the issues related to before/after studies, several studies have begun to study performance using on/off techniques. While an on/off approach may be more scientifically defensible, such a study is more difficult to support politically by the agency owners. Laypeople and nontechnical stakeholders frequently view the intentional disabling of a technology as imprudent.

The largest contributor to the uncertainty about the benefits of ASCT is due to the quality of timings that the new system has been compared to. Some reports of hugely successful deployments (90 percent reductions in stops, etc.) have compared the ASCT to poorly configured or significantly outdated timings. Other studies report only modest improvements due to ASCT. These studies most commonly compare the ASCT to recently optimized timings or timings that are already largely suitable for typical conditions. Traffic engineering principles are based on sound theory, so there is no reason to assume that an ASCT can outperform traditional signal timing under stable-flow conditions. Both situations are accurate assessments of ASCT value, but tend to inappropriately distort comparisons between systems when they are used to decide which system to implement.

Reporting percent differences contributes possibly the most to the uncertainty of ASCT performance since percentages amplify differences in small numbers. While percentages are easy for human brains to process, there is definitely a need to identify reporting methods that allow fairer comparisons of performance. Some evaluations have used other ways of aggregating performance (MnDOT, 1995; NCHRP 03-90, in press) such as accumulating the number or percentage of links that were “better,” “worse,” or “same” into bins by time of day and direction of travel. Other studies have explored similar aggregation methods and data presentation methods for summary performance reporting (Papamichail et al., 2009; Pesti et al., 1999).

Most studies also make an attempt to extrapolate the performance improvements of a given “after” performance to a benefit/cost ratio and to compute aggregate impacts to emissions and fuel savings. These extrapolations typically assume that the percentage savings would accrue at the same rate for the system life cycle, which is probably not accurate since most agencies retime signals on some schedule (three to five years) or due to repeated trouble calls

Finally, it was found that most agencies and evaluators did not articulate specific objectives or provided targets for performance on certain measures.

Summary of Literature Review

From the literature, the following common themes for best practices and pitfalls to avoid were identified (see Table 2). The recommended validation approach will use the best practices and address the common challenges.

Table 2. Issues Identified in Literature Review.

Issue	Accommodation
Limited articulation of operational objectives	Validation of ASCT effectiveness in meeting specific operational goals.
Side-street performance usually measured manually	Use high-resolution signal phase and detector data to estimate performance directly and 24/7.
Limited number of probe data runs	Combine probe runs with vehicle re-identification equipment to fill in gaps 24/7; smartphone apps for data collection with only one person.
Limited/no focus or study of abnormal conditions and incidents	Stage/simulate abnormal conditions if mitigation is an objective.
Limited/no focus or study of pre- and post-peak-period performance	Recommend measurement of new metrics for pre- and post-peak congestion management performance.
Presentation of percentage improvements skews comparisons when “before” timings are poor	Use measures that can be compared “apples to apples.”
Separation of data in before and after time periods	Use ON/OFF or ensure that after data is collected as close as possible to before data collection.
Limited use of volume data for aggregate performance assessment	Provide methodology for estimation of variation in performance and “total” system performance using volume data collection.
Queue performance usually measured manually	Provide automated methods for queue estimation from high-resolution signal phase and detector data.
Extrapolation methods for B/C estimation make unrealistic assumptions	Provide recommended methodology for more accurate B/C estimation.
No emphasis on reliability of performance	Provide recommended methods to estimate reliability.