Measures of Effectiveness and Validation Guidance for Adaptive Signal Control Technologies

Chapter 5. Validation Guidance

This project has developed tools and measures to assist agencies in satisfying the validation steps of the systems engineering process. This chapter summarizes the data and measures that address some of the operational objectives defined earlier. A few categories of important suggestions are then discussed for improving state of the practice in validation of agency operational objectives.

Summary of Data and Measures of Effectiveness to Validate Operational Objectives

This project has identified data and MOEs that can be used to validate some common signal operational objectives, both for ASCT operation and traditional signal timing. These measures and data sources are identified in Table 7.

Table 7. Identification of Data Sources and MOEs for each Operational Objective.

MOEs	Data Sources	Operational Objectives (FHWA-HOP-11-27, PG 94, References 3.4.4)
Smooth flow	Import travel time data from vehicle re-identification scanners Import trajectory data from GPS probes Import high-resolution signal timing and detector data	Route travel time Route travel delay Route average speed Link travel time, delay Number of stops per mile on route Percent arrivals on green, by link Platoon ratio, by link
Access Equity	Import high-resolution signal timing and detector data	Green-Occupancy-Ratio Min, Max, and Std. Deviation of GOR Served V/C ratio by movement Min, Max, and Std. Deviation of GOR
Throughput	Import count data from tube counter file	Total traffic volume on route Time to process equivalent volume
Travel time reliability	Import travel time data from Bluetooth scanner Import trajectory data from GPS probe Import high-resolution signal timing and detector data	Buffer time Planning time Min, Max, and Std Deviation of platoon ratio Min, Max, and Std. Deviation of percent arrivals on green

Additional common objectives such as shifting objectives by time of day, long-term performance reliability, handling of incidents and events, preventing oversaturation, and managing queues may be extended from this project in future work.

Smooth Flow Objective

The smooth flow objective is perhaps the most commonly studied and validated operational objective in suburban settings. This objective can be addressed with vehicle re-identification systems, GPS probe runs, and occupancy data from advance detectors connected to the signal controller. As discussed previously, each data source has benefits and limitations for computing performance. Vehicle re-identification systems can provide a wealth of data 24x7, but only on the point to point travel time. GPS probe runs provide more detailed information on link-by-link performance and can more easily pinpoint trouble areas, but are expensive to collect in order to generate a large data set. MOEs from the signal controller produce link-by-link performance 24x7 and also efficiently identify trouble spots like probes, but new methods are needed to aggregate these data into information about the performance of a route; and many signal controllers aren’t already equipped with such detection. The conduct of travel time studies and placement of detection is vital to obtain accurate validation results.

Configuration of Travel Time Routes and Conducting Travel Time Runs

The way that travel times are calculated is an important component of measuring a system’s performance. In particular, it is important that travel time trips have several key components:

• GPS Probe trips must begin before the first intersection considered “in the system” and upstream of any typical queuing at that location during the red interval. This is particularly important since holding a significant queue at the first intersection (and even worse, one that experiences phase failures) can have the effect of artificially improving the travel time on the rest of the route. This gives a false impression of the quality of travel along the facility, whether operated by an ASCT or traditional control methods.
• GPS probe trips should end after clearing the last intersection considered “in the system”. If the GPS probe application provided as part of this project is used for the data collection, the system will automatically discard any extra trip time past the last time point configured for the route, so drivers do not have to be as precise as to when they press the “stop” button after safely stopping their vehicle.
• Start of GPS probe trips should be randomized to have the first signal in both green and red phases an approximately equal amount of time, if enough trips can be executed within budgetary constraints. If not, all trips should start with the light being red for a random amount of time when the trip is started. Otherwise, as noted above, the queue delay at the first intersection is not captured in the reported performance.
• GPS probe trips need to follow the general speed of prevailing traffic. Drivers should behave as most drivers would, such as by passing slow-moving busses. If busses or other impediments are frequent and common along a route, however, those travel times should not be discarded in order to display only the best possible trips.
• Trips that occur during Oversaturation and events should not be immediately discarded. In particular, the occurrence of construction or other events that affect travel on a route for extended periods of time should be considered an opportunity to collect data during atypical situations, as discussed further in following sections.

Similarly, use of vehicle re-identification detectors for recording travel times requires careful placement and consideration of the range of the reader or placement of the detector, depending on the technology being applied. In the case of Bluetooth, most antennas provide a circular coverage zone which can result in initial identification of the vehicle on the exit side of the first intersection and/or re-identification of the vehicle on the entry side of the last intersection. This can skew the reporting of the travel times to exclude some of the queue delay in either or both situations. This phenomenon was demonstrated in this project since the Bluetooth travel times along Power Road were reported as significantly less than the travel times recording using the GPS probe method. Reliability estimates are also affected by excluding one or both of the queue delay conditions. If only one or the other method (i.e. either GPS probes or vehicle re-identification systems) is used in a validation effort, the comparative analysis of two or more operating conditions is not generally affected; but should be noted in the summary results. System vendors are aware of these issues and most are actively developing technologies and algorithms to improve their accuracy.

Access Equity Objective

Access equity is also a common operational objective. A balance between access equity and smooth flow operation is common in most suburban settings with some variation in agency and locality preferences. This objective can be addressed with detector occupancy and green time data from stop-bar detectors connected to the signal controller. The main challenge in many systems will be that some agencies do not utilize stop-bar detection for phases that are coordinated 24x7. If advance detection zones or loops are reasonably close to the stop bar, some anecdotal research indicates that GOR measures can be computed and compared with stop bar zones from side-street detectors at the stop bar. More research is needed to validate this further. Statistics (i.e. reliability) of the GOR and served V/C are also important metrics for determining the range of performance between operational strategies.

Throughput Objective

Throughput on a route is an important objective for many agencies. Rather than simply counting total traffic volume, measuring the time it takes to process a given number of vehicles provides a better measure of the efficiency of the traffic system. This objective can be addressed fairly simply with tube counters or other traffic counting equipment (video, laser, etc.) deployed at a specific location. In addition, counting detectors connected to the signal controller can also be leveraged for this purpose given they are located far enough from the stop bar so that queues do not habitually form on top of those zones. Exit detection is particularly suited for counting vehicles when the distance to the next intersection is significant. Since data is taken at a specific point, this measure addresses both through traffic and turning traffic and does not directly reflect throughput on a specific route. Additional techniques using O-D synthesis and multiple counting points are needed to extract likely route flows.

Travel Time Reliability Objective

Reliability of system operation is receiving increased focus in recent years with a variety of research and development projects as part of the SHRP2 research program (SHRP2 – Reliability). Travel time reliability can be measured using vehicle re-identification data, GPS probes, and detector occupancy data from advance detectors connected to the signal controller. For GPS probes and vehicle re-identification systems, buffer time is the primary measure of route reliability. Statistics of percent arrivals on green and platoon ratio can be computed from signal controller data to estimate reliability. As mentioned previously for the smooth flow objective, additional methods are needed to synthesize link-by-link statistics into reliability of performance along a route. Techniques for this type of data “fusion” are of high interest because they can reduce the cost of agency performance measurement significantly since no new field devices need to be deployed and expensive probe runs can be avoided (Although the advance or exit zones are still needed, which some agencies do not currently use). Vehicle re-identification systems pose a significant advantage for reliability estimation since they collect data 24x7.

Improvements to Validation Processes

Validation of operational objectives is not an event but rather a process. The data collection and processing tools provided in this project can help extend the typical traditional “evaluation” study into a process of on-going performance measurement. Additional suggestions to improve state of the practice in validation of traffic signal operational objectives are provided in the following sub-sections.

Data Collection During Off-Peak Periods

Validating the performance of traffic signal operational objectives during peak-periods is, of course, critical since these are the most important times of operation of any traffic control strategy. As shown in the test case during this project, performance of a new mode of operation that does not significantly exceed the previous mode of operation in a peak period does not mean that the investment in the technology was wasted. Significant performance improvements are achievable during off-peak times, when traffic conditions can be more variable that the predictable and heaviest demands during the peak periods. When budgets are heavily constrained for validation efforts, it is not surprising that peak period data are collected first. Approaches demonstrated in this project such as deploying temporary (or permanent) Bluetooth detectors or using the data collected by the traffic controllers directly can help to reduce the effort involved in analyzing and reporting performance during off-peak times.

Validation of Objectives Instead of Comparison with Existing Operations

Management of expectations is an important component of deploying and implementing any new technology. ASCT is particularly challenging in this regard since it carries a certain expectation from the decision-makers and the general public that traffic congestion will be magically wiped away. When the investment cannot be shown to improve operations by X% over the previous type of traffic control, in many cases a poor light is cast on the technology investment. The history of past validation and evaluation approaches also muddy the waters because often they either (a) intentionally make the “before” case as poor as possible or (b) intentionally make the “before” case as good as possible or even (c) update the signal timings to be as good as possible and then apply ASCT on top of the improved timings. Since most people are accustomed to discussing performance with respect to percentages, it is no wonder why there is uncertainty about the relative benefits of one technology versus another.

In this project we took some steps towards providing MOEs that could independently assess the performance of a traffic control strategy, without having to compare the strategy against something else; however, much more work is still needed to develop higher level performance objectives that combine measurements at multiple intersections and along routes to validate that a particular strategy is meeting the agency’s objectives for deployment. Such measures need to be objective ones, similar to the scores assigned by the HCM methods.

While it is probably unrealistic to expect that a validation methodology can be created that will eliminate the need for comparisons with a “before” condition, our guidance is that validation teams should at least consider explaining the quality of the timing parameters and operation in the before condition to identify where the problem areas lie as well as to identify where the operation is already acceptable.

Consideration of More Traditional Engineering Modifications that Cost Less

As noted in the preceding section, in the process of deploying ASCT it is common that “easy” fixes are identified for certain situations that really don’t require an ASCT. These fixes might require simple updates to signal timing parameters such as splits, offset, or sequence; repair of broken detectors; or providing a reliable time source to keep clocks synced on an arterial. These kinds of basic traffic engineering fixes cost a fraction of the deployment cost of an ASCT; however, these fixes may just create a need to perform additional tweaks in two weeks or two months. The value of ASCT, in that it continually re-optimizes settings, is unquestioned. The challenge is that the validation work is typically executed shortly after the installation of the ASCT, and the benefits measured at that time might just as easily been achieved by simpler, much less expensive investments.

Before launching into an investment in ASCT, agencies might also consider a program of check-ups by qualified personnel on a more frequent basis than the typical “once every three years” cycle that is common in the industry. Perhaps the MOE and data collection techniques developed and provided in this project can help agencies determine when their own staff needs to make those modifications. Some agencies may find, however, that the investment in ASCT systems is invaluable since the operations are continually being optimized and agency staff can focus efforts on their other public works duties, which might vary substantially on any given day.

Consider an ON and OFF Study

Traditional “before” and “after” validation processes present a host of challenges to the comparison of one traffic operations approach with another. The further apart in time that the before and after data are collected, the larger the potential discrepancy between the traffic patterns becomes. This is typically accompanied by an additional effort to validate using volume counters that the two traffic conditions were the same. If the two conditions are not statistically similar, the comparison of results becomes quite difficult to explain. An ON and OFF process mitigates some of this problem since it is not typical that traffic patterns vary significantly from day to day, as long as the number of ON and OFF days for each day of the week are approximately the same. We propose that ON and OFF testing during the deployment phase of an ASCT or new signal timing strategy should be adopted as a best practice whenever practicable.

There is considerable (and understandable) political push-back to turn “OFF” an ASCT (or change the timings back to the previous settings) that has been running for some time in order to validate its operation since the common perception is that the agency is intentionally making things worse. This dovetails with the identified need to develop standards for MOE performance that can validate system operation without the need to turn “OFF” the system at all. This is an important area of recommended future research and development, as identified in other sections of this report.

Consider Measurement of Failure Modes and Incident Conditions

Atypical traffic conditions and system component failures are elements of technology deployment that occur in the real world. ASCT systems by their very nature are better suited to handle anomalous traffic conditions than traffic operations strategies that operate with fixed parameters. Similar to the political concerns over ON and OFF approaches, there is frequent concern about the reporting of performance during incident conditions or the inducement of incident conditions through artificial means (e.g., traffic cones, staged lane blockages, etc.). If more objective MOE definitions can be identified that can assess ASCT performance during atypical conditions, it may not be necessary to intentionally create incidents and then turn the system “OFF” as well, in order to see how bad it really gets. If possible, planners should try to use the data collected during real incidents in a meaningful manner. Performance results should probably be excluded from the calculations for MOE averages, but there is important information in the way the system reacts to the situation that should not just be discarded. At minimum, simply observing the modifications made by the system is important information about how the ASCT is attempting to address the situation. This is also an important recommended area for future research and development since it was not studied in detail in this validation project.

From the perspective of component failures, it is absolutely essential to identify what is going to happen when one or more ASCT components is off-line. Does this system fail “free”? Does it go back to coordinated operation? With what coordination parameters? Worst of all, some ASCT systems can get “stuck” in certain phases of signal indications for extended periods of time or induce signal flash. These situations (which can also be created by traditional signal operations), are to be avoided at all costs since they generate phone calls at an extraordinary rate and mobility performance is damaged significantly. Testing and validation that the ASCT fails to generate an appropriate type of operation should be a central component of any validation study. This is particularly important when deploying any system that includes a new type of control method, a new controller interface, an “upgraded” piece of hardware or firmware, a new detector type, or anything else that has not been sufficiently proven elsewhere.

Consider a Longer-Term Program of Performance Monitoring

The value of ASCT systems and managing traffic signal operations to performance objectives lies not only in the initial impacts but in its long-term function of continuing to adapt to changing traffic patterns over months and years. We recommend that if possible, agencies treat the validation step as a program of continual, or at least periodic, activities, rather than a singular event. In signal system deployments, there are typically two components of the systems engineering process: the acceptance test, which verifies that the procured features are available and function as intended, and the “burn in” period where the operation is checked for several months to identify if any systemic issues arise due to the passing of time. A similar approach is recommended here, but extended much further into the future. The measures and tools provided in this project can help enhance the ability to conduct “check-ups” over time. This recommendation is in line with the over-arching goal of MAP-21 to move agencies towards on-going performance measurement as part the core mission of traffic management agencies.