Measures of Effectiveness and Validation Guidance for Adaptive Signal Control Technologies

Appendix E. Finding for Validation of ASCT Operational Objectives

The previous chapter presented specific results for the various MOEs that were tested in this project. As presented in the previous chapter, the analysis of various results indicates that the ASCT and coordination operation have very similar characteristics for this particular deployment location. Neither operation was found to be uniformly more effective than the other at performance on the collected MOEs when assessed at aggregate levels typical of most validation efforts. However, the goal of validation is not to generate a specific percentage or level of improvement; the goal of validation is to determine that the strategy or tactics meets the operational objectives of the agency in implementing the strategy. If the existing strategy is acceptable and the new strategy cannot improve upon it, this is not a “failure” of the new strategy; it most likely means that the existing operation is largely acceptable. On the other hand, it can mean that neither operation is performing acceptable for the agency objective. Some analysis of specific findings is presented here to illustrate some of the challenges in interpretation. Again, our goal in this project is not to evaluate the specific system in Mesa, Arizona but rather to demonstrate how each MOE can be used to validate a common agency objective for deploying ASCT.

Findings for the Access Equity Objective

Both types of operation meet the agency objective to provide access equity at each of the intersections as measured by qualitative review of the GOR values. At most intersections during most periods of the day, the maximum values of average GOR were lower than coordinated operation, indicating that the ASCT modifies the split timings more often to reduce the possibility of phase failures. This indicates that the ASCT is indeed reacting to the changes in the traffic flows, which is not possible with coordination timings with fixed split values, although in most cases the gap-out logic of the coordinated operation still provides acceptable performance.

In some cases, GORs were found to not be as balanced as possible during ASCT operation because of the influences of other system users, primarily pedestrians. For example, the crossing phases (2 and 6) at the critical intersection (Power & Southern) were extended significantly past the time needed for crossing traffic in order to time the pedestrian clearance. This results in significant queues on the coordination phases (4 and 8) and lower GOR values for the crossing phases (2 and 6) due to the extended phase durations. It seemed plausible that the ASCT should have increased the cycle time to lower the degree of saturation on the coordinated phases and bring the levels of service of all phases more in balance (in the process improving progression for the north and south (4 and 8)), but it did not. Why not? This is a very difficult question to answer since there are a number of components to the decision making process in any ASCT that are unobservable.

Assuming that we are not observing issues that are the result of programming errors or logical faults (in this case, the ASCT is quite mature and unlikely to include basic logical or algorithmic errors), it is more likely that the un-common-sense outcomes (a) are due to the complexity of the decision making process, which cannot easily be explained in the documentation, (b) may be too complicated for simple “if-then” descriptions, or (c) are influenced by thresholds, parameters, and configuration settings that we have not considered.

In this deployment, agency staff were not accustomed to “playing” with the parameters and setup of the system (most of the staff involved in the procurement and installation of the ASCT system have since taken other positions with other agencies or retired) and were certainly hesitant to start modifying settings of the system when they viewed the operation as largely acceptable. Since the project was ongoing, any tweaking of the operation would have created a third regime to be evaluated (coordination, original ASCT settings, and modified ASCT settings). The good news, however, is that if the agency decides to study the before and after differences of parameter changes, they can relatively quickly and inexpensively assess those modifications with the tools and methodology developed in this project from the data collected and uploaded from the controllers. GOR values identify acceptability of phase durations to meet traffic demand. As an agency starts the process of optimizing a system to meet certain performance objectives, the changes are easily revealed by observing the GOR values obtained from the high-resolution signal timing data without extensive additional deployment of observers in the field. Consistently high GORs can easily indicate the need for more split time on a phase. Consistently low GORs can easily indicate the need for less split time for a phase. Care should be taken in the case of low GOR values since there are can be other influences that result in low GOR such as the need to cross a barrier with a phase in another ring or because of pedestrian actuations.

Findings for the Pipeline Objective

While both actuated-coordinated operation and the ASCT perform adequately to provide access equity, the pipeline objective on the main north-south arterial in the system could not be qualitatively validated for either type of operation. With over 2.5 stops/mile on average in both directions of travel on a route that is less than a mile in length, the pipeline objective is clearly not being achieved. In this system, there is a freeway interchange in the mix, which creates some additional complexity due to the heavy off ramp flows at certain times of day. Even so, both types of operation generally had poor progression performance at the critical intersection for both north and southbound travel as revealed by all of the pipeline-related MOEs. Arrivals on green, stops per mile, platoon ratio, and link and route travel times were shown to all be acceptable MOEs that can validate that an operational strategy meets or does not meet a pipeline objective. The vehicle re-identification method is weakest at pinpointing specific areas for improvement but the 24x7 coverage can provide a wealth of observations for analysis. Probe travel time runs provide the richest set of details about specific characteristics of the route performance, but are expensive to collect in order to generate a reasonable performance sample. Percent arrivals on green and platoon ratio measures require advance detection on approaches to intersections and provide link by link view of progression performance. Methods are still needed to combine these link MOEs together to generate route MOEs.

Findings for the Travel Time Reliability Objective

Significant differences in reliability of travel times for several routes were identified using both GPS probe runs and Bluetooth travel time detectors. Coordinated operation was shown to have lower buffer times and thus higher reliability for several of the routes. The northbound route on Power Road with ASCT was found to be more reliable than coordinated operation, which correlates with other findings that the ASCT favors northbound travel at the expense of southbound traffic. While it was theorized that we might find that the ASCT was able to improve reliability even when the average travel time might be slightly worse than coordinated operation, this was not the case. This reinforces the case for traditional traffic engineering approaches in situations with steady, predictable traffic flows, because appropriately configured coordinated operation can be superior to, if not as least as effective as, ASCT systems with their fluctuating natures. Buffer time was shown to be an acceptable metric for validating that an operational strategy can satisfy a reliability performance objective. Buffer time is easiest to compute when there are many observations in a time period, such as with vehicle re-identification systems. When computing reliability metrics for travel times from GPS probe data, the main challenge is collection of enough trips in a time period for capturing a reasonable estimate of the true buffer time. Methods based on Bayesian statistics are likely necessary (Feng, et al, 2012) when the number of probe trips are very low.

Findings for the Throughput Objective

Using the data from the traffic volume counters, the ASCT objective was shown to induce measurable improvements to the throughput performance at several locations. These differences cannot be explained by randomness in the vehicle flows or systemic differences in the ON and OFF data sets, since the measurements were averaged over many days of data collection and using the same days of the week in both ON and OFF conditions. With respect to several of the point locations, at least, this supports the theory that ASCT can increase throughput by subtly modifying the splits of a given phase, cycle by cycle, thus reducing the number of phase failures.

In at least one location, the improvement was manifested during all times of day, but particularly when the traffic flows transitioned from PM peak to post-peak conditions, the ASCT operation tended to process vehicles more quickly to the desired target volume than coordinated operation. This corroborates the often conjectured ability of ASCT to be able to curb the peak period performance condition sooner (which can typically be quite poor, with standing queues and significant congestion) than coordinated operation. Since traditional signal operations do not vary the splits and other parameters with respect to traffic demands, the small changes made by ASCT can have a positive effect in improving throughput. Perhaps this suggests that some coordinated systems should be transitioned from a pipeline objective (coordination) to an access equity objective (free) sooner than typically scheduled in most fixed-parameter operations; it is more indicative, however, of the capability of ASCT to balance both objectives (access equity and pipeline) together during the shoulder periods, since both flows (left turns from the side-streets and through flows from the main line) contribute to the total throughput at the outbound measurement locations.

Measurement of total flow across a point location over a given time period was shown to be an acceptable MOE for determining that a signal timing strategy meets the objective of improving throughput. Care must be taken to average the results over many days and similar days of the week to avoid making erroneous conclusions that are due only to fluctuations in traffic demand or differences in traffic patterns.

Qualitative versus Quantitative Validation

We have been careful to use the words “qualitative” validation in the preceding sections on analysis of the findings, because there is no existing quantitative specification for what GOR values or percent arrivals on green at an intersection or along a route constitute acceptable performance for a given objective. These criteria need additional research and development, perhaps starting from the concepts established from the HCM criteria of average delay for a phase or average speed on a link. In particular, standards need to be established for evaluating performance without the need to compare a certain type of operation with what was being done before. Comparing ASCT performance with existing performance is the largest contributor to uncertainty of benefits because of the variation in baseline performance across jurisdictions.

HCM criteria measure the level of demand as much the quality of service. A particular type of operation has little chance of obtaining an “A” score if the level of demand is simply too large. Similarly, an objective to “manage queues” may mean providing a certain level of throughput on a particular direction of travel. But to what extent can such an objective be considered validated without comparing the operation to some other type of control? Because there are no criteria for the speed with which standing queues should be dissipated once they begin to grow, we are left to reason that if a certain type of operation improves upon the previous operation, it is superior, and if it does not, it was a waste of funds, time, and effort.

As has been shown in this particular test, there are operational regimes in which ASCT is simply not able to improve significantly upon traditional coordinated (or “free”, for that matter) operation, when the traffic flows are predictable and stable and the signal timing parameters are set appropriately. This is not a strike against ASCT, but rather a strong endorsement of the fact that traffic engineering has evolved to address the cumulative feature set of actuated-coordinated operation that functions well under minor flow fluctuations. When the fluctuations exceed the bounds of actuated-coordinated assumptions, and there are valid trade-offs that can be made, there is no doubt that ASCT can improve system operation. How these performance improvements can be quantitatively validated is still elusive. In many past studies, such situations are simply not measured (indicating that ASCT systems have “no impact”) and in others those situations are easily identified or induced (indicating that ASCT systems have “enormous impact”). What is needed are ways to quantitatively demonstrate that ASCT as well as traditional signal timing strategies are handling anomalies in an efficient manner and meeting agency operational objectives.