Transportation Management Plan Effectiveness Framework and Pilot
Chapter 3. A Framework for Measuring Work Zone Transportation Management Plan Strategy Effectiveness
DEFINING THE DESIRED OUTCOME OF A TRANSPORTATION MANAGEMENT PLAN STRATEGY EFFECTIVENESS EVALUATION
Evaluating the effectiveness of a TMP strategy implementation can take several forms, depending on time and money available for evaluation, skill set of the evaluator(s), and most importantly, the desired outcome of the evaluation. This has resulted in multiple approaches and scopes being used for evaluating the effectiveness of TMP strategies.
In simple terms, evaluation outcomes are defined by the questions they try to answer, such as:
- Do we think this TMP strategy (or set of TMP strategies together) had some type of effect upon mobility, safety, customer satisfaction, and/or agency or contractor productivity and efficiency? In simplest terms, do we think this strategy was "effective?"
- How much of an effect did this TMP strategy/set of strategies have upon mobility, safety, customer satisfaction, and/or agency or contractor productivity and efficiency MOEs?
- How does the effectiveness of this TMP strategy/set of strategies upon mobility, safety, customer satisfaction, and/or agency or contractor productivity and efficiency vary as a function of differences in roadway, traffic, and work zone characteristics?
An answer to the first question would be much simpler to obtain than an answer to the third question.
COMMON APPROACHES TO ASSESSING TRANSPORTATION MANAGEMENT PLAN STRATEGY EFFECTIVENESS
From a practitioner’s perspective, multiple ways exist to assess whether a TMP strategy had some type of effect on one or more measures used to assess work zone safety and mobility impacts. For example, a previously-published report regarding the feasibility, usefulness and possible approaches for assessing TMP strategy effectiveness described three possible approaches: (5)
- Qualitative assessments.
- Quantitative assessments.
- Some hybrid of the two.
Qualitative assessments rely on subjective statements (perhaps a simple "it worked well" or "it didn't work well" statement, along with an explanation) regarding a TMP strategy or strategy package. Those providing the qualitative assessment can be the traveling public, a work zone expert, or even project staff. Even positive or negative media reports about a project or series of projects are a type of qualitative assessment. Qualitative assessments are most often used when assessing the effectiveness of strategies that are not easily evaluated with quantitative methods, such as the effect of public information on motorist satisfaction, or when actual data are not available for a quantitative assessment. However, qualitative methods also tend to yield varying results from person to person because of personal biases and other factors, making comparison of strategy effectiveness across projects or regions difficult.
Quantitative assessments, on the other hand, utilize data-driven MOEs to evaluate the effectiveness of TMP strategies and strategy packages. Historically, agencies have not utilized quantitative assessments significantly for TMP strategy effective evaluations, due to the lack of available data, the high cost of obtaining such data, and complexities of how to best perform a quantitative assessment using that data. In those few instances where data and quantitative evaluation did occur, the MOEs used and reported varied and so the results across studies also have not been easily aggregated.
The hybrid approach as described in reference 5 combines qualitative and quantitative assessments to try to counter each of their weaknesses. Typically, qualitative information (public opinion surveys, agency staff feedback) are obtained to gauge effectiveness of a set of implemented strategies in general terms, while quantitative assessments assess certain strategies in more detail. It is important to recognize that this approach does maintain some subjectivity in the analysis that can influence conclusions. For instance, it could be difficult to determine the effectiveness of multiple strategies of a TMP that were implemented simultaneously with either a qualitative or a hybrid approach.
SCOPES OF TRANSPORTATION MANAGEMENT PLAN STRATEGY EVALUATIONS
The previously-published report (5) also discusses five scopes of evaluations that have, or could, be performed to assess TMP strategy effectiveness:
- Full-scale evaluation of all strategies on a project.
- Agency-wide evaluation of a single TMP strategy.
- Research evaluation of a single strategy implemented by several agencies.
- Case study of a single strategy at one location.
- Process review.
Full-scale evaluations can provide an indication about how well the collective set of TMP strategies implemented was able to manage and mitigate the impacts of the work zone. Qualitative assessments by project staff or the traveling public can provide a general sense as to whether or not the impacts were tolerable or within the expected range. Meanwhile, quantitative assessments may include an alternatives analysis of expected impacts (usually via simulation) of various possible TMP strategies and strategy packages, or may involve a comparison of actual impacts during the work zone to what existed prior to the start of the project or to what had been predicted to occur if the strategies had not been implemented. Typically, these types of evaluations are performed for only a limited number of projects due to a high level of effort required to accomplish them.
Agency-wide evaluations of a single TMP strategy involves the implementation of the strategy of interest across multiple projects and assessing (qualitatively or quantitatively) its effect on work zone impacts. This approach usually means that the strategy of interest is combined with other strategies and different roadway and traffic characteristics, which provides insight as to the effect of the strategy across the range of other confounding strategies and site characteristics. These types of evaluations can also be data and labor intensive, and so are used relatively infrequently by agencies.
Research evaluations of a single strategy implemented by several agencies are similar to, but slightly differ from, the agency-wide evaluation of a single TMP strategy. For one thing, this type of evaluation usually involves consultants or researchers who have increased evaluations skills. Also, this approach can introduce differences in how various agencies select, implement, and evaluate a strategy into the assessment. The range of information collected through this evaluation scope can enable the evaluation to provide a broader review of the strategy and maximize the applicability of the assessment. However, this evaluation scope may produce conflicting results if there are confounding variables influencing travel behaviors across the projects that cannot be accounted for in the analysis. This concern can also be present when performing agency-wide evaluations of a single TMP strategy.
Case study of a single strategy at one location allows for study depth in terms of how the strategy of interest performs under finite field conditions. An example of this would be the evaluation of the speed-reducing effects of a speed display sign implemented at a particular location. If properly designed, this type of evaluation could also assess how the strategy may be affected by other implemented strategies at the site. While case studies can yield highly-precise measures of strategy effectiveness at that location, they are not widely accepted by practitioners as a realistic evaluation tool for specific TMP strategies due to the concern that the results may apply only to that location and not be replicated at other locations.
Process reviews related to TMP strategy effectiveness are a broad-based assessment of the use of various strategies within an agency. For example, the agency may be able to define statewide trends in TMP practices and strategies, which could lead to adjustments in policy, guidelines, and required training of employees and its contractors. This broad-based assessment could reveal important information that would not have been discussed if only the specific TMP strategies were evaluated at the project level. However, this type of evaluation would require an agency to focus fairly extensively on the evaluation of TMP strategies in the review, and would likely require some planning to collect the type of data that would make an assessment within the process review worthwhile.
Evaluation Methods for Assessing Transportation Management Plan Strategy Effectiveness
Common Transportation Management Plan Strategy Effectiveness Evaluation Challenges
Evaluation of TMP strategy effectiveness poses a number of significant challenges. One of the key confounding factors to measuring the effectiveness of a TMP strategy are project-specific characteristics such as the roadway network where the work zone is location, type of roadway, work zone design features and work tasks that must be accomplished. To better measure the effectiveness of a TMP strategy, the project characteristics and geographic conditions surrounding the evaluation should be included as part of a strategy effectiveness assessment.
The dynamic nature of work zones in general is another significant challenge to assessing TMP strategy effectiveness. The effects of a work zone on travelers typically vary as a project progresses through the various construction stages or phases. Unless data are obtained during times when the impacts of a project are most significant, an assessment of TMP strategies may not provide a realistic indication of the effect of those strategies.
Finally, the confounding effect of several TMP strategies implemented together on a particular project is still another significant challenge to assessing strategy effectiveness. While researchers have had some success isolating and assessing the effectiveness of single TMP strategies in a few special case studies, practitioners often do not have that possibility on ongoing projects. Additionally, even after the effectiveness of a single TMP strategy is quantified through a research case study, the question remains as to how that effectiveness might change when coupled with others in a real-world work zone. This is particularly relevant to evaluations involving the safety effects of certain strategies, and to evaluations of mobility effects of strategies that are commonly implemented as "packages" and which influence motorist travel choice decisions and behaviors (route, departure, and mode of travel).
Methods of Determining "Do We Think This Strategy (or These Strategies) Had an Effect?"
Evaluations that focus on answering this type of question typically rely on qualitative methods. From a mobility and safety standpoint, the approach could be to interview or survey the project engineer and/or inspectors during or after the project, assessing their opinions as to whether the impacts during the project were acceptable and whether the implemented TMP strategy or strategies were helpful in keeping the impacts acceptable. At the end of a construction season or as part of an agency's bi-annual process review, opinions gathered across multiple projects could be consolidated and critiqued to further gauge whether strategies used on multiple projects consistently yielded project outcomes that were positive.
Other ways in which qualitative methods can be used to assess TMP strategy effectiveness involve the traveling public and other stakeholders such as nearby residents and business owners. In-person or on-line surveys can be used to gauge overall level of satisfaction with the TMP strategies implemented, as can an absence of citizen complaints. These types of assessments usually fall under the full-scale evaluation of all strategies at a given location scope of assessment, although survey questions can sometimes be formulated to examine whether the respondent liked or approved of a particular strategy or subset of strategies that were implemented.
It is important to recognize that the degree to which such analysis results will transfer from one location to the next for these types of assessments is likely to be limited, as societal norms themselves can differ from location to location. For instance, customer opinions about how beneficial the TMP strategies were at a work zone that increases travel times by 5 minutes in a large metropolitan are likely to be different than those where the same strategies and 5-minute travel time increase occurs in a small town. It is also important to remember that although the outcomes obtained from these types of assessments can give an agency some confidence when considering whether to utilize the strategy again on a project in the future, it does not provide feedback useful in gauging whether the costs of implementing the strategy will be offset by the benefits gained in terms of reduce user costs or reduced project costs, for example.
Methods of Determining "How Much of an Effect Did This Strategy (or These Strategies) Have?"
Efforts to answer this type of question require a quantitative approach. A case study evaluation of a single strategy that is implemented at a single project is a common example of this method, as is the agency-wide evaluation of a strategy across multiple projects and the research evaluation of a strategy across multiple agencies. A without-with (equivalent to a before-after) type of study design is perhaps the easiest way to accomplish this type of assessment of a strategy. Data relative to the MOE of interest is obtained when the work zone is present but without the strategy implemented. The strategy is then installed, and data again collected. The two sets of data are then compared, and the difference between the two assumed to represent the effect of the strategy. Traffic control device strategies are commonly evaluated with this method. The statistical strength of this study design is sometimes enhanced through the use of control sites that are believed comparable to the sites of interest but where the TMP strategy is not implemented.
Although the without-with study design works well for several of the TMP strategies, it does not for others. In particular, strategies implemented to mitigate mobility-related and safety-related impacts often cannot be initially withheld and then introduced in a controlled fashion to accomplish a without-with type of analysis, due to liability or public backlash concerns. In these cases, more extensive, research-based analyses are usually required.
For assessments of strategy effects upon traffic safety, guidance is available for designing and conducting evaluations that would yield crash reduction metrics. (6) However, these methodologies require data from multiple projects and expertise in statistical analyses. Even then, the results of an analysis of a particular strategy or strategy package will likely include considerable variability in the estimate, due to the many other work zone site-specific factors that confound to also affect crashes at each location. One of the difficulties of applying these methods is in obtaining adequate sample sizes of projects that are similar enough in scope, roadway characteristics, and similar TMP strategies to be grouped together for analysis.
Assessing the effectiveness of TMP strategies intended to mitigate mobility impacts likewise creates significant challenges. For some of the strategies, the selection of an appropriate MOE would seem fairly obvious. For example, the effects of strategies that alter the traffic-carrying capacity of the work zone or alternative routes are fairly well understood, and a MOE related to the change in throughput would be an obvious choice. In fact, the expected capacity reductions and enhancements of many of those types of strategies can now be estimated by applying methodologies found in the Highway Capacity Manual (7) or other resources.
Beyond the use of throughput MOEs, the main challenge to the use of mobility measures is the determination of the proper baseline against which the effect of a strategy or set of strategies should be compared. A TMP may have any number of strategies included to ultimately improve mobility. For example, those related to traffic demand management strive to reduce or spread out that demand to different routes and times as a way to reduce the delays and queues experienced, whereas other strategies focus on improving work zone or alternative route capacity and speeds and thus reduce those impacts. However, experiences have shown that even in the absence of a coordinated implementation of mitigation strategies, motorists will adjust their travel patterns in response to change in operating conditions due to a work zone. (8) Consequently, the goal of an evaluation would be to measure the effect of a strategy or set of strategies against what would have occurred if the strategy had not been implemented, rather than simply how conditions changed from what they were prior to when the work zone was present.
Unfortunately, it is currently difficult to accurately predict how a work zone will alter travel patterns. Figure 1illustrates this concept. If introducing a work zone on a route adversely affects operating conditions on the route to a significant degree, a certain percentage of drivers will choose to alter their trip-making behavior by departing at a different time, changing their route, or perhaps even changing their choice of travel mode. This will occur even if no TMP strategies to encourage those changes are implemented. Conversely, the number of travelers modifying their trip-making behavior will affect how significantly the work zone itself affects operating conditions on that route. This same type of circular relationship between changes in trip-making decisions and the resulting operating conditions will also exist on alternative routes in the corridor. Therefore, strategies that attempt to also affect trip-making decisions and behaviors are not measured against what was happening before the work zone was introduced into the corridor, but rather measured relative to what would have occurred had the strategy not been implemented.
Figure 1. Diagram. Circular relationship between changes in
operating conditions and traveler trip-making decisions due to work zones.
A few simulation analysis tools have been developed to attempt and evaluate such behavioral changes in response to changing operational conditions, but their validation for work zone situations has been fairly limited. (9,10) Nevertheless, the flowchart provided in Figure 2 presents a logical process for using such analytical tools for estimating the incremental effectiveness of TMP strategies upon mobility. The methodology requires the use of a network-based model that performs route assignment and can also represent the effect of TMP strategies being considered for a project. The model is first calibrated and used to determine the TMP strategy or set of strategies that will be implemented for the project. After implementation, the model with the TMP strategies incorporated into it is recalibrated to conditions being observed at the project. The model is then used to systematically remove each strategy or strategy package and estimate what the mobility impacts would have been had the strategies not been implemented.
Figure 2. Diagram. Possible approach for estimating Transportation
Management Plan strategy effectiveness upon mobility impacts.
Methods of Determining "How Does the Effectiveness of a Transportation Management Plan strategy Vary as a Function of Differences in Roadway, Traffic, and Work Zone Characteristics?"
For the reasons previously mentioned, it is likely that the effectiveness of some TMP strategies in mitigating work zone impacts is not constant, but rather varies as a function of roadway, traffic, work zone, and other implemented TMP strategy variables. Ideally, assessments of TMP strategy effectiveness would be constructed as predictive models based on these input variables, so that the effect of implementing that strategy at a future work zone could be accurately predicted. A theoretical example of such a relationship might be the effectiveness of temporary concrete barrier implemented to separate the traffic space from the work space as a function of traffic volumes, type of work operations (which would correlate to the amount and size of equipment and materials in the work space), and lateral distance of the barrier from the traffic space. It is anticipated that methods such as these would need to be established through a significant research effort, possibly across multiple agencies, in order to obtain enough data with the range of input variables needed to establish the relationships.
Previous | Next
