Scoping and Conducting Data-Driven 21^st Century Transportation System Analyses

Module 2. Data-Driven Transportation Analysis Project Scoping

Following the characterization of system dynamics and the development of an analytic project statement (presented in Module 1), Module 2 provides guidance on how to develop a transportation analysis scoping plan. Transportation agencies can use this scoping guidance and accompanying scoping tool to develop preliminary scopes and budgets for their transportation analysis projects. Information about the scoping tool is provided later in this module. Figure 20 illustrates the orientation of this module with respect to the diagnostic efforts that precede it and the data preparation efforts that are initiated after the completion of the project scoping summary. Actions in this module include project definition, identification of performance measures to be used in the analysis, data needs for the analysis, and cost and schedule estimation.

Figure 20. Diagram. Project scoping within the 21^st Century analytic project scoping process.

(Source: Federal Highway Administration.)

Need for and Characteristics of an Analytic Project Scoping Plan

A scoping plan provides a valuable tool for communicating the project scope, a critical element—indeed, the foundation—of success in an analytic project. The plan includes initial planning and scoping, as well as iterative updates to assumptions, scope, and agreements as the project moves forward. Scoping plan development is the primary mechanism for securing a clear and mutual understanding of expectations and assumptions among stakeholders. It helps identify flaws or technical issues in the project concept that may have been otherwise overlooked.

The scoping plan confirms both the stakeholder agreements regarding the scope of the analysis and the most appropriate approach for the analysis based on an enhanced understanding of project objectives, the study area conditions, the strategies being implemented, and the available tools and data. The benefits of developing a scoping plan include a better allocation of resources appropriate to the study objectives; a clear and shared understanding of roles, responsibilities, and expectations among project participants; and the ability of project participants to effectively communicate the project vision to the broader stakeholders. It also helps maintain agreement and project continuity as stakeholders leave positions and new staff comes in mid-stream.

The scoping plan documents the common understanding among all stakeholders regarding the purpose, nature, and extent of the proposed analytical project. It compiles project information and understanding developed to date and provides a single-source document for the proposed analysis approach. While the scoping plan must be sufficiently detailed to provide practical guidance on the actual conduct of analysis, it should retain some flexibility to adapt to project contingencies as they are encountered. The following are signs that a transportation analysis project is on track and likely to be successful:

• A detailed scoping plan is prepared that identifies appropriate analysis tools, data required, performance measures, and all parties responsible for various parts of the analysis.
• The purpose of the analysis is clear.
• Key stakeholders are engaged from the start of the analysis to its completion.
• Tools used in the analysis can convincingly demonstrate their ability to replicate observed traffic conditions using quality-checked, internally consistent, observed data.
• Interim and final results can be independently reproduced.
• Analytical results can be clearly communicated relative to analytical objectives.

A scoping plan should include the following interrelated components (the order may vary depending on project characteristics):

• Project definition.
• Geographic and temporal scope.
• Selection of the appropriate analysis tool type.
• Performance measures to be used in the analysis.
• Analysis data requirements.
• Preliminary list of alternatives to be studied, including analysis scenarios and transportation mitigation strategies.
• Expected cost, schedule, and responsibilities for the analysis.

Defining and Scoping the Project

The Analytic Project Summary prepared in Module 1 does not contain sufficient detail to perform project scoping. Additional details are needed to focus and clarify the nature and objectives of the proposed analytic effort. Project definition includes the problem statement and project goals and objectives, existing operational conditions, problem diagnosis, affected stakeholders, analysis time horizon, analysis periods, study area and geographic scope for the analysis, and modes and facilities affected.

Problem Statement and Project Goals and Objectives

This section of the scoping plan should describe the purpose of the project, provide the project background, and present the problems and issues that the analysis is intended to address. The project understanding should include clear descriptions of the transportation project purpose and need, elements that relate to the transportation problem to be analyzed, the project study area, affected communities and stakeholders, and traffic analysis objectives and hypotheses.

The overall goals and objectives for the mitigation strategies being considered should be assessed and used to shape the goals and objectives of the analysis effort. A clear definition of the "what" and "why" for conducting the analysis provides a foundation for the analysis plan. The objectives should be "SMART" (specific, measurable, actionable, realistic, and time-bound).

Document Existing Operational Conditions

Existing traffic conditions should also be documented, including average daily and peak traffic levels; directionality of traffic flow; variability of traffic flow; status of construction activities; known bottlenecks; queuing conditions; free flow and average peak speeds; and summary incident and accident statistics for the study area.

Available documentation should be compiled and reviewed by the analysis team to familiarize themselves with the operating characteristics of the study area and identify substantial issues. Individual interviews with project partners can help analysts understand study area conditions and the strategies being considered. Previous studies, archived data systems, and accident/incident data reports can all provide valuable insight into the current operational characteristics of the study area. The analysis team should seek out information on the mitigation strategies being considered and the impact these strategies have had in other regions.

The scoping team should visit the study area to gain a better understanding of traffic conditions and characteristics. The site visit(s) should include a comprehensive review of the different facilities, modes of transportation, and major mode transfer locations throughout the study area. The site visit may include visits to the regional traffic management center or toll authority, as appropriate. Depending on the characteristics of the mitigation strategies being considered, the scoping team may want to plan to visit the site on multiple occasions (e.g., peak period versus off-peak, or good weather versus inclement weather) in order to gain further insight into how traffic characteristics vary.

In documenting existing conditions, it can be useful to analyze and document the factors that influence congestion in the study area (e.g., frequency of special events). This analysis activity eventually feeds into the identification of analysis scenarios completed in a subsequent step. The analysis of influencing factors can include demand variations, incidents, or weather. This analysis helps project analysts and stakeholders to better understand the causes of congestion in the study area and the frequency with which these causal events occur. Data related to these factors can be compiled and analyzed to illustrate the effects of the factors on existing traffic conditions in the study area.

In the scoping activity, the description of operational conditions can be more qualitative than the systematic methods of characterizing operational conditions discussed in Module 3. However, if past projects have conducted more detailed and quantitative characterizations, these can serve as references to assist in the high-level description.

Identification of Underlying Causes

Key outcomes of the assessment of the existing operating conditions can be thought of as a "problem definition" and a "problem diagnosis." The scoping team should carefully evaluate any needs assessments and problem definitions included in other documents to see that they are consistent with the existing conditions data and material compiled. If a modified or more discrete problem definition is required, the scoping team should work with the stakeholders to firmly define the problem being addressed. The problem diagnosis should include a more thorough assessment of the study area conditions to ensure that the needs are properly defined, underlying causes are documented, and a small number of relevant hypotheses are created about what will help resolve the issue or mitigate the problem. Reviewing results from previously conducted assessments and comparing these with high-level assessments of existing conditions data helps identify the likely cause(s) and extent of the identified problem. Additionally, the scoping team should carefully assess any project goals and objectives identified to date and map these to the problem diagnosis to evaluate their applicability. Beginning to assemble and evaluate influencing factors in this way affords an opportunity to identify the best combinations of multiple scenarios that are most representative of actual conditions.

Affected Stakeholders

It is essential to identify a complete set of stakeholders and partners who fully represent the agencies and organizations affected by the project. Stakeholders and partners include representatives of agencies from different jurisdictions managing parts of the study area, and components and modes impacted by the strategies (e.g., highway or roadway agencies, transit agencies, program managers and stakeholders, freight industry groups, bike/pedestrian groups).

Developing a stakeholder and partner database from the beginning is important because it can provide a mechanism for tracking contact information, special concerns, and stakeholder engagement. At a minimum, it can be helpful to track stakeholders and partners by name, organization, segment (state/local/private sector), title, role on the project, mailing address (with state, city/county noted in manner that can be sorted), and the individual's contact information.

Project-specific stakeholders and partners may include more technical stakeholders and could include nontraditional members, such as emergency responders, toll authorities, and media representatives, depending on the priorities and objectives of the project (and proposed alternatives to be assessed). Guiding principles help stakeholders and outside interests better understand the focus and boundaries of the analysis effort and help ensure that any key stakeholder concerns are honored as part of the process. A key principle that should be applied in all analyses is "The overall analysis effort must take place within the budget and timeframe specified in the Scoping Plan." To minimize the risk of having to redo parts of the analysis late in the process, agencies with reviewing and/or approving authority over the analysis should be at the table from the start of the project. At the very least, project partners should have an opportunity to review and comment on the project statement as early as possible.

Analysis Study Area, Time Periods, Time Horizon, and Modes and Facilities Affected

Once an initial understanding of the study area and its operating characteristics has been defined, stakeholders refine their understanding in light of the area's major transportation issues. These issues form the foundation for analysis scenarios and mitigation strategies likely to be considered in the analysis. Analysis scenarios are developed based on the study area's geographic scope; infrastructure and facilities; and causes and patterns associated with recurrent congestion (capacity, weave zones, etc.) and non-recurrent congestion. The scenarios also consider specific causes of non-recurrent congestion, such as traffic incidents (number of incidents per day in study area, number of lanes blocked, response time, high-frequency crash locations, root cause where known [e.g., merge or weave zones, lane drops, physical characteristic such as a blind curve], incident response protocols for various incident types, etc.). The problem definition and diagnosis documented in the assembly and analysis of the existing conditions completed in the previous steps form the foundation for the identification of suitable alternatives.

Although the alternatives identification discussed in this section is presented as a linear process, in reality it is typically an iterative process. The initial alternatives identification takes place in close concert with the design phase—formulating likely strategies and combinations of strategies, then mapping these to the existing conditions outputs generated in the previous stage. As the analysis continues and the initial results are reviewed and shared with the design team, modifications or new alternatives may be proposed as existing alternatives are found to be impractical or result in unforeseen negative impacts. For example, an analysis of a strategy may reveal that the strategy is creating a bottleneck at a downstream location that was not foreseen prior to analysis. This unexpected result may promote a change in the strategy that causes the analysis to be re-run for alternatives containing the strategy. In reality, the alternatives definition and design process continue in an iterative manner throughout the process. Please refer to Module 1 for more information on system performance.

Scenarios should be developed for the range of operational conditions of greatest interest to the site in light of its analysis objectives. While traffic incidents are often the single largest cause of non-recurrent congestion, stakeholders are encouraged to investigate and understand other influences, including special events, weather, fluctuations in demand, and work zones. This initial analysis includes developing a preliminary understanding of both supply side (infrastructure/capacity) and demand-side influences on the study area across all modes (including underlying causes of demand, such as directionality or day of week). The goal is to identify potential issues (where demand exceeds supply to an extent believed to interfere with study area performance) and opportunities (underused capacity/supply that could potentially help absorb demand).

Practitioners should compile data on the frequency and severity of conditions linked with elevated congestion levels. This data creates the foundation for identifying "operational clusters" that characterize the operational conditions found in the study area and can later be used to organize the analysis (described in Modules 3 and 4). Comparisons or distributions of various sources of delay should be assembled and evaluated to identify the relative frequency of events/conditions related to congestion. From this assessment, practitioners should critically assess the potential impact of various scenarios. Scenarios identified as having a low frequency of occurrence and limited impact should be considered with lower relative priority for the analysis effort, since in the final analysis these scenarios has little overall impact on system performance. This problem diagnosis based on severity and frequency of scenarios may reveal needs previously unknown to the practitioners, help to weed out inconsequential scenarios, and greatly assist in targeting resources to provide the greatest expected value from the analysis.

The data analysis required in this step includes identifying the frequency and likely impact of the scenario. Scenarios recording the greatest frequency and the greatest impacts should be given the primary priority. Scenarios with a low likelihood but major impact (e.g., major snowstorms) and scenarios with a frequent occurrence but limited impact (e.g., minor incidents occurring on otherwise normal days) should be assigned secondary priority. Scenarios with low frequency and low impacts should be assigned tertiary priority. Often impacts for tertiary scenarios can be inferred or estimated from other results, so direct modeling of these scenarios may be eliminated from the project scoping plan. There are no set thresholds for the inclusion or dismissal of scenarios. The analysis team needs to apply engineering judgment and common sense at this point in the process. This section of the scoping plan should include the following components that help define the analysis as clearly as possible:

• Study Area, Facility Types, and Affected Modes. The spatial extent of the study area includes intersections, highways, and other facilities to be analyzed. The study area must cover beyond the end of the full spatial extent of queues and congestion in the baseline and future years of analysis. All transportation modes in the study area (heavy or light rail, bus transit, High-Occupancy Vehicle [HOV] or High-Occupancy Toll [HOT] lanes, bicycle, etc.) should be included in the analysis as deemed necessary; the consideration of mode shift may be important in the analysis of near- and longer-term mitigation strategies in the study area.
• Analysis Time Period. Define the project analysis time period, (AM/PM/Midday peak hour and/or peak period, off-peak period, etc.). This time period must cover the beginning and end of full temporal extent of queues and congestion in the baseline and future years of analysis.
• Alternatives Definition. Scenarios should include geometric and operational alternatives to be analyzed and compared to the baselines. Generally, it is a good idea to define a finite set of alternatives to be analyzed but also allow for a small number of presently unknown alternatives to be identified in the future. Module 4 is devoted to Alternatives Analysis.
• Analysis Time Horizon. A future baseline model (or future no-build alternative) is an essential part in the analysis process. It is the basis for comparison between alternatives but in a future time horizon as required by analysis needs. For the purposes of transportation analysis, a common methodology for developing future demand forecasts is a regional Travel Demand Model. This model takes into account regional growth due to land use, demographics, and socioeconomic activity. In cases where a demand model does not exist, it is acceptable to use a trend projection of travel demand.

Selecting the Appropriate Analysis Tool Type

Once the analysis scenarios and strategies are identified, the analysis team begin to explore and select the appropriate analysis approach to be applied. This is a critical step in the analysis methodology; selecting the appropriate approach and tool type (and the selection of a specific tool) ensures that the analysis meets the needs of the study and streamlines the analysis process. This step follows the preliminary scoping steps because before they can select a tool, project partners must first understand the mitigation strategies, scenarios, and operational conditions of interest for the analysis and have a general grasp of the available data. One critical assessment is whether the questions posed in the project are inherently retrospective or if a predictive or prospective analysis is required. Useful retrospective analyses may be conducted through the analysis of archived performance data to differentiate the effectiveness of distinct, current operational practices. However, for new operational practices, a prospective analysis using a predictive tool is likely required. If partners want to model the potential impact of some proposed (but not yet implemented) road pricing strategies, it is important to select a tool that can accommodate this. Modeling of traveler diversion requires a combination of tools to produce results that can be assimilated to build the desired understanding. This is often a multi-iteration process where the analysis team initially focuses on identifying a high-level category of tool to use, then focuses on identifying key capabilities provided by different tool types, and ultimately selects the specific tool to apply in the analysis. Since transportation analysis science and applications are continuously evolving, as are the performance measures used in transportation analysis, the contents of this scoping guide will evolve in light of new developments in the transportation analysis field. Key steps in the evaluation and selection of the analysis tools are identified below:

• Research and identify available analysis tool type(s) for the study area. In this step, the analysis team researches and compiles information on models and analysis tool type(s) currently in use in the region. This may include models used on a continual basis in the region and individual models used for specific, one-time analysis in or near the study area. For each model or tool, the analysis team should identify the following:
  • The analysis package or tool (name and version of the software).
  • The year of the analysis.
  • The time periods available for analysis.
  • Facilities represented in the mode.
  • Modes represented in the tool.
  • Any special scenarios available in the model (e.g., incidents, special events, weather).
  • High-level assessment of capabilities and limitations.
  The availability of a model or tool in a region is a first step; it should not be used as the sole determinant for selecting the models and tools used in the analysis. In subsequent steps, the analysis team assesses their individual needs and maps these to an appropriate tool or combination of integrated tools. This scan of available tools helps educate project partners about the range of tools available and their relative advantages and disadvantages. It is useful for selecting the appropriate set of tools and for identifying potential data sources for the model development. The Federal Highway Administration (FHWA) Traffic Analysis Toolbox is a useful resource for this step.
• Identify factors for selecting tool type(s). The analysis team identifies and performs a critical analysis of the key factors that determine the required robustness of the analysis toolset selected. The resulting analysis is used in a subsequent step to map project needs to an appropriate tool or combination of tools.
• Select the appropriate tool type(s). Once the scan of available tools is complete and the analysis team has had the chance to identify and critically analyze the selection criteria based on the project needs, the team is ready to select the tool type(s). The FHWA Traffic Analysis Toolbox initiative includes a spreadsheet-based decision support tool that the team can use to weigh various factors to determine the appropriate tool(s). The decision support tool does not recommend a specific software vendor's tool; it identifies an appropriate category of tool based on the input factors. The team must evaluate specific vendor products within these categories. Figure 21 provides an overview of the basic factors the FHWA Traffic Analysis Toolbox method considers.

The tasks completed to develop the scoping plan to this point provide most of the input necessary to complete Categories 1 through 6 in the FHWA Traffic Analysis Toolbox Decision Support Methodology, shown in Figure 21. Particular care should go into assessing the seventh category on Tool/Cost Effectiveness. Assessing this final factor involves evaluating the cost-performance tradeoffs associated with qualifying tool options that satisfy the first six criteria. Once a qualifying set of candidate tools has been identified, the analysis team determines which tools can deliver the greatest value for the estimated cost in software and configuration/calibration load. This determination is influenced, in part, by the general understanding of available data (and workload required to render that data useful for the modeling effort, which may vary by tool in light of their various capabilities), staff skills, previous modeling efforts that can potentially be leveraged for this effort, and so forth.

In assessing the needs of project analysis, multiple tools may be required. A single tool may not be sufficiently robust to handle the analysis needs, and the analysis team may need to consider integrating the analysis capabilities from multiple tools to achieve the necessary abilities. In this step, the analysis team also specifies requirements for interfacing between different tools.

Once appropriate tool categories have been selected, the analysis team uses the documentation provided with the FHWA Traffic Analysis Toolbox Volume III to research the range and capabilities of individual software packages and tools within the selected category.

Figure 21. Flowchart. Federal Highway Administration Traffic Analysis Toolbox: Overview of analysis factors to be considered in selecting appropriate analysis tools.

(Source: U.S. Department of Transportation.)

The U.S. Department of Transportation (USDOT) documentation includes web links to individual research organizations and vendors supporting the various packages for more information; peer research can be valuable as well. From this research, the analysis team can make a high-confidence decision regarding the specific tool—or combination of tools—that best meet the requirements of the analysis project's needs.

Accuracy and Precision Requirements for the Analysis

Generally, there are three sources of uncertainty related to the conduct of transportation analysis:

• Uncertainty associated with forecasting future travel demand, future economic factors, and/or other traveler-related characteristics, such as a traveler's willingness to pay for use of various components of the transportation system. The analyst has little control over "unknowables" in the future, but techniques can be used to minimize or bound future uncertainty; for example, the future location of jobs will determine commute patterns and the congestion along certain corridors. These techniques typically involve sensitivity or risk analysis implemented by varying analysis parameters and assumptions using probabilistic distributions of these parameters based on "back-casting" methods. These methods result in a probabilistic representation of future impacts that represent a statistical likelihood that a project will have a certain impact. As long as analytical assumptions and parameters do not vary across different alternatives, the analysis can produce valid conclusions on the relative effectiveness of different alternatives to, say, improve mobility.
• A second type of uncertainty in transportation analysis relates to traveler responses to different operational conditions and deployed mitigation strategies. Traveler choices are being affected by different operational conditions (e.g., rain versus no rain) and by different mitigation strategies deployed to address a transportation problem; depending on the prevailing operational conditions, travelers may leave their origin earlier or later or even choose a new destination. There are very limited data on this type of traveler responses and the motivation behind each response. Until such data become available, transportation analysis must rely on a robust sensitivity analysis that would test the analysis assumptions by varying the relevant analysis parameters until a satisfactory solution has been reached.
• A more manageable type of uncertainty in transportation analysis relates to the ability of analysis tools to replicate observed conditions and traveler behaviors; this practice is usually referred to as "model calibration." The remainder of this section focuses on how this type of uncertainty can be mitigated so that more trusted models can be produced to analyze the effects of different mitigation strategies.

Model calibration is the process of systematically adjusting model parameters so that the model can reproduce the observed traffic conditions. The process continues until the error between the performance measures taken from the field data and the performance measures calculated in the modeling is less than a predetermined margin of error. Once the model reproduces observed conditions, model calibration can focus on specific performance measures, such as volumes, speeds, travel times, and bottlenecks. In the model calibration process, there needs to be a tradeoff between required precision and available resources to collect data and conduct modeling.

The goal of calibration is to make the model represent local, observed traffic conditions as closely as possible. However, since traffic may vary greatly from day to day, it is not possible for one model to accurately represent all possible traffic conditions. Most modeling software are developed using a limited amount of data. The model parameter values are estimated using the limited data.

Driver behavior differs by region, and it may differ significantly from normal to non-typical days. For example, poor visibility, severe weather, incidents, presence of trucks, and pavement conditions all have an impact on driver behavior. We do not recommend using a model developed using data from one region to represent future traffic conditions in another region. Investment decisions made using a model that has not been calibrated to local field conditions will be flawed.

Guidance on the overall model calibration process is presented in Volume III of the Traffic Analysis Tools suite. The overall model calibration process can be divided into four main steps, as described below:

• Identify the performance measures related to the analysis objectives and critical locations for the models to be calibrated against.
• Determine the strategy for calibration, consistent with Volume III of the Traffic Analysis Tools suite.
• Determine the statistical methodology to be used to compare modeled results to the field data. Identify Tolerance E (measure of precision) and Confidence Level (measure of accuracy or variability among samples).
• Conduct model calibration runs following the strategy and conduct statistical checks. When statistical analysis falls within acceptable ranges, then the model is calibrated.

The updated Volume III of the Traffic Analysis Tools suite update allows for the analysis of projects under different operational conditions with a better-expected statistical fit than a "typical day" trying to represent a reliability space including operational conditions such as different incidents, work zones, weather events, special events, etc.

During the model scoping process and the development of the data collection plan the types of required field measurements are determined, including speeds, volumes, queuing, and other congestion observations at different locations in the network. Since traffic conditions fluctuate daily, it is important to obtain field data from multiple days. The multiple days of data serve as a database in which field variations are used to determine the tolerance of error in the simulated results.

During model development, it is recommended that the spatial and temporal model limits extend beyond where and when the congestion in the field occurs. The statistical calibration should not necessarily include every link in the model, but it should focus on the critical design elements within the primary study area in the model.

The number of locations selected for comparing the performance measures in field data against model outputs needs to be balanced against the quality and location of the available data, the desired level of statistical confidence, and the availability of resources. The model outputs and reporting for these statistical tests should be similar to the performance measures that will be used later on in the analysis of the study area.

The selection of the number of data days to be used should be based on an analysis of available data in terms of what data is available and cost-effective to collect. In an urban area where freeway sensor data are archived and readily available, more days of data can be used. In areas where there is no surveillance and where manual or temporary data collection devices are used, the amount of data to be collected is more resource-intensive.

Cost Implications

Generally, building and calibrating analysis models for large or more complex study areas requires a disproportionately greater level of effort compared to smaller projects. This is attributed to the increased complexity of modeling travel demand and traffic operations in a larger geographic area and to more extensive project reporting, alternatives analysis, and presentation requirements. Seemingly similar projects may require different levels of effort for a number of reasons:

• Experience of project manager, analysis team, and reviewers.
• The project purpose, objectives, and scope.
• The availability of good data for model calibration.
• Temporal and spatial resolution requirements for the analysis.
• The number and complexity of the alternatives being analyzed.
• Performance measures used.
• Software used.
• The amount of documentation, meetings, and presentations required.
• Number and effectiveness of project reviews conducted.
• The extent of stakeholder involvement.

Selecting Performance Measures

Early in the project, the analysis team defines performance measures in line with the project objectives, mitigation strategies under consideration, analysis scenarios, and operational conditions (shaped by the understanding of available data) identified for the project. This provides the "home" where these measures are documented in the scoping plan, and helps the analysis team crystallize its vision and scope for the analysis effort.

In order to begin to understand how the proposed transportation improvements perform and whether they meet stakeholder expectations, and even to help stakeholders develop realistic expectations for the improvements, project managers must first be willing to articulate them. The analysis effort helps illuminate which expectations are realistic, which may be unrealistic, and why. It identifies opportunities that optimize the study area's transportation network by allowing analysts to experiment with adjusting mitigation strategies for the price of a model run rather than myopically making such adjustments to the actual deployed system, where the cascading second and third-order effects are more difficult to perceive in real-time.

Performance measures should be closely tied to the identified overall project goals and objectives and the expected traveler responses. For many improvement strategies, it is important to consider a set of performance measures that are sensitive to recurring and nonrecurring congestion. The scoping plan should identify the selected performance measures and the approach for calculating the performance measures based on the expected model capabilities and available data.

An effective way to identify appropriate performance measures is to test one or more specific hypotheses for each objective. These hypotheses can indicate a change in travel conditions (such as: The strategies will reduce travel times during an incident by 5 percent.) or can be neutral in the prediction of an impact (such as: The strategies will not result in a change in emissions rates.). Performance measures that support the testing of the formulated hypothesis should be identified. Using this method ensures that the performance measures are appropriately mapped to the project goals and objectives.

To compare different investments within a study area, it is important to define and apply a consistent set of performance measures. The performance measures should:

• Provide an understanding of travel conditions in the study area, including localized and system-wide metrics representing impacts in the immediate vicinity of the proposed improvement and in the larger study area.
• Be consistent with lead agency overall performance measures used to evaluate all sorts of transportation improvements.
• Demonstrate the ability of improvement strategies to improve mobility, throughput, and travel reliability based on current and future conditions.
• Help prioritize individual investments or investment packages within the study area.

To the extent possible, the measures selected should be reported for the overall system and by:

• Mode—Single Occupancy Vehicles (SOV), High Occupancy Vehicles (HOV), transit, freight, etc.
• Facility Type—Freeways, expressways, arterials, local streets, etc.
• Jurisdiction—Region, county, city, neighborhood, and study area-wide.

Transportation analysis performance measures typically focus on the key areas described below. However, customized measures may be selected based on unique impacts of individual mitigation strategies. There are five key performance areas:

• Mobility. Mobility describes how well people and freight move in the study area. Mobility performance measures are readily forecast. Three primary types of measures are used to quantify mobility: travel time, delay, and throughput. Travel time and delay are fairly straightforward to calculate using model outputs. Throughput is calculated by comparing travel times under the incident scenarios to those under no incident. By comparing the percentage of trips under the same threshold travel time in the pre- and post-mitigation scenarios, the relative influence of the strategies on reducing extreme travel times can be estimated.
• Reliability and Variability of Travel Time. Reliability and variability capture the relative predictability of the public's travel time. Unlike mobility, which measures how many people are moving at what rate, reliability/variability measures focus on how mobility varies from day to day. Travel-time reliability/variability is reported in terms of changes in the Planning Index and changes in the standard deviation of travel time.
• Transportation Safety. This is another performance area that may be of interest to transportation analysis. Safety is typically measured in terms of accidents or crashes in the study area, including fatalities, injuries, and property-damage-only accidents. Currently available safety analysis and prediction methodologies are not sensitive to transportation improvement strategies. At best, available safety analysis methods rely on crude measures such as Volume-to-Capacity ratio (V/C) or empirical comparison methods, such as identifying safety benefits that result from implementing a certain type of mitigation strategy and applying the same expected improvement rate to a future implementation of the same or similar strategy. Clearly, this is an area deserving new research.
• Emissions and Fuel Consumption. Emissions and fuel consumption rates are used to produce estimates based on variables such as facility type, vehicle mix, and travel speed.
• Cost Estimation. Planning-level cost estimates are based on life-cycle costs, including capital, operating, and maintenance costs. Costs are typically expressed in terms of the net present value of various components. Annualized costs represent the average annual expenditure that is expected in order to deploy, operate, and maintain the transportation improvement and replace equipment as they reach the end of their useful life.

In attempting to optimize a system's performance across all these measures, the analyst may be faced with conflicting objectives and constraints as they relate to different performance measures. For example, minimizing travel time or maximizing travel speed are often in conflict with minimizing emissions of fuel consumption. In order to overcome this conflict, the stakeholders and analysts must make tradeoffs that typically result in less-than optimal system elements for a specific measure but also result in a system that strikes a balance between multiple performance measures, hopefully optimized in different ways for different operational conditions.

Analyzing Data Requirements

The precise data requirements for developing and calibrating a transportation analysis model vary depending on the software tool selected; however, they all require five basic types of input:

• Roadway geometry.
• Traffic control data.
• Travel demand, traffic volumes, and intersection turning movements.
• Performance data, such as queue locations, queue lengths, travel times, and speeds.
• Data on vehicle characteristics, such as vehicle classifications or vehicle mix.

As models are calibrated to the data collected for the study, the true validity of any model is dependent on the quality of the data that goes into it. To get an early estimate of effort required, data requirements should be evaluated during the initial stages of a project. Quality/variability of existing data affect sample sizes, so early statistical evaluations of the data (Margin of Error) can prove highly valuable for subsequent development of effort required. Key model calibration performance measures should be identified early on to help determine the data needed to estimate these performance measures.

Since data collection can be one of the most important components of an analytical study, it is critical to identify the key data needed for the study early in the process. Often, it is possible to use existing data, but the data may be outdated or from different timeframes for different parts of the network. In that case, resources need to be allocated for new data collection. If there is limited funding, resources need to be spent judiciously to collect sufficient quality data to conduct a study that helps inform decision-makers of the potential implications of their proposed transportation investments.

Given the importance and costs of data for the modeling process, it is important to document the information. Documentation methods should include a clear file-naming structure, an explanation of the sources of data, and a database structure that can be readily incorporated into model inputs and reports. Efficiency of managing the data can be further enhanced through the development of automated procedures. Module 3 in this guide provides more detailed guidance on data storage, quality assurance processes, and metadata requirements.

Types of Data

Types of data can be categorized in three main areas: travel demand, traffic control, and physical geometry. Travel demand data include traffic counts, vehicle classification counts, speeds, travel times, congestion, and queuing observations; these data require the majority of the data collection effort. Traffic control data include signs, signal control, and timing plans. Physical geometry can be obtained from rectified aerial photography and base mapping files that may be prepared as part of the design effort for projects. The tenets of data collection and data management that are important for conducting an effective analytical study include the following:

• The type of analysis being conducted should dictate the quantity of data collected.
• The required accuracy should drive the quantity of data collected.
• Since the quality of the analysis and the resulting decisions depend on the data used, it is important to use data that is recent, internally consistent, and relevant.
• Collection of traffic data should capture the temporal variations in demand and performance and should be finite enough to allow for an appropriate analysis to be conducted.

Helpful tips on data collection include:

• Do not simply rely on the veracity of historical data; allocate resources to verify the data through current field inspection.
• Measure flows and estimate or forecast demand—procedures for estimating demand are given in Volume III of the FHWA Traffic Analysis Toolbox.
• Check flow conservation—entry flows should match the exit flows.
• Always check for data quality—use the most recent Geographical Information System (GIS) files, maps, and drawings.
• Use counts from before the onset of congestion to after congestion dissipates.
• Broaden the geographic scope to cover bottlenecks and spatial and temporal extent of queues.

Data Sources

• Travel Demand. The basic demand data needed by most transportation analysis software are the entry volumes—the travel demand entering the study area—at different points of the network. At intersections, the turning volumes or percentages should be specified.
• Origin-Destination (O-D). O-D trips can be estimated from a combination of travel demand model trip tables and traffic counts. While O-D data can be obtained from the local Metropolitan Planning Organization's (MPO) regional travel demand model, these are generally 24-hour estimates, so they must be adjusted and refined to produce hourly or peak period estimates for use in simulation or other transportation analysis models; typically, these estimates are further disaggregated to represent 5- or 15-minute estimates required for more detailed analysis. License plate matching surveys can be used to estimate hourly trips, but this is resource-intensive. Depending on project complexity, a cost-effective way is to adjust the travel demand model O-D trip estimates based on field counts. If the study area has transit, HOV, and trucks in the vehicle mix or if there is significant interaction with bicycles and pedestrians, corresponding demand data would be needed. In addition, vehicle dimensions and vehicle performance characteristics (e.g., maximum acceleration and deceleration) are required. Even if only the peak periods are being examined, demand data should be collected before the onset of congestion and should continue until after the congestion is dissipated. Also, to capture the temporal variations in demand, it is best not to aggregate demand data to longer than 15-minute intervals.
• Vehicle Characteristics. This data can be obtained from the state DOT or air quality management agency. National data can be obtained from car manufacturers, the Environmental Protection Agency (EPA), and the FHWA.
• Traffic Control Data. Data from traffic control devices at intersections or junctions are required. Control data refers to the type of control device (traffic signal, stop sign, ramp meter, etc.), the locations of these control devices, and the signal-timing plans. Traffic control data can be obtained from the agencies that operate the traffic control devices in the given study area. Traffic operations and management data on links are also needed. These include location and type of warning signs—for lane drops, exits, guide signing; type and location of regulatory signs. If there are HOV lanes, information on the HOV lane requirement (HOV-2 versus HOV-3), their hours of operation, and the location of signs are needed; if there are HOT lanes, information on the pricing strategy is required.
• Operational Conditions. If there are Variable Message Signs (VMS) in the study area, the type of information displayed, the location, and—where possible—the actual message displayed are needed. Most of this information can be obtained from the public agencies responsible for operating the VMS. The type of signs and locations can be obtained from GIS files, aerial photographs, and construction drawings. Event data can be received from public agencies, such as the Traffic Management Center (TMC) logs. Crash databases should be verified since data may not always be recent and may not be for the specific study area. The team can usually obtain workzone data from state DOTs and weather data from the National Oceanic and Atmospheric Administration (NOAA). Overall, the data should be from concurrent timeframes so that it is easier for the analysis to establish cause and effect.
• Transit Data. This data can be obtained from the local and regional transit operators. It can include schedules and stop locations. Calibration data could include Transit Automatic Vehicle Location (AVL) data, boarding and alighting data, and dwell time at stops.
• Mobile Source Data. Mobile source data include data derived from mobile phones, blue tooth devices, and other mobile sources can be used to augment other data collected. The primary type of data obtained from mobile sources are speeds and travel times. In most cases, the mobile source techniques use samples of vehicles in the traffic stream, but they may not be reliable sources for traffic counts and vehicle composition. Mobile source data are also being used to develop Origin-Destination (O-D) matrices or trip tables. The mobile source data are typically obtained, stored, and sold by private vendors. Before purchasing these types of data, it is a good idea to have a demonstration of the data and a means to compare the data supplied by the vendor with a real observation of what is happening in the field. Consider how the data are provided (format, structure, and software) and take into account how the data will be used for traffic modeling and other purposes.

Challenges with Data

Going through a systematic process of collecting the critical data, verifying data quality, and documenting any assumptions are key to justifying the results of a study to decision-makers and the public. A statistical analysis of collected and previously available data can be helpful in determining the statistical data variability and the margin of error contained in the data.

Data Comprehensiveness. Comprehensive data cover different performance measures (volumes, speeds, bottlenecks, queuing, and congestion data) across freeways and arterial streets, as well as transit data and incident data. Traffic counts should be taken at key locations in the study area; key locations include major facilities (freeway segments, major intersections, and interchanges) and major on- and off-ramps. If possible, data collection should be done simultaneously at all key locations. Otherwise, the counts should be taken at least during similar timeframes with similar demand patterns and weather conditions.

Data Reliability. Automated data sources are often best for collecting the long-term data needed to develop and calibrate transportation analysis models. However, many existing automated data collection systems lack the robustness or reliability to effectively compile relevant data sets. A thorough assessment of the data quality from all sources is recommended to identify any potential problems early in the process and establish methods to address any deficiencies.

Accuracy is the measure of the degree of agreement between data values and a source assumed to be correct. It also is defined as a qualitative assessment of error. It is important to have accurate, internally consistent, and recent data. If information on future traffic conditions is not available, it could be helpful to study data from other regions that may have the bottlenecks and traffic patterns that are envisioned for the study area. This is critical since the goal of model calibration is to not only see if the model can represent observed conditions, but also to examine if the model can handle the future congestion. With respect to data filtering and fusion, it is crucial to adopt standard ways to accept or reject field data and to address data gaps and missing data. Overall, data accuracy is paramount for an analysis project since it percolates throughout the analysis process.

A small margin of error in the collected data is required to increase the validity of analysis results. It is necessary to collect enough data points for each performance measure (volumes, speeds, etc.) so that the sample is an accurate representation of the mean and standard deviation of this performance measure. Depending on the mean to standard deviation ratio and the desired margin of error, the required sample size may vary greatly. Module 3 in this guide presents a discussion on cluster analysis and how it can be used to improve the analytical accuracy.

Documentation of Data Gaps

At the end of the data assessment task, the analysis team should develop a summary of what data needs are covered by in-house or available data and what primary data collection may be needed. This is a critical item because data collection is going to be costed out since estimates for data collection, quality control, and analysis will need to be developed separately from tool acquisition, from base model development and calibration, and from alternatives analysis.

Refining Alternatives and Mitigation Strategies

Interviews with project partners can be useful in understanding study area conditions and the improvement strategies being considered. Additional documents can provide valuable insight into the current operational characteristics of the study area: previous studies; transit data and studies on topics such as ridership and parking occupancy (supply and demand); archived data systems; and crash/incident data reports. Regional and long-term transportation plans provide insight into congestion hotspots, but these must be supplemented by more detailed studies.

The analysis team should also seek out information on the improvement strategies being considered and the impact these strategies have had in other regions. Likewise, developing an understanding of previous capacity and operational strategy projects in the study area—including an understanding of the expected and unexpected results of those projects—can be beneficial. Peer-to-peer contact with agencies that have undergone similar planning and deployment is a valuable way to gain this insight.

Once the analysis scenarios have been identified, the next step is to identify the improvement strategies and define under which analysis scenarios the strategies will be activated. It is also critical to understand precisely when (under what conditions) the strategies will be applied and how their application may vary under different conditions. To better understand what factors influence congestion and the frequency in which these factors occur, the analysis team works with the stakeholders to identify the combinations of travel demand, incidents, special events, and weather events that affect study area operations. A transportation improvement project is typically concerned with nonrecurring congestion on a level equal to or greater than typical recurring congestion levels. Therefore, it is critical that the analysis team recognize the non-typical factors that affect nonrecurring congestion.

The analysis team begins by exploring preliminary scoping hypotheses and assumptions and by identifying possible opportunities and constraints associated with the application of improvement strategies identified under specific operational conditions. For example, freeway managers may be interested in opportunities to divert drivers from the freeway to arterials as an incident management strategy. They talk with local arterial managers to understand whether the local jurisdiction can accommodate this diversion and activate these strategies to accommodate the desired performance in a sufficiently timely manner. This step also includes discussion of the potential to avoid problems—such as changing signal timing or VMS to avoid queues that may lead to collisions at critical locations. If there are contributing circumstances that can be avoided that lead to greater non-recurrent congestion, the team must devise strategies that may avoid these contributing circumstances. They also seek to understand and address any concerns or constraints the local jurisdiction may have. Is the technology in place to accommodate the needed signal timing? Can the timing strategies be activated in a sufficiently timely fashion to make the strategy feasible?

Likewise, if local agencies are interested in exploring opportunities to divert freeway or arterial traffic to transit, they will engage in collaborative dialogue with the regional transit managers to understand possibilities for creating available transit capacity, including parking facilities, to accommodate the possible influx of demand under certain scenarios.

These discussions begin to yield initial insights into the eventual performance measures for the analysis effort. Because the models have not yet been run, these are preliminary scoping discussions at this phase, the purpose of which is to begin to shape possibilities and identify limiting factors that will form the foundation of the more detailed planning documented in the analysis plan. As discussions progress and more quantitative data becomes known, stakeholders update and refine assumptions and hypotheses, bringing further clarity and detail to the hypotheses and assumptions associated with operational conditions and strategies to be analyzed. The analysis team identifies which parts of the envisioned strategies they may want to make dynamic (strategies that could be manipulated in response to changing operational conditions, such as ramp metering, HOV, or pricing strategies, or traveler behavior such as mode choice).

Cost, Schedule and Responsibility for the Analysis

Analysis roles must be defined and clarified among the various project stakeholders. Table 3 provides a high-level example of how key project roles can be displayed. The project team also includes a summary of the estimated budget and timeframe in the analysis plan. These will be updated regularly as the analysis effort moves forward.

Although the scoping plan is designated as a single deliverable, there may be multiple sub-deliverables (e.g., technical memos and presentations) that are generated as the plan is developed, depending on workflow and individuals responsible for compiling the necessary information. The scoping and analysis plans undergo some major revisions prior to finalization to allow for additional levels of detail as the process moves forward and to allow for stakeholder input and comment. Key output of the scoping and analysis plans includes Memoranda of Understandings/Agreements (MOUs/MOAs) among initiative stakeholder organizations, documenting project scope and anticipated roles and levels of effort.

Table 3. Example high-level allocation of analysis responsibilities.

Work Step	Analysis Project Manager	Operations Manager	Planning Manager	Modelers	Systems Manager	Stakeholders
Develop Analysis Plan	Primary Responsibility	Secondary Responsibility	Secondary Responsibility	Secondary Responsibility	Secondary Responsibility	Secondary Responsibility
Develop Data Collection Plan and Collect Data	Primary Responsibility	Secondary Responsibility	Secondary Responsibility	Secondary Responsibility	Secondary Responsibility	Secondary Responsibility
Model Setup and Calibration	Primary Responsibility	Secondary Responsibility	-	Primary Responsibility	-	-
Alternatives Analysis and Documentation	Primary Responsibility	Secondary Responsibility	Secondary Responsibility	Primary Responsibility	Secondary Responsibility	-

There are many variables that affect the estimated level of effort for the analysis, including the existence of—and level of precision in—available project documentation; quantity, quality and availability of needed data; cohesion in stakeholder vision for the analysis effort; and experience of the project staff with modeling tools in previous efforts. Although each analysis varies due to these factors, a rough order of magnitude estimate of the proportion of analysis resources that may be required of the different analysis steps includes:

• Develop Analysis Plan—20 percent.
• Develop Data Collection Plan and Collect Data—20 percent.
• Set up and Calibrate Model—30 percent.
• Analyze and Document Alternatives—30 percent.

While the analysis steps are presented sequentially, in actuality the analysis process is iterative and requires flexibility in its application. The implementation of these steps must be carefully configured to the individual needs of each analysis effort and appropriately readjusted throughout the process as conditions and needs change.

The FHWA publication FHWA-HRT-13-026 (March 2014) "Guidance on the Level of Effort Required to Conduct Traffic Analysis Using Microsimulation" provides analysts and modeling managers with a frame of reference on the level of effort required to complete a transportation analysis using microsimulation. (Guidance on the Level of Effort Required to Conduct Traffic Analysis Using Microsimulation.)

The reader of this guide should use the labor-hour estimates as a point of reference and not as an absolute number to apply to projects. Seemingly similar projects can require different levels of effort for a number of reasons:

• The project purpose, objectives, and scope.
• The availability of data (how much data had to be collected).
• The number and complexity of the alternatives being analyzed.
• Performance measures used.
• Software used.
• The amount of documentation required.
• Experience of project manager, analyst, and reviewers.
• Number and effectiveness of project reviews conducted.
• The amount of stakeholder involvement.

A comparison of the total level of effort and the amount of effort expended specifically for building a base model and the calibration process reveals a trend that is a consistent order of magnitude regarding the model building and calibration effort with the model size. Larger models require a disproportionately greater level of effort compared to smaller projects. This is attributed to (1) the increased complexity of a larger analysis model as it relates to modeling travel demand and traffic operations in a larger geographic area, and (2) extensive project reporting, alternatives analysis, and presentation time expended.

The Scoping Tool

As part of work on this guide, a software tool was developed to produce ballpark estimates of staff-hours to complete the tasks needed to support a transportation analysis consistent with discussions developed within the Scoping Guide. The objectives of the scoping tool are (1) to inform a procurement decision made by a transportation agency before procuring transportation analysis services; (2) to provide a rough order labor hour estimate for conducting the transportation analysis; and (3) to incorporate an estimate of relative risk associated with various analysis tasks.

The focus of the scoping tool is on traditional, off-line project analysis—not on analysis conducted in real-time to actively managing aspects of the transportation system. The focus of the tool is also on time-dynamic analytics, such as all types of simulation and Highway Capacity Manual analysis, but not including travel demand modeling.

The scoping tool is a simple spreadsheet tool that produces ballpark estimates of labor hours required for analysis based on a small number of user inputs. The tool first asks users to choose among small number of options related to the expected analytical complexity and risk. Based on user inputs, the tool uses factors to assess different complexity and risk levels and to produce outputs, including ranges of labor hours by task for three labor categories plus total labor hours by task. The Scoping Guide team used records from 40+ analysis projects to inform the calculation engine inside the scoping tool. The scoping tool methodology was validated through iterative stress tests. Assumptions and methodology were vetted in several review cycles, including both internal and external teams of analysis experts. User inputs and options provided by the scoping tool are identified below:

• Number of intersections in the study area.
• Number of on- and off- freeway ramps.
• Baseline model availability (No, Partial, Full).
• Baseline model calibrated? (No, Yes).
• Data collection requirements (Low, Medium or High Effort).
• Number of time periods to be analyzed (e.g., AM, mid-day and PM peak periods).
• Number of alternatives to be included in the analysis.
• Number of operational conditions to be analyzed.
• Number of analysis horizons (Baseline year plus X number of future years).
• Complexity of analysis scenarios (multiple modes, pricing, etc.—Simple or Complex).
• Complexity of analysis methodology (Deterministic or Dynamic/Stochastic).
• Complexity of output performance measures (Simple or Comprehensive).
• Analyst Experience (Some or Considerable).

Complexity and risk factors taken into account in the scoping tool include:

• A large number of intersections and/or on- and off-freeway ramps in the study area.
• No baseline model available.
• Baseline model not calibrated.
• High data collection requirements.
• Complex analysis scenarios (multiple modes, pricing, etc.).
• Complex analysis methodology (Dynamic).
• Reliability analysis (multiple operational conditions).
• Complex output performance measures (such as reliability of travel time, or emissions).
• No significant analyst experience.

At this time, there is no documentation provided with the scoping tool. Figure 22 and Figure 23 provide example screenshots from the Scoping Tool.

Figure 22. Screenshot. Analysis scoping tool—summary of example user inputs.

(Source: Federal Highway Administration.)

Figure 23. Screenshot. Analysis scoping tool—example output.

(Source: Federal Highway Administration.)

Moving to Module 3: Summarizing the Project Scoping Plan

A scoping plan should, at a minimum, include the elements listed in Table 4. These components of the scoping plan can guide the data preparation, gap analysis, and data collection planning and execution described in Module 3, and the analysis activities described in Module 4 of this Guide.

Table 4. Project scoping summary elements.

Project Definition	A concise statement of the overall system problem includes cross-validation and other insights from stakeholders on the nature of the issue and potential solutions.
Geographic Scope	The geographic area to be covered by the analytical project includes a statement of the required detail of representation within this geographical area.
Temporal Scope	The times of day, days of week, seasonality, and years of operation are assessed in the analytical effort. This includes an assessment of the simulation horizon.
Candidate Hypothesis	The candidate hypothesis represents the leading underlying cause of the system performance issue.
Analytical Approach	This element describes of the proposed method for evaluating the effectiveness of the mitigating strategies in resolving the system performance issue.
Selected Tool Type(s)	The one or more tool types will be used in the analytical approach. This section should identify if existing models are to be employed, or if new models must be developed.
Data Requirements	A summary of data will be used to characterize operational conditions, represent alternatives, and model the geographic and temporal aspects of the system.
Preliminary List of Alternatives	High-level description of the alternative solutions and/or operational practices will be assessed within the analytical project.
Key Operational Conditions	The set of travel demand, incident, and weather conditions under which a meaningful examination of alternative impacts must be conducted.
Selected Performance Measures	The measures of system performance selected for the effort. These measures should be most suited to differentiate alternatives, be meaningful to stakeholders, and can be well-represented/estimated within the proposed analytical approach.
Expected Costs	The projected cost of the analytical project, including data collection.
Expected Schedule	The projected time to conduct the analysis, including data collection.
Expected Assignment of Responsibilities	An assessment of responsibilities related to the project and how those responsibilities are allocated among departments, contractors, and other organizations engaged in the effort.
Risks	A summary of risks comprising risks in data collection, technical risks, and non-technical risks.

You may need the Adobe® Reader® to view the PDFs on this page.