4.0 Recommended Evaluation Framework

This chapter presents a recommended framework for before-after evaluations of the air quality and environmental justice impacts of congestion pricing projects. The recommendations presented here represent a general framework that provides the foundation for the development of project specific evaluation approaches. This framework is intended to stimulate, rather than replace, explicit project-specific evaluation methodology development processes that will include a wide range of stakeholders and which will carefully consider local conditions, project objectives and potential impacts, evaluation objectives, and evaluation resources.

The framework recommended here is consistent with the evaluation methodologies being utilized in the U.S. DOT evaluation of the Urban Partnership Agreement and Congestion Reduction Demonstration deployments. The general evaluation approaches are presented in the UPA/CRD National Evaluation Framework (http://ntl.bts.gov/lib/30000/30700/30764/14446.pdf). The detailed evaluation methods are described in a series of test plan documents that are being developed for each UPA/CRD. Test plans for the Minneapolis UPA site are currently available, and plans for the other sites will be made available as they are completed, on the U.S. DOT UPA/CRD “Publications, Legislation and Guidance” webpage: http://www.upa.dot.gov/pub.htm.

In keeping with this study’s interest in travel impacts in so much as they relate to environmental impact evaluation, recommendations for assessing travel impacts in and of themselves are not presented here. Rather, those travel impacts that are directly relevant given the recommended evaluation methodologies are discussed within the context of the environmental impact framework.

Congestion pricing projects rarely can be expected to produce significant noise impacts and therefore noise analysis is not recommended as part of a standard analysis framework. Consideration of other impacts which may or may not be universally categorized as “environmental impacts,” such as impacts on business and the economy may be appropriate for many congestion pricing projects but were not considered in this study.

Table 4-1 summarizes the overall recommended framework. Recommendations are elaborated in the text that follows (Section 4.1 and 4.2). The recommendations presented in Section 4.1 and 4.2, generally describe recommended best practice without regard to project-specific resource availability. Table 4.1 includes recommendations for cost-savings when resources are especially constrained.

 

Table 4‑1. Summary of Recommended Environmental Impact Evaluation Framework

Evaluation Consideration	Recommended Approach	Options for Limiting Costs
Air Quality
Ambient Monitoring versus Calculated Vehicle Emissions	Calculate project-related vehicle emissions rather than trying to discern them within monitored ambient pollutant levels	None (vehicle emission calculation is not resource intensive)
Dispersion Modeling	Only perform if significant increases in local traffic delay are expected, especially near sensitive land uses	Will not be needed for most evaluations
Vehicle Emission Calculation Procedures	Use a project-level analysis that sums emissions among individual study roadway links under before (without project) and after	None
Pollutants	Consult with state/local and Federal agencies within the analysis region Typically include criteria pollutants (CO, NOx, VOC and PM), greenhouse gas related pollutants (CO2, methane) and mobile source air toxics (benzene)	None (very little cost implication for calculating one versus several pollutants)
Geographic Area of Analysis	As many impacted roadway links (+ 5 percent project-attributable traffic volume change) as possible; at least all major roadways in the priced zone and all major, adjacent alternate routes	Focus on only the most significantly impacted roadway links and caveat conclusions appropriately
Traffic Data Collection Timeframe	More data is better—a minimum of several months with a full year being best, so as to control for seasonal and other cyclical changes	Limit to a few days or weeks of data collection, making sure to control for seasonal and other cyclical changes impacted before-after data.
Traffic Inputs and their Derivation	Use observed rather than modeled data At a minimum, include roadway link-specific VMT and average speeds; include driving cycle data when impacts are present and if resources permit collection of observed data Use any of various, proven speed data collection methods including probe vehicles or various roadway detectors (e.g., license plate readers, inductive loops, microwave, magnetic, Bluetooth, etc.) Calculate VMT based on actual link lengths and observed traffic volumes collected using any of various proven, specific data collection methods, e.g., inductive loops, microwave, Bluetooth, etc.	Eliminate driving cycle data and caveat findings appropriately Minimize or eliminate special data collection on less important links (rely mostly or entirely on existing detector data)
Project-Attributable Traffic Changes	Base the vehicle calculations on project-attributable rather than cumulative or total observed before-after changes Consider and apply a wide range of techniques to control for exogenous factors, including use of controls, statistical modeling, household travel diary data, etc.	Use cumulative, observed changes but qualitatively assess the potential influence of exogenous factors and caveat findings appropriately
Hourly Emission Estimates	Calculate total daily emissions as the sum of calculated hour-by-hour emissions	Assuming traffic data is available, there are limited cost implications of calculating by hour
Vehicle Mix	Break link VMT into VMT by major vehicle types and apply appropriate vehicle type-specific emission factors if project has impacted vehicle mix	Assume no project change in vehicle mix and use a single all-vehicle VMT figure
Emission Rates	Derive using an emission rate model selected based on local preference and familiarity If observed vehicle driving cycle data is available, utilize EPA MOVES model Carefully review and understand all model inputs, including default values	If MOVES not used, there are few cost implications because emission factors are often available from regional air quality agencies and/or can be fairly easily developed Collection of observed driving cycle data and use of MOVES should only be eliminated if no significant driving cycle changes are present and if resources preclude MOVES modeling
Noise
Inclusion in the Environmental Analysis	Not recommended as a “standard” practice because most congestion pricing projects will have little or no discernable impact Include only if it is a key local issue	If included at all, either monitor or model, but not both
Environmental Justice
Use of Prevailing, Fundamental Tools	Prevailing methods and tools provide a solid foundation and should be utilized in most analyses; tools/techniques include mapping of travel impacts in relation to minority and low income populations, attitudinal surveys and focus groups with various populations, and travel diaries completed by various populations	GIS mapping is more powerful, but simpler manual overlays may be substituted Panel surveys are best but cross-sectional surveys may be substitute Travel diaries are best but general stated behavior surveys may be substituted
Travel Impacts to Consider	Consider the potential environmental justice implications of any and all documented travel impacts (see text for recommended measures)	When household travel diaries/surveys are cost-prohibitive, may use individual travel diaries/surveys Eliminate travel diaries completely if resources preclude them and rely more on general stated behavior (from surveys and/or focus groups) Leverage existing system-based data collection and cut back on “special” data collection as resources dictate When resources dictate “picking and choosing” of impacts to consider, focus on those that are expected to be most significant and which can be most accurately measured, such as traffic volumes and speeds and transit ridership
Populations to Consider	Consider as many different populations as possible (see text), including those based on socio-economic, transportation mode, employment type, and trip purpose factors	If resources require “picking and choosing,” focus on socio-economic factors, especially income and race
Charging Revenues	Explicitly consider how charging revenues will be reinvested and relationship between who pays charges and who benefits from sharing revenue investments	In most cases, the scope and scale of the consideration of reinvestment will be much less extensive than the consideration of the direct travel impacts of the congestion pricing project itself and may be subjective
Sources of Demographic Data	Build in as much demographic data collection as possible into all manner of travel and other evaluation data collection; not just surveys	When resource constraints dictate, surveys or even zip code data can be used as a minimum

 

4.1 Air Quality Analysis Framework

This section presents a general recommended framework for evaluating the air quality impacts of deployed congestion pricing projects such as the U.S. DOT Urban Partnership Agreement deployments.

4.1.1 Ambient Monitoring versus Calculated Vehicle Emissions

Experience from other congestion pricing evaluations indicates that roadside monitoring of ambient air quality levels is not an effective approach to gauging project impacts; it is simply impossible to differentiate project-related impacts from other, exogenous factors. As such, the first and most fundamental recommendation is that the air quality impacts of congestion pricing projects be evaluated using the one method that allows direct estimation of project-attributable air quality impacts: by comparing before (without pricing) and after (with pricing) calculated vehicle emissions.

4.1.2 Dispersion Modeling

Dispersion models, or hot spot models, estimate the localized ambient concentrations of vehicle emissions, as compared to the “calculated vehicle emissions” discussed above which calculate the volume of pollutants being emitted directly from vehicles. Dispersion modeling takes into account atmospheric and site topography considerations to estimate roadside concentrations of pollutants emitted from all traffic. No examples of dispersion modeling for before-after congestion pricing projects were found among the eight study projects or in the general literature.

It is recommended that dispersion modeling only be performed if the traffic analysis indicates that the congestion pricing project has created significant increases in localized traffic delay, especially near sensitive land uses such as nursing homes, parks, or schools. Such impacts may be possible with some congestion pricing projects, such as those that may divert significant traffic away from a priced facility onto one or two already congested parallel routes. If dispersion modeling is appropriate selection of a model and specific methodologies should be based on local considerations and should reflect an interagency consultation including state and local air quality and traffic agencies and regional U.S. DOT representatives. Examples of dispersion models include the EPA CAL3QHCR or CARB CALINE4 carbon monoxide dispersion models.

4.1.3 Calculation of Vehicle Emissions

This section presents the recommended framework for what will, in most cases, constitute the air quality evaluation: calculation of before and after vehicle emissions. The recommended framework is as follows:

• Project-level Analysis – vehicle emission calculations for deployed congestion pricing projects should, logically, use a project-level analysis. A project-level approach calculates project-related changes in traffic volumes and speeds on specific affected roadway links and calculates total vehicle emissions as the sum of calculated emissions for each link.
• Pollutants – the selection of pollutants for consideration in the air quality analysis will be driven by local air quality issues. The selection of pollutants should be made via an interagency consultation including state and local air quality and traffic agencies and regional U.S. DOT representatives. Many analyses will likely focus on all or some of the following pollutants associated with vehicle activity:
  • Some criteria pollutants (pollutants associated with National Ambient Air Quality Standards): carbon monoxide, ozone precursors nitrogen oxides, volatile organic compounds, and particulate matter
  • Greenhouse gas related pollutants: carbon dioxide and methane
  • Mobile source air toxics (MSATS): benzene.
• Geographic Area of Analysis – The geographic area of analysis should include as many of the roadway links as possible that are expected to be affected by the project, a determination that will be made in the traffic analysis that precedes the air quality analysis. Generally, at a minimum, the air quality analysis should include the major roadways (highway and major arterials) within the priced zone as well as the major likely alternate routes to those roadways. It has been suggested that “affected” roadways could be defined as those links where the average annual daily traffic is expected to change by more than ±5 percent as a result of the project.⁵⁸ Experience has shown that traffic diversion can be significant: in pre-deployment surveys for the Seattle region UPA congestion pricing deployment, 40 percent of respondents indicated that they will take an alternative, non-priced route;⁵⁹ in London, it was estimated that 20-30 percent of car trips no longer made into the priced zone are now made on non-priced roads.⁶⁰
• Data Collection Timeframe – A minimum of several months of before and after data is recommended and up to a full year of data pre- and post-pricing project implementation is better. Collection of a full year of before and after data allows for seasonal variation to be controlled for or explicitly evaluated, allows random variations to be averaged out, and allows examination of both the immediate (first few weeks/months) as well as somewhat more mature (one year) impacts of the pricing project be investigated. If using only a few months of data, it is important to compare before and after data that are from the same season or month to control for seasonal variation.
• Traffic Impacts and their Derivation – At a minimum, observed rather than modeled roadway link-specific VMT and average speeds should be utilized. Link speeds can be collected using any of various, proven speed data collection methods including probe vehicles (e.g., floating cars operated by the evaluators, GPS-equipped vehicles operated by public volunteers, or toll-tag equipped vehicles operated by the general public) or various roadway detector types such as license plate readers, inductive loops, Bluetooth, microwave, infrared, acoustic, etc. Link VMT should be calculated based on actual link lengths and observed traffic volumes. Traffic volumes should be collected using any of various proven, specific data collection methods, e.g., inductive loops, microwave, etc. If the project has been shown by the traffic analysis to have significantly impacted traffic flow (i.e., driving cycle), observed driving cycle data should be collected and used in the calculation of vehicle emissions (see discussion of Emission Rates, below).
• Project-Attributable Traffic Changes – Emission calculations should utilize project-attributable changes in link speeds, VMT (and, if applicable, driving cycle) rather than total observed changes in these metrics. That is, ideally, the influence of exogenous factors should be controlled in such a way that observed traffic data can be quantitatively adjusted to eliminate the non-project related portion of total before-after variability. If exogenous factors such as fuel price changes, employment changes, and other transportation projects and programs are believed to have exerted little or no influence on travel in the study area, then observed post-deployment VMT and speeds can be used directly. However, if—based on thorough tracking from the pre-deployment through the post-deployment data collection period—there is reason to believe that exogenous factors have significantly impacted observed post-deployment traffic data, efforts should be made to adjust the post-deployment traffic data to eliminate the influence of exogenous factors. If that is not possible, the results of the emissions calculations should caveat the results appropriately. The influence of exogenous factors on driving cycle is especially problematic. If it is determined that exogenous factors have significantly impacted traffic conditions, in many cases consideration of driving cycle impacts may need to be eliminated. This is because it is much harder to quantitatively adjust driving cycle data than volume or average speed data—if possible at all it would require resource-intensive traffic simulation. Further, even if adjustments can be made, the resulting estimates may be so uncertain as to eliminate the fundamental value in considering driving cycle data at all. Recommended methods for determining whether and how much exogenous factors have influenced traffic volume and speed changes include:
  • Comparisons to control roadways/corridors/areas
  • Statistical modeling that can remove or control for the effect of exogenous factors by including such variables in multivariate equations
  • Utilization of household survey data (travel diary data being ideal) to understand the causes behind reported changes in travel behavior, including the following considerations:
    • Using adequate sample sizes given intra and inter-household changes over the analysis period.
    • Collection of detailed information from participants to better understand the reasons behind the reported travel behavior.
    • Consideration of case study approaches in which the impact of changes in household demographics are thoroughly explored at the individual household level, potentially including home interviews and focus groups.
  • Tracking fuel prices and employment levels
  • Elimination of traffic data from times and locations within the study area characterized by severe weather, significant traffic incidents, and significant roadway construction
  • Collecting before and after tracking data for the same month(s) of the year
  • Examination of historic traffic trends.
• Hourly Emission Estimates – It is recommended that total daily vehicle emissions for both the before and after periods be developed by summing hourly emission estimates. Those hourly estimates should use hourly VMT and speed data.
• Vehicle Mix – Traffic analyses of congestion pricing projects should test for before-after changes in vehicle mix and any changes should be reflected in the air quality analysis by breaking VMT down by vehicle type.
• Emission Rates (Factors) – Emission factors should be derived from an emission model such as the EPA MOBILE6, MOVES or California Air Resources Board EMFAC models. Selection of the model will be driven by local preference and familiarity (that is, using a model that is accepted and well adapted to the analysis region) as well as the approach taken to driving cycle.
  • Resources permitting and assuming that a traffic analysis has shown that driving cycle impacts (changes in flow—the pattern of acceleration, deceleration, idling and cruising on study roadway links) are likely, these impacts should be explicitly considered in the air quality analysis. That will require collection of before and after driving cycle data using instrumented test vehicles performing floating car runs on all of the study roadways and during all of the time periods of interest. Multiple runs will be required on each link at different times of the day, made by at least a couple of different types of vehicles. When observed driving cycle data are available, it will be important to utilize the EPA MOVES model to derive emissions factors because the MOVES model provides by far the most effective means for taking driving cycle changes into account.
  • Collection of observed driving cycle data can be expensive. For example, as part of the preliminary planning for the U.S. DOT evaluation of the Seattle-Lake Washington Corridor Urban Partnership Agreement, it was estimated that collection of driving cycle data for that analysis would cost between $50,000 and $100,000.⁶¹ In those cases where resources do not permit the collection of driving cycle data, the selection of the emission factor model should be based on local considerations.
  • Regardless of which emission factor model is selected for a specific analysis, evaluators should carefully review and understand the model inputs and default and user-defined inputs and options. Local inputs such as fleet registration and fuel-type distributions, vehicle emission inspection and maintenance (IM) programs, and vehicle fuel programs should be accurately reflected in the model.

4.2 Environmental Justice Analysis Framework

This section presents recommendations for evaluating the environmental justice impacts of congestion pricing projects. Perhaps more so than any other areas of evaluation, the specific issues to be considered in the environmental justice evaluation should be driven by local, site-specific issues and objectives. However, there are several principles that will be widely applicable.

4.2.1 The Prevailing, Fundamental Tools and Techniques Provide a Solid Foundation

In addition to the recommended enhancements noted below, the fundamental tools and techniques that currently constitute state of the practice for investigating environmental justice are useful and should continue to be utilized. These methods include:

• Using regional geographic information systems to map the locations of low-income and minority populations within the likely impact area.
• Using attitudinal surveys, interviews and focus groups of the general public, corridor travelers and specific types of residents and travelers to gather attitude and perception as well as general travel behavior data.
• Using travel diary surveys to gather detailed, specific travel behavior data of various groups of interest.

4.2.2 Travel Impacts

The specification of travel impacts relevant to an environmental justice analysis is more complex than the specification of traffic impacts relevant to an air quality analysis. In the case of air quality, only a few “bottom line” traffic impacts (VMT, speeds and driving cycle) manifest directly in terms of air quality. Consideration of other travel impacts (e.g., mode choice) is irrelevant because such changes either ultimately translate to changes in VMT, speed and/or driving cycle or they are totally unrelated to air quality. In the case of environmental justice, it is not the case that only certain travel impacts are relevant, although certainly some, such as charges paid by socio-economic category, are more relevant. Rather, an environmental justice analysis seeks to understand a particular dimension (differential impacts among populations) of essentially any and all significant travel impacts.

As such, the first and most fundamental consideration in regard to assessing environmental justice impacts is to start with a comprehensive travel impacts evaluation that assesses as many potentially significant project-related travel changes as possible. These impacts will vary according to the type of project and its regional setting, but generally, a comprehensive travel impacts analysis (and one which will support a robust environmental justice evaluation) will consider the following impacts:

• Traffic volumes
• Vehicle miles traveled
• Average speeds
• Person and vehicle throughput
• Travel times, including some “indexed” measure such as the travel time index utilized by the Texas Transportation Institute in their urban traffic monitoring program for U.S. DOT
• Travel time reliability, such as represented by a “buffer index” or “planning index”, both of which capture the extra increment of time travelers need to plan for given observed variability in travel conditions
• Geographic and temporal extent of congestion, e.g., hours of congestion and miles of congested roadway
• Vehicle classification/vehicle mix
• Average vehicle occupancy
• Mode choice/mode split
• Accident rates and contributing factors
• System operator and/or traveler (all modes) perceptions of safety and congestion
• Transit ridership
• Transit travel time
• Transit schedule adherence/on-time performance
• Household traveler behavior (travel diaries completed by each member of the household documenting routes, modes, foregone trips, times of travel, etc. with accompanying attitudinal surveys)
• Congestion pricing charges paid

An environmental justice analysis should consider how each of these travel impacts associated with the project in question impacts different populations (see Section 4.2.3). For example, what types of people use the roads where traffic volumes decreased and increased? What types of people made what sorts of mode choice changes and how did those changes impact the quality of their travel experience? Which types of people paid various amounts of congestion charging fees?

Note that depending on the project, it may be useful to collect additional travel impacts—what could be considered first order impacts—that improve the understanding of the project impact (and exogenous factor impact) on the “bottom line,” or second order, impacts. Consider the example of a congestion pricing project that is accompanied by supporting transit improvements including park-and-ride lot enhancements. In that case, collecting data on park-and-ride lot utilization in addition to transit ridership will help explain whether it was the project (in this case the park-and-ride lot enhancements) or other factors (e.g., fuel prices) that drove any ridership changes.

4.2.3 Consider Broadly How Impacts Differ Among a Wide Range of Users

One of the primary recommendations is that evaluations of environmental justice should focus more broadly on understanding how impacts will vary for a wide range of users rather than only on how impacts may vary based on income and minority status. Income and minority status are certainly very important categories to consider. However, the investigation of differential impacts in those areas should be part of a broader area of enquiry that permeates many different individual evaluation analyses (e.g., traffic, travel behavior) and which seeks to understand all of the ways in which some people may be impacted differently than other people by the project. Depending on the project, the types of differential impacts which should be considered for investigation include:

• People of varying income and education levels
• People of various racial groups
• People with various employment status, including full-time, part-time and unemployed
• Users of different transportation modes, including drive alone, rideshare, telecommuters, bicyclists, users of various transit services, and pedestrians
• People with varying travel origins and destinations (namely residential and work locations)
• Users with varying degrees of flexibility in changing their time, route or mode of travel
• Different trip purposes
• Travel during different days of the week and time of day
• Frequent travelers versus occasional travelers
• People with disabilities
• People with differential access to various travel modes, including transit and private auto
• New residents versus long-term residents
• Visitors versus permanent residents.

Understanding which of these sorts of distinctions are going to be important will have significant implications on the development of the evaluation plan, as the data needs associated with these distinctions will impact various data collection areas across the evaluation. Surveys will of course be significantly impacted, where it will be important to categorize the respondent according to all of the demographic and other characteristics of interest, but so will the collection of objective data such as traffic data. For example, understanding how impacts differ between carpoolers and single occupant vehicles could mean that collecting average vehicle occupancy data is important.

4.2.4 Explicitly Consider the Uses of Charging Revenues

Evaluations should endeavor to take into consideration how any public revenues raised by the congestion pricing scheme will be reinvested in any transportation programs or projects and the potential implications of those investments on the net environmental justice impacts of the pricing project. Reinvestment can significantly impact net equity effects.

4.2.5 Collect Demographic and other User Data Wherever Possible

Key to understanding environmental justice impacts of congestion pricing projects is the ability to associate specific impacts, e.g., changes in traffic volumes and speeds, with various user groups (demographics), including those that vary by income, minority status, access to transit, access to private auto, residential location, work location, etc. Therefore, it is important build in as much demographic data collection as possible into all manner of travel and environmental data collection throughout the evaluation. This includes the obvious example of including demographic questions in all surveys. But other opportunities should also be investigated, for example, objective traffic data collected using license plate readers may—if supportable under applicable state and local privacy laws—also provide some understanding, through vehicle registration data, of the residential locations associated with the vehicles. That information in turn could provide some understanding of origin-destination.

 

---

 

⁵⁸ Claggett, Michael and Miller, Terry, “A Methodology for Evaluating the Mobile Source Air Toxic Emissions Among Project Alternatives,” May 2006.

⁵⁹ Washington State Department of Transportation, “SR 520 Bridge Tolling Good to Go! Baseline Survey – Telephone Survey Report, Draft,” prepared by Pacific Rim Resources, Inc. May 2010.

⁶⁰ Transport for London “Central London Congestion Charging Impacts Monitoring Second Annual Report,” April 2004. http://www.tfl.gov.uk/assets/downloads/Impacts-monitoring-report-2.pdf

⁶¹ Unpublished technical memorandum, “Decision Support Information for UPA/CRD Environmental Analysis Approach,” prepared by Battelle for the United States Department of Transportation, November 2009.

To view PDF files, you need the Adobe Acrobat Reader.