Chapter 4 – Performance
Monitoring and Evaluation
Page 2 of 3
4.2.5 Information Gathering
Obviously, a direct relationship exists between the performance measures
selected and the data needed in the performance measurement process. The
data and information used in decision-making must be of high quality.
They must originate from reliable, consistent sources and meet the needs
of the decision makers. Moreover, the decision makers must have confidence
in the information, or it will not be used.
The most common data problems are acquiring the required information
and in ascertaining the quality of the data. The "garbage in, garbage
out" concept applies to the data used in a performance measurement
system. If the data gathered are highly uncertain, then the conclusions
drawn by converting those data into performance measures also will be
highly uncertain and will have reduced value in managing the agency. For
this reason, great care needs to be taken in data collection. Investments
in accurate, high-quality data collection systems are essential to successful
performance measurement and, by extension, to achieving the overall strategic
goals of the agency. In reality, however, some things either cannot be
measured accurately or cannot be measured accurately at an acceptable
cost. Transportation agencies need to consider the uncertainty introduced
by inaccurate data when taking action based on their system of performance
measures (7).
References 3 and 8 discuss the concept of a "Performance Monitoring
Plan" as a mechanism for collecting the data needed to quantify
performance measures. Such a plan is essential for coordinating and allocating
resources and for controlling the quality of the information that is used
for evaluations. The monitoring plan specifies such things as:
- The data to be collected
- Frequency of data collection / schedule
- Data collection locations
- Data collection responsibilities
- Data analysis techniques and responsibilities
- Database management requirements
- Performance analysis reporting
Once the desired data are in hand, the focus shifts to the analysis and
reporting of results. In this stage, the most challenging problem is often
separating the impact of the activities of the transportation agency from
the impacts generated from beyond those activities. For example, highway
crashes are influenced by many factors besides highway design. If an agency
uses the total number of highway crashes as a performance measure, does
an increase in crashes indicate that the agency's safety programs
are ineffective? Before that conclusion is drawn, the impact of changes
in other causal factors (e.g., weather) clearly needs to be understood.
The necessity of separating the impacts of external factors has direct
implications for data collection. Even though statistical techniques might
be available to allow the impacts of several factors to be isolated, the
techniques require large numbers of observations to be used reliably.
Thus, it is necessary to have a data collection system that increases
the number of observations by maintaining data with some degree of disaggregation
in both time and space (7).
As noted in the overview section at the beginning of this chapter, the
detection and surveillance subsystem of a Freeway Management System represents
a potentially valuable data source for performance monitoring. Typically,
the FMS generates massive amounts of data about the state of travel that
are used by transportation authorities to effectively operate and manage
their transportation systems, including traveler information. As a general
rule, this information is collected and used in real time at a TMC to
continually improve the operational performance of the system. The increasing
deployment of FMS and the amount and variety of FMS-generated data throughout
the nation offer great potential for longer-term transportation planning
and performance monitoring. The same information collected at the TMC
may also be used – but no longer in the context of real time applications
– at the ITMS and agency tiers to identify deficiencies, and then
to design and establish short term operational improvements such as incident
response plans. These same data may also be applied at the state / regional
tier, being incorporated into the transportation planning process for
analyzing and evaluating alternative transportation improvements.
In order to monitor the long-term performance of the transportation
network, the real time operations data collected by the FMS and /or ITMS
must be systematically retained and reused – a process known as
"data archiving" or data warehousing.
4.2.5.1 Data Archiving
The primary reasons for archiving FMS-generated data are:
- Provide more and better information for managing and operating the
system — The first step in proactive management is knowing where
problems are likely to occur before they actually do, then preventing
or mitigating the impacts of those problems. Archived operations data
can be used to predict when and where problems may occur again, as well
as helping to evaluate alternative strategies for preventing or mitigating
the problem.
- Maximize cost-effectiveness of data collection infrastructure —
Data archiving permits transportation agencies to maximize their investments
in data collection infrastructure by re-using the same data for numerous
transportation planning, design, operations and research needs.
- Much less expensive than manual data collection — Data archiving
is significantly less expensive than having a planning or design workgroup
re-collect even a small percentage of the data using manual methods
or special studies.
- Established business practice in other industries — The retention
and analysis of operational data is an established practice in most
competitive industries that use data to manage their business activities.
(12).
Given that archived FMS-generated data can provide a valuable longer-term
resource for a variety of stakeholders, the Archived Data User Service
(ADUS) was incorporated into the National ITS Architecture in September
1999 to help realize the potential usefulness of ITS data. A U.S. Department
of Transportation multi-agency, 5-year ITS Data Archiving Program Plan
was developed based upon the vision of "improving transportation decisions
through the archiving and sharing of ITS generated data."
Attempting to use data to meet information needs for which the data were
not originally intended can be a challenging endeavor. In the context
of ADUS, data issues are multi-faceted and complex, including data quality,
format, integrity, compatibility, and consistency. Moreover, with ITS-generated
data being so temporally extensive (e.g., collected every 30 seconds)
but spatially limited (e.g., covering 30 miles of roads), ADUS data sometime
need to be integrated with data from traditional sources in order to be
useful.
The "Guidelines for Developing ITS Data Archiving Systems"
(Reference 13) provides a number of
basic principles that can be applied regardless of archive size or design,
including:
- Determine the workgroup(s) or agency(ies) that should have primary
responsibility for operating and maintaining the data archive. This
may seem like a simple matter; in many cases, though, data archiving
systems have not been further developed because no one has taken responsibility
for their operation and maintenance.
- Discussion and dialogue in early stages among all stakeholders should
assess the demand for archived data as well as the strengths and weaknesses
of which agency or workgroup in a region maintains data archives. In
some cases, there may be several agencies that each operate their own
data archive, but which are connected and integrated through a "virtual
data warehouse". In other cases, it may be logical for a regional
planning agency with strong information management capabilities to warehouse
data that can be shared among other agencies in the region. In any case,
sharing data between agencies will be necessary, and will require some
level of agreement on data definition and geographic units. (Refer to
Chapter 16 on Regional Integration).
- Start small but think long-term, and begin with modest prototypes
focused on a single source of data (e.g., freeway detector data).
- Develop the data archiving system in a way that permits ordinary users
with typical desktop computers to access and analyze the data. Effective
data archiving systems make large operations data archives available
to ordinary computer users without requiring them to have specialized
database or programming skills. These systems use a "point-and-click"
interface, either through a Windows-based application or a web browser,
to provide access to the data archives.
- Provide access to and distribution of archived data through the Internet
or portable storage devices such as CDs or DVDs. Internet-based access
and distribution of data are some of the most common and effective means
to share archived data. CDs or DVDs are used as an alternative to Internet-based
data archives, permitting the data archiving agency to maintain greater
control and security over the data.
- Save original data as collected from the field for some specified
period of time, but make summaries of this data available for most users.
Many data archiving systems aggregate data to a consistent time interval
(5 minutes is most common) for loading into a data archive. Because
there will always be some users interested in the original data, a mechanism
should be developed to store this for a short period of time or to store
it permanently off-line.
- Use quality control methods to flag or remove suspect or erroneous
data from the data archive. The rigor of the quality control ultimately
depends upon how and for what purpose the data will be used. Two different
philosophies exist for what to do with data that has failed quality
control:
- Simply identify or flag the data records that have failed quality
control; or
- Remove the data records that have failed quality control and replace
with better estimates.
These business rules (for how to deal with data failing quality control)
will depend upon who will be using the data and for what purpose. There
is no single correct answer for quality control.
- Provide adequate documentation on the data archive and the corresponding
data collection system. With data archiving systems, many data users
will be from outside the operations workgroup or agency that collected
the data. Thus, they may have little knowledge about the operations
data that is collected, how it is collected, and how it is processed
by operations before it is archived. Adequate documentation for data
archives primarily includes (but is not limited to) an "audit
trail" of how the data have been processed since they were collected
in the field (e.g., information about the results of quality control,
any summarization or aggregation steps, and any estimates or changes
that have been made to original, field-collected data), and information
on the data collection system (e.g., the type, location, and other identification
for detectors, the detectors that were considered "online"
for a particular hour or day, and information about equipment calibration
and maintenance).
4.2.5.2 Examples of Data Archiving
California PeMS Data Archiving
The Operations Division in Caltrans' Headquarters office has worked
with researchers at the University of California at Berkeley in creating
PeMS, a freeway Performance Measurement System. PeMS gathers raw freeway
detector data in real-time from several of Caltrans' districts,
including Los Angeles, Orange County, and Sacramento. The detector data
for these participating districts are summarized and processed as follows:
- Aggregates 30-second flow and occupancy values into lane-by-lane,
5-minute values;
- Calculates the g-factor for each loop, and then the speed for each
lane. (Most detectors in California are single loop, and only report
flow and occupancy. PeMS adaptively estimates the g-factor for each
loop and time interval.
- Aggregates lane-by-lane values of flow, occupancy, and speed across
all lanes at each detector station. PeMS has flow, occupancy, and speed
for each 5-minute interval for each detector station (one station typically
serves the detectors in all the lanes at one location);
- Computes basic performance measures such as congestion delay, vehicle-miles
traveled, vehicle-hours-traveled, and travel times.
- The data archives are then made available through the Internet
for anyone that has access privileges (i.e., the site is password-protected).
PeMS has several applications and built-in data summary and reporting
tools on the web site. One of these involves trip travel time estimates
and shortest routes. A user can bring up the district freeway map on the
Web browser, and select an origin and destination. PeMS displays 15 shortest
routes, along with the estimates of the corresponding travel times. PeMS
also provides travel time predictions – for example, what will be
the travel time 30 minutes from now. The travel time prediction algorithm
combines historical and real time data.
Another application, called "plots across space," can assist
in identifying bottleneck locations for more detailed investigation. To
use the application, the engineer selects a section of freeway, a time,
and a performance variable such as speed, flow, or delay. PeMS returns
a plot of the variable across space. Having quickly determined the existence
of these bottlenecks, the engineer can go on to determine their cause,
such as the location of interchanges, the highway geometry, large flows
at ramps, etc, and propose potential solutions to alleviate the bottleneck.
Furthermore, any scheme implemented to relieve a bottleneck can be rigorously
evaluated by a thorough before-and-after comparison.
The impetus for this data archive was state legislation that required
Caltrans to monitor the performance of their transportation system. Because
Caltrans has extensive detector coverage on freeways in several districts,
they chose to archive existing data rather than manually re-collect system
performance data. Caltrans' PeMS data warehouse is unique because
it is one of the few statewide operations data archives in existence.
Time and experience will reveal how useful a centralized statewide data
archive is to local agencies and workgroups at the district level.
Washington State DOT
WSDOT has been archiving freeway detector data since 1981 in some shape
or form, although early efforts were difficult because of the expense
of data storage and the difficulty of data transfer (pre-Internet). The
agencies have made numerous improvements to their data archive over the
years and, for the most part, the data archives have been institutionalized
within WSDOT. Freeway detector data (i.e., vehicle volumes and lane occupancy
by direction) are collected every 20-seconds from field controllers as
part of the Seattle area freeway management system. The data are converted
into estimates of vehicle speed and travel time, and summarized to the
5-minute level in the data archive. Quality control is also performed
before the detector data is loaded into the archive, and the archive documents
the number of data records that have failed quality control.
The Washington State DOT and the Washington State Transportation Center
(TRAC) at the University of Washington have developed a CD-based data
archive for the Seattle freeways, which they use to distribute the archived
operations data. Each data archive CD contains data extraction and summary
tools.
An analysis process developed by TRAC produces facility performance information
based on these data. This process also fuses the basic freeway surveillance
data with independently collected transit ridership and car occupancy
data to estimate person throughput. The data are used for a wide variety
of purposes, including answering key policy questions and evaluating operational
improvements such as ramp metering or HOV lanes, freeway performance monitoring,
pavement design, and freight performance analysis.
A paper by Mark Hallenbeck, Director of TRAC (Reference
14), summarizes the experience and lessons learned from this data
archiving system as follows:
- "The good news is that ITS surveillance systems being built
for traffic management purposes provide much of the data needed to perform
these types of analyses; therefore, lots of "new" data are
not necessary. Instead, the data already collected must be retained,
analyzed, and reported."
- "Storing and analyzing the data are not free. However, a large number
of potential users exist for the information that the surveillance system
generates. The key is to work with potential users to fund the modest
costs of storing, analyzing, and reporting the data already collected.
The agency must also determine who will operate the
database."
- "It is important to recognize that not all surveillance data
are "good." Therefore, the analytical procedures must be
able to identify and handle "unreliable" data. Mechanisms
should also be in place to repair and calibrate unreliable sensors.
(After all, unreliable data also hinder the operational control decisions
that are based on those data.)"
- "Because most traffic management systems have limited equipment maintenance
budgets, repair activities have to be prioritized. A key to consider
when balancing cost versus data availability is that obtaining useful
performance information does not require all detectors
to be operating. (Does an agency really need to report
volumes based on continuous data collection at 300 locations in the
urban area, or will 12 to 20 sites spread strategically around the region
reveal the important facts?) The reality is that necessary data can
be obtained with a moderate amount of planning and cooperation."
- "When this cooperation occurs, it becomes truly possible to
manage the roadway system. This is because an agency now has the data
necessary to understand how the roads are actually performing and how
that performance changes as a result of various management and operations
activities."
4.2.5.3 Field Measurements / Manual Data Collection
As previously discussed, Freeway Management Systems (FMS) offer the
potential to automate much of the data collection required for performance-based evaluations. That said, the reality is (as of the date of
this writing) that less than one-third of the freeways in the nation's
urban areas are instrumented with surveillance subsystems, the data collected
by many of these systems does not include all the information required
by outcome-based performance measures, detectors don't always
function properly, and some information just cannot be collected without
some sort of manual activity.
The "Manual of Transportation Engineering Studies" (Reference
15) is an updated and expanded version of the 4th edition to the Manual
of Traffic Engineering Studies. It is designed to "aid transportation
professionals and communities to study their transportation problems in
a structured manual, following procedures accepted by the profession."
The primary focus is on how to conduct "transportation engineering studies
in the field". Each chapter introduces a type of study
and describes the methods of data collection, the types of equipment used,
the personnel and level of training needed, the amount of data required,
the procedures to follow, and the techniques available to reduce and analyze
the data. Applications of the collected data or information are discussed
only briefly. Individual chapters include volume studies, spot speed studies,
travel-time and delay studies, inventories, transportation planning data
(e.g., origin – destination), traffic accident studies, traffic control
device studies, roadway lighting, and goods movement studies. Additionally,
there are appendices covering statistical analysis, written reports, and
presentations. Another valuable reference is the "Travel Time Data
Collection Handbook" (TTI, Report FHWA-PL-98-035, March 1998).
4.2.6 Reporting
As previously discussed, a good performance measuring program cannot
help but improve communications with an agency's customer base and
constituency, including decision makers and other agencies and entities
that are involved with the operation and management of the surface transportation
network. To achieve this improved communications, however, requires that
the performance measure data be translated into reports for dissemination
to stakeholders. Many of the criteria discussed for performance measures
are directly applicable to performance reporting, including reporting
results in stakeholder terms, that the information necessary to improve
decision making is conveyed in these reports, and that the information
is presented in a manner that is easy for the audience to understand and
interpret.
Visual depictions of the data can assist users in understanding trends,
operational performance, and the meaning of complex data interactions.
As an example, the Washington State DOT and the Washington State Transportation
Center (at University of Washington) convert their archived data (previously
discussed in section 4.2.5.2) into a variety of presentation graphs – showing congestion problems, benefits from operational improvements, comparisons
of alternatives, etc. – as a means of discussing freeway operations
and the associated policy issues with managers and other decision makers.
A few examples are shown and described below in terms of possible policy
and operational questions (from References
14 and 16).
What does the congestion picture really look like?
This basic "volume-by-time-of-day" graphic can be extended
to illustrate when congestion occurs and its effect on vehicle speed and
throughput. Average speed is color coded to indicate how conditions routinely
change by time of day. Then, because conditions vary considerably from
day to day, reliability at this point in the roadway can be examined by
defining "congestion" (in this case, the occurrence of LOS
F conditions) and reporting on the frequency with which that congestion
occurs. Graphically, it is possible to lay the "frequency of congestion"
over the same graphic that illustrates vehicle volumes and average speeds.
This is shown in Figure 4-1 (read "Vehicle Volume Per Lane"
on the left axis, and "Frequency of Congestion" on the right
axis.) This graphic shows that this specific location experiences LOS
F conditions more than 80 percent of all weekdays (four times a week).
It is also possible to see the slight decrease in vehicle throughput,
caused by congestion, which occurs in the heart of the morning peak period.
Figure 4-1: Estimated Frequency of Congestion, Volumes
and Speeds (Reference 14) D
Another approach is to produce an average daily corridor profile to depict
lane-occupancy percentage at each location along a corridor for a specified
direction of travel. As shown in Figure 4-2, the resulting graph is a
contour map, color-coded according to the estimated congestion
level.
Figure 4-2: "Temperature" Diagram of Traffic
Flow Conditions (Reference 18) D
What delays are the public experiencing?
Using vehicle speed data that can be obtained from the freeway surveillance
system, it is possible to estimate vehicle travel times throughout the
day. Again, by saving these data, it is possible to describe not only
today's travel times (excellent for measuring the effects of an
incident), but also an entire year's travel times. Graphics like
Figure 4-3 allow the analysis and reporting of travel conditions throughout
the day.
Figure 4-3: Travel Times (by time of day) for a Specific
Route (Reference 14) D
The graphic illustrates the actual travel times experienced (by time
of day) for a specific route of interest (in this case the northbound
trip using the southern half of the I-405 corridor). The green line represents
the average travel time for a trip starting at a given time. The red line
illustrates the 90th percentile trip. This is essentially the worst travel
time a motorist could expect to experience once every two weeks. (As previously
discussed, the Mobility Monitoring Program uses the "Buffer Index"
as a measure of travel reliability. Changing the graphic to illustrate
the 95th percentile trip time would represent the Buffer Index.)
Figure 4-3 also includes a measure of "congestion frequency." In this
case, "congestion" is defined as the average speed for a trip of less
than 35 mph. The blue histogram describes the frequency with which a motorist
can expect to experience a trip that averages less than
35 mph for the entire trip duration.
Statistics such as the ones presented in the Figure 4-3, when tracked
over time, allow freeway operations personnel to measure and present the
broad, overall effects of the traffic control strategies they implement.
These statistics also lead to more informed discussion of the travel conditions
that exist (e.g., How bad is off-peak congestion? Is off-peak operation
of the service patrol program necessary?), which in turn leads to more
informed debate about the need for and relative merits of alternative
operations strategies.
What improvements have ramp metering produced?
Any time significant operational changes are implemented within the
surveillance area, the resulting changes in vehicle throughput and performance
can be measured. WSDOT has operated ramp meters in the afternoon on SR
520 in Seattle for a number of years. Until recently, the ramp meters
were not used in the morning. When morning metering was implemented, significant
improvements in freeway performance occurred. Those improvements, illustrated
in Figure 4-4, included an increase of over 170 vehicles per lane per
hour and a decrease in the occurrence of LOS F conditions of one day per
week. Ramp meters may not have "solved" the congestion problem;
but they did make a considerable improvement.
Volume and Congestion on Eastbound SR-520 on the Viaduct
Figure 4-4: The Effect of Ramp Meters on Vehicle Volume
(per lane) Throughput and Frequency of LOS F Operations (Reference
14) D
4.2.7 Emerging Trends and Needs
The use of performance measures – particularly those that measure
"outcome" – for operating and managing the transportation
network, and for longer-range planning and decision making, is itself
an emerging trend. The same holds true for data archiving. A recent problem
statement developed by the TRB Committee on Freeway Operations, entitled
"Freeway Performance Monitoring, Evaluation, and Reporting",
states: "a consensus does not exist and technical guidance has not
been developed regarding the appropriate measures, methods, data requirements,
evaluation tools, procedures, level of effort, and resources required
to properly support the monitoring, evaluation, and reporting of freeway
performance. Research and technical guidance is needed to provide direction
and ensure that transportation professionals are effectively integrating
the performance of freeways into the appropriate planning and decision
making processes of agencies."
References 2, 3, 5, 6 and 10 address future issues and research needs,
as summarized below:
- Gather examples, case studies, and tools to effectively communicate
performance measures to policy makers, legislatures, and the public.
Information is needed on how performance measurement is effectively
communicated to decision makers to allow them to make informed decisions.
- Clarify (standardize) terminology and differences between organizational
or managerial measures and system measures. Align the definition of
goals across the industry to the extent possible, then standardize the
measures used. Create consistent standards, so that performance measures
can be reliably compared across agencies. Reporting standard errors
or confidence intervals should be included.
- Develop training for managers and policy makers to apply and use
performance measurement systems. Provide tools for managers and policy
makers in applying and using performance measures.
- Gather information on how to incorporate community or society goals
(or "soft" measures) into the performance measurement process.
Create quality-of-life and sustainability performance indicators.
- Operational performance measures that address evacuations from man-made
or natural disasters are needed, particularly for use during the operations
of these events and tailoring strategies to maximize / optimize performance
based on these measures.
- The maximum benefits will not be realized until considerable integration
is achieved. Performance measurement can and should be the lingua
franca for such integration, with mutually acceptable and well-defined
outcomes acting almost like common denominators.
- With respect to archived data, significantly enlarge roadway sensor
coverage (i.e., freeways and arterials) and experiment with data sources;
transit operating data should be added to get a more complete system
picture; encourage the local use of the archived data; improve the calibration
and maintenance of data collection equipment; and add "event"
databases (e.g., incidents, weather and work zone locations, which have
significant impacts on roadway travel times).
|