CHAPTER 1. INTRODUCTION

Note: Unless accompanied by a citation to statute or regulations, the practices, methodologies, and specifications discussed below are not required under Federal law or regulations.

BACKGROUND

Analytical tools and simulation models are the primary tools used to support evaluation, design and planning of operational measures and policies aimed at improving mobility and service quality offered by our transportation systems. Models vary in level of detail, geographic scope and granularity, temporal dynamics, behavioral richness and so on, usually tailored to the needs of a particular study or application. However, in all cases it is expected that the models provide a reasonable and realistic representation of reality that is appropriate for the application at hand. At their core, models typically entail mathematical relations and algorithmic procedures that take certain variables as input to produce the output variables of interest. The relations and procedures are typically specified up to a set of parameters whose values are estimated for the context at hand. The process of estimating the model parameters, and otherwise determining the values of parameters that govern various aspects of the model so as to best represent the given area and context is referred to as calibration. Calibration methods vary in rigor and complexity depending on the nature of the models and the available data. Calibration inherently requires observation of the existing system conditions, as the analyst seeks to estimate the model parameter values so as to best match the available observations.

Current practice typically assumes that parameter values obtained using observations of current conditions will remain applicable under different future conditions, which may thus be modeled or simulated by providing input values reflecting these future conditions. Whether they describe mathematical relations between physical quantities (e.g., fundamental diagrams at a macroscopic level) or characteristics of individual behavior (e.g., driver risk aversion or traveler preferences at a microscopic level), parameters values may be different under these future conditions than they are under the conditions for which the model was calibrated. This is especially true when new technologies are introduced (new modes, new fuels or propulsion technologies, connected and/or automated vehicles), or major changes are implemented in the operation of the transportation system. To the extent that models are developed to evaluate contemplated interventions and measures to be implemented in the future, rather than merely replicating present and past conditions, it is essential that the models have the capability to produce realistic predictions of the impact of these interventions under potential future conditions. In addition to being able to represent the changes through the variables incorporated in the model formulation/specification, it is highly desirable that the parameter values used in prediction be reflective of the future conditions under evaluation. Unfortunately, typical calibration methods require actual observations of the conditions of interest, and hence are generally possible only for conditions that may be observed. To the extent that the future conditions have not yet occurred, it is obviously challenging to perform a similar calibration exercise for future conditions. Addressing this challenge requires changes not only in calibration procedures and practices, but also in model formulation and development. The main objective is for traffic models to produce meaningful and reliable predictions of system performance under future conditions and contemplated policy/operational interventions.

Accordingly, different calibration methods need to be developed so that the tools are calibrated to produce results that are reflective of what the future condition will be. The scope of this study is to provide information for transportation professionals on how to calibrate analytical/simulation models and tools to data that are reflective of the target future conditions. The developed methodology/framework will allow agencies and transportation professionals to develop and calibrate analytical/simulation tools that are valid for a wide range of different future conditions.

OBJECTIVE

Traffic analysis tools (TATs) are designed to assist transportation professionals in evaluating the transportation improvements that best address the transportation needs of their jurisdiction. TATs can help practitioners improve the decision-making process, evaluate and prioritize improvement alternatives, and improve project design and operations. Over the past 15 years, the Federal Highway Administration (FHWA) TAT Program has developed the TAT Toolbox, a compendium of TAT informational documents (see https://ops.fhwa.dot.gov/trafficanalysistools/index.htm). The TAT Toolbox has helped to establish consistency in practice for traffic analysts across the nation.

Calibration is a key step in the application of TATs to a project or study. Calibration is the adjustment of model parameters to improve the model’s ability to reproduce local driver behavior and traffic performance characteristics. The importance of calibration cannot be overemphasized because no single model can be expected to be equally accurate for all possible traffic conditions.

Current practice calls for analysts to calibrate their analytical tools to a base (or existing) condition and then use those tools to predict performance of a future condition. This practice is consistent with information currently in the TAT Toolbox. However, many times these future conditions incorporate improvements that are significantly different than the base condition modeled when the analysis tool was calibrated. This can inhibit accuracy of the performance outcomes.

Different calibration methods are developed so that the tools are calibrated to data that are reflective of what the future condition will be. The scope of this report is to provide information for transportation professionals on how to calibrate analytical tools to data that are reflective of the conditions that the tool is being used to predict future performance. Users should be provided with an understanding of how to amass, understand, prepare, and analyze a wide range of future conditions that impact performance. The methodology/framework that will be presented in this report will allow agencies and transportation professionals to calibrate their analytical tools to enable validity no matter how different the future condition modeled is from the base condition.

The objective of this report is to describe a methodology/framework allowing traffic analysis tools calibration to account for not only base conditions, but the future conditions being modeled. This will enable validity of the analysis tool for improvement alternatives being considered. Such a methodology/framework will enable more accurate traffic analyses, which will in turn lead to improved trust in analysis tools and improved transportation decision-making overall.

CHALLENGES AND SOLUTIONS

Addressing the above objective entails several challenges that are unique to this effort. The challenges arise from the very nature of the problem itself, which entails determining model parameter values for conditions that do not yet exist, and hence my not be observable, either directly or indirectly. Below are selected challenges and proposed mitigation approaches.

CALIBRATION NEEDS

Calibration methods vary in rigor and complexity depending on the nature of the models and the available data. Statistical or econometric estimation is the most common method for finding parameter values that provide a best fit to the available data. Other optimization approaches with varying degrees of formalism may also be used to make the model outputs best match available data on the system of interest. Conceptually, in simple terms, the model output Y can be formulated as a function of input variables X, and a set of parameters β that characterize that relation, i.e.:

Where ε is an error term that is the discrepancy between the actual value of the output Y and the one calculated using the function f (.) The calibration problem, in its simple form, is to find the values of β, given joint observations of the input and output variables (X,Y). Calibration, inherently, requires knowledge and observation/measurement of actual conditions in order to be meaningful. Application of the model for prediction generally entails the input of new values of X (say at some future date), and calculation of the corresponding Y using the function with calibrated parameter values. What is typically assumed is that the relation established for "current" or "past" conditions will continue to hold in the future, including the values of the model parameters β. The only elements assumed to change when the model is applied to forecast future conditions consist of the values of X. These are typically forecast independently, or determined as the result of a particular intervention.

In what follows, future conditions, with or without intervention, are referred to as a scenario. A scenario is defined by set of operational conditions (reflecting external events, such as weather, demand surges, etc.), interventions (infrastructure changes, control actions, dynamic system management schemes, etc.), as well as characteristics of the general activity system (land use, activity locations) and associated technologies (e.g., connected and autonomous vehicles, Internet of Things, smart cities). The features and characteristics of a given scenario are represented through the values of X in the above simple model.

In general, the longer the horizon over which a scenario is defined, i.e., the further out into the future one is trying to forecast, the greater the uncertainty associated with the values of X. This is especially true not only for the activity system variables and the associated technologies, but also the social fabric and associated preferences/lifestyles/norms. Traffic planners generally deal with the uncertainty in the future values of X by defining alternative future scenarios corresponding to different rates of population and economic growth, alternative assumptions on urban core densification vs. sprawl, and so on. However, in modeling the impacts of these changes for a given scenario, the values of model parameters β are assumed to remain constant. There are of course many reasons to assume that these parameters would change due to the same forces that are influencing the values of X, but also cases where these parameters may remain stable and still applicable to the new scenario conditions.

The main problem addressed in this document is how to develop and calibrate models so that the output they produce for a future scenario is meaningful, and provides a realistic and accurate depiction of future conditions defined by the scenario of interest. Note that model development and calibration are intrinsically related from the standpoint of the model's ability to predict future scenario conditions. It is important that the model be responsive to the scenario features and associated interventions, i.e., that it allows the analyst to represent these features in the model—this is an issue of model specification. As a pre-requisite, the specification should be responsive to future scenarios, and include the appropriate X variables in the model.

Given the specification, the main challenge addressed is how to ensure that the parameter values used in applying the model to predict traffic system performance under the future scenario of interest are adequate, i.e., that the model is "calibrated for future conditions."" Of course, since these future conditions have not yet occurred at the time the model is used, it is not possible to directly observe actual conditions for such scenarios as a basis for model calibration. Addressing this challenge requires a significant shift in the mindset of traffic modelers—from model estimation and calibration aimed at replicating existing and past conditions, to greater emphasis on prediction quality and accuracy.