3.4.5       Trade Studies

Introduction

Trade studies compare the relative merits of alternative approaches to ensure that the most cost-effective system is developed. They maintain traceability of design decisions back to fundamental regional operations strategies, needs, and requirements. Trade studies do this by comparing alternatives at various levels for the system being developed. They may be applied to concept, design, implementation, verification, support, and other areas. They provide a documented, analytical rationale for choices made in system development.

Description

Trade studies can be used in various phases and at different depths throughout system development to select from alternatives or to understand the impact of a decision. The inputs vary depending on what is being analyzed. For example, early in system development, the alternatives may be broad concepts. Later, in the design phase, they will be design alternatives.  During hardware and software implementation, the implementation team may look at alternative hardware components or language and development environment alternatives for software.  In each case, the alternatives are compared against a set of criteria and the decisions are documented.  Stakeholder input may be solicited to define and rate the criteria and to validate the results. The analysis may be done qualitatively, or by a model or simulation.  Trade study activities typically include the following activities.

Figure 28: Trade Study Activities

(Source: FHWA)

Understand the objectives

First, define the question the trade study is to answer. This may be the selection of the most cost-effective concept or design. It may be to narrow down choices to support a more detailed evaluation. It may be to demonstrate that the choice made is the best one.

Define criteria and weightings

Experienced staff will draw from the available inputs to identify the key evaluation criteria for the decision under consideration. These are typically measures of effectiveness, metrics that compare how well alternatives meet the needs and requirements. Examples are capacity [vehicles per hour], response time, throughput, and expandability.  Ideally, the criteria are developed early in the trade study process, before the alternatives are selected, to avoid subconsciously biasing the criteria to favor a pre-determined preferred alternative.

Generally, there are multiple criteria and they aren’t always correlated.  For example, an alternative that minimizes schedule impact might increase risk or cost.  It is important to rank or weight the criteria so that the selected alternative scores best against the most important criteria.  Stakeholder input may be solicited to assist with the identification and weighting of the criteria based on relative importance.

Identify alternatives

A trade study starts with alternative concepts or designs that are to be evaluated. Be sure that all reasonable alternatives are on the table.  It is often good practice to include a “Do Nothing” alternative as the base case, depending on the nature of the trade study.

Evaluate alternatives

The identified alternatives are evaluated against the defined criteria.  Generally, the emphasis is on performance criteria such as speed or effectiveness. For each alternative, the criteria may be evaluated quantitatively or qualitatively supported by methods including simulation, performance data gathered from similar systems, surveys, and engineering judgment. These disparate evaluations are merged using the weighting factors to give a measure of overall effectiveness for each alternative.

In most trade study analyses, the cost and schedule impacts associated with each alternative are estimated.  Estimate the cost of each alternative including the development cost and the life cycle cost, which includes operation and maintenance. Use the techniques of risk assessment to compare the alternatives relative to technical or project risk. Determine the impact of each alternative on the schedule. Eliminate those that introduce too much cost or schedule risk.

For complex trade studies with many alternatives, the overall evaluation may involve multiple passes where a limited set of criteria is applied to a broad set of alternatives in order to focus additional analysis on the most promising alternatives.  A more complete analysis can then be performed on the subset of alternatives that scored well in the first pass analysis.  Sensitivity analysis may also be used, especially with simulation, to see the effect of changes in sub-system parameters on overall system performance. The sensitivity analysis and the evaluations may suggest other, better alternatives.

Select and document the preferred candidate, typically using a trade study matrix that lists the alternatives on one axis, the criteria on the second axis, and the relative scores of each alternative versus each criteria in the body of the matrix.  Plotting each alternative's overall effectiveness, based on the combined weighted metrics, cost, or the other factors is also useful for evaluating the relative merits of each.

 

Validate the choice

The trade study documentation should support review of the decision by stakeholders, so the trade matrix should be easily comprehensible to management and other stakeholders who may not be as familiar with the technical details.  Document the decision, the parties involved, and the rationale behind it to provide traceability back to the higher-level objectives. The documented trade study is also a repository of alternatives in case a change is needed down the road.  It is also possible that stakeholder input may require iteration and refinement of the trade analysis to reflect evolution in criteria, additional identified alternatives, or desired evaluation changes.

 

Checklist

The following checklist can help answer the question “Are all the bases covered?” once the activities above have been planned or performed.  The checklist is not sufficient by itself.

R  Has a broad and reasonable selection of alternatives been examined?

R  Was a fair and balanced set of criteria used to evaluate the alternatives?

R  Is the selection rationale documented?

R  Does the rationale for the trade study conclusions flow out of the trade study inputs –strategies, needs, and/or requirements?

R  Is the sensitivity of system effectiveness to changes in key parameters well understood?

R  Have key stakeholders reviewed and support the trade study results?