Model Systems Engineering Documents for Adaptive Signal Control Technology (ASCT) Systems

G. VALIDATION PLAN GUIDANCE

The chapters required for the Validation Plan document are:

1. Purpose of Document
2. Scope of Project
3. Referenced Documents
4. Conducting Validation
5. Validation Identification

This document describes the activity of validation that the system being built meets the user needs and scenarios developed in the concept of operations. The validation documents will generally include three levels of validation documents:

• A plan to initially lay out the specific validation effort
• The user's/operator's manual and/or a validation plan that defines the detailed operational procedures
• A report on the results of the validation activities

To ensure user needs and scenarios are validated by this activity, trace each need and scenario into a validation case, then into appropriate steps in the validation procedure.

A separate Validation Plan and procedures may be minimal for the simplest projects, especially where the system is commercially available and does not involve any custom software development, and where the agency staff have a very clear understanding of the purpose of the system. Preparation a validation plan is strongly advised if:

• The system is more complex
• There are several separate validation activities
• Multiple deployment sites are involved
• Multiple stakeholders have to be satisfied

There is also the question of how comprehensive to make the validation effort. It is impossible to validate all possible combinations of actions under all possible operational situations. A good rule of thumb is: if it was important enough to write down as a need or scenario, then it should be validated, at least once. This may not, for example, validate all possible failure mode conditions or all possible incident scenarios. In-process⁵ validation performed on the needs will help ensure that end to end validation of the system will meet the stakeholder needs.

Once the Validation Plan has been prepared, use this checklist to ensure all critical information has been included.

Does the Validation Plan answer all the questions of who, what, where, and when?

Does the Validation Plan make clear what needs to happen if an unexpected situation or a failure is encountered?

Does the Validation Plan document the configuration of the hardware and software?

Are all applicable needs and scenarios traced to a validation case?

1 Purpose of Document (Chapter 1 of the Validation Plan)

This chapter identifies the type of validation activity to be performed within this Validation Plan. For instance, this activity may validate the entire system, a sub-system, the deployment at a site, a burn-in, or any other validation activity called for in a relevant Program Plan or SEMP.

2 Scope of Project (Chapter 2 of the Validation Plan)

This chapter gives a brief description of the planned project and the purpose of the system to be built. Special emphasis is placed on the project's user needs and issues that must be addressed and validated.

This chapter also describes the environment in which the project operates. It identifies the organization structures that encompass all stakeholders. It also gives a brief description of the role to be played by each stakeholder. This includes ad hoc and existing management work groups and multi-disciplinary technical teams that should be formed to support the project.

3 Referenced Documents (Chapter 3 of the Validation Plan)

This is a list of all documents used in the preparation of this Validation Plan. This almost always includes the Project Plan, the SEMP (if one was written), and the applicable Requirements Documents. However, reference of other documents, such as descriptions of external systems, standards, a Concept of Operations, and manuals may need to be included.

4 Conducting Validation (Chapter 4 of the Validation Plan)

This chapter provides details on how validation is accomplished. It defines: who does the validation; when and where it is to be done; the responsibilities of each participant before, during, and after validation; the deployed hardware and software configuration; and the documents to be prepared as a record of the validation activity. This chapter defines how anomalies are to be handled (that is, what to do when an unexpected situation or a failure occurs during validation).

In general, the following information should be included in this chapter:

• A description of the participating organizations and personnel and identification of their roles and responsibilities. This may include for example, a validation conductor, validation recorder, operators, and/or engineering support.
• Identification of the location of the validation effort, that is, the place, or places, where the validation must be observed.
• The deployed hardware and software configuration for all of the validation cases, including hardware and software under validation and any supporting equipment, software, or external systems. Several configurations may be necessary.
• Identification of the documents to be prepared to support the validation, including Validation Procedures, a Validation Report and descriptions of special equipment and software.
• Details on the actual conduct of Validation, including:
  • Notification of participants
  • Emphasis on the management role of the validation conductor
  • Procedures for approving last minute changes to the procedures.

The processes for handling a failure, including recording of critical information, determination of whether to stop the validation, restart, or skip a procedure, resolution of the cause of a failure (e.g. fix the software, reset the system, and/or change the requirements), and determination of the re-validation activities necessary as a result of the failure.

5 Validation Identification (Chapter 5 of the Validation Plan)

This chapter identifies the specific validation cases to be performed. A validation case is a logical grouping of functions and performance criteria (all from the Concept of Operations Documents) that are to be validated together. For instance, a specific validation case may cover all the control of traffic during the AM peak hour. There may be several individual objectives and associated performance measures that define this capability, and they all are validated in one case. The actual grouping of objectives into a case is arbitrary. They should be related and easily combined into a reasonable set of procedure actions.

Each case should contain at least the following information:

• A description name and a reference number.
• A complete list of needs and scenarios to be validated. For ease of tracing of needs and scenarios into the Validation Plan and other documents, the needs and scenarios are given numbers. They can be accurately and conveniently referenced without repetition.
• A description of the objective of the validation case, usually taken from the wording of the need or scenario, to aid the reader understanding the scope of the case.
• Any data to be recorded or noted during Validation, such as expected results of a step. Other data, such as a recording of a digital message sent to an external system, may be required to validate the performance of the system.
• A statement of the pass/fail criteria. Often, this is just a statement that the system operates per the need or scenario.
• A description of the validation configuration. That is a list of the hardware and software items needed for validation and how they should be connected (this should be in most cases the deployed system configuration). Often, the same configuration is used for several validation cases.
• A list of any other important assumptions and constraints necessary for conduct of the validation case.

previous | next