Model Systems Engineering Documents for Adaptive Signal Control Technology (ASCT) Systems

C. THE SYSTEMS ENGINEERING DOCUMENTS

OVERVIEW

The system engineering documents that need to be produced using this process are:

• Concept of Operations
• System Requirements
• Validation Plan
• Verification Plan

A template is provided for each of these documents in Appendix A and is described below. In each case, the template is intended as a guide. There is no limitation (upper or lower) on the amount of information included in each document. However, it needs to be sufficient to illustrate that your needs are clearly defined, alternative methods of accommodating those needs have been investigated, there is clear justification for having selected this course of action, there will be no unintended consequences, the design of the system will accommodate constraints within which it must function, the correct operation of the system will be verified and the success of the system in meeting your needs will be validated.

Each of the templates sets out a structure that should be followed, although you are at liberty to modify the chapter titles. The description you provide in each section need not be extensive, provided it covers the elements mentioned in the template. The relationships between the guidance document, templates and the final systems engineering documents you will produce are illustrated in Figure 4.

Figure 4. Relationships between This Guide and Final Documents

RESPONSIBILITIES

The responsibility for preparation of these documents rests entirely with the agency. While information may be sought from vendors during the preparation of these documents, a vendor should never be tasked with the responsibility of preparing the systems engineering documents. This would be a clear conflict of interest. However, it is appropriate for agencies to seek support from qualified consultants who are independent of vendors. Of the documents described in this guideline, only the verification procedures may have significant input from a vendor, since verification testing requires exercising all elements of the system that are the subject of requirements. The vendor should not prepare the verification plan, and should only prepare procedures if the user manuals are inadequate or not yet completed. Any documentation prepared by a vendor must be thoroughly audited by the agency to confirm its completeness and compliance with the Concept of Operations and the System Requirements.

CONCEPT OF OPERATION

The Concept of Operations is written from the perspective of the system operator, whose story is documented therein. The primary audience for the concept of operations is composed of stakeholders who will share the operation of the system or be directly affected by it. The stakeholders are tasked with describing their needs and objectives succinctly enough to determine what functions the proposed system must be capable of fulfilling. Stakeholders should include system managers, operators, and maintenance staff responsible for the system; administrators, decisionmakers, elected officials, other non-technical readers may also be included. This document is the key to the success of the project. Every element of all the other documents must be able to be traced to statements of need in the Concept of Operations. Detailed guidance to help you with each chapter is included in Section D.

The Concept of Operations is a non-technical description of HOW the system will be USED. This provides a bridge between the often vague needs that motivated the project to begin with and the specific technical requirements. There are several reasons for developing a Concept of Operations.

• Get stakeholder agreement to describe how the system is to be operated, who is responsible for what, and what the lines of communication are
• Define the high-level system concept to enable evaluation of alternatives
• Define the environment in which the system will operate
• Derive high-level requirements, especially user requirements
• Provide scenarios and criteria to be used for validation (via the Validation Plan) of the completed system

When defining the concept and justifying its selection, you must be extremely careful to relate the expected capabilities and potential benefits to your identified deficiencies and limitations.

REQUIREMENTS

The target audience for this document is comprised of technical staff, system users, system designers and vendors. This document describes WHAT needs to be achieved by the system. It specifically should not describe HOW the system will satisfy the needs. Each of the requirements listed in this document must be linked to a corresponding need described in the Concept of Operations. If you define a requirement that cannot be traced to a statement of need defined in the Concept of Operations, then either the Concept of Operations document must be revised (so its readers will clearly understand why the requirement exists), or the requirement should be deleted. Detailed guidance to help you with each chapter is included in Chapter E of this guidance document.

The types of requirements typically needed to define a proposed system are described in Table 1, although you do not need to differentiate your requirements into these categories.

Table 1. Categorization of Requirements

Requirement category	Description
Functional requirements	What the system is to do
Performance requirements	How well it is to perform
Non-functional requirements	Under what conditions it will perform
Enabling requirements	What other actions must be taken in order for the system to become fully operational
Constraints	Limitations imposed on the design by agency's policies and practices, such as type of software, type of equipment and external standards
Interface requirements	Definitions of the interfaces between sub-systems or with external systems
Data requirements	Definitions of data flows between sub-systems or with external systems

In all cases, it is essential to describe the functional, performance, non-functional and enabling requirements and the constraints. If the proposed system will interface with another system or requirements are also required at the sub-system level (which may be the case if you are requiring some customization or new modules specifically for your application), then interface requirements and data requirements may also be required.

This document does not define how the system is to be built. This document sets the technical scope of the system to be built. It is the basis for verifying (via the Verification Plan) the system and sub-systems when delivered.

VERIFICATION PLAN

The target audience for this document is the same as for the Requirements document. This document describes how the system will be tested to ensure that it meets the requirements. Detailed guidance to help you with each chapter is included in Chapter F.

These documents plan, describe and record the activities which verify that the system being built meets the user needs and scenarios developed in the concept of operations, by fulfilling the requirements described in the requirements documents. Usually, for even moderately complex systems, the following three levels of verification documents are prepared:

• A plan to initially lay out the specific verification effort
• The detailed verification plan that defines a detailed mapping of the requirements to verification cases
• A report on the results of the verification activities

A critical issue is assuring that all requirements are verified by this activity. This is best done by tracing each requirement into a verification case and then into appropriate steps in the verification procedures.

It will not describe detailed procedures for data collection and analysis, although existing procedures defined elsewhere may be referenced. In general, the procedures that will map to the verification plan will be developed by the system developer, supplier or vendor, and approved by the agency, once the system (or appropriate sub-system) has been installed.

Conducting the verification is generally the responsibility of the system developer, supplier or vendor, although the verification tests may be undertaken by the agency. The verification tests should be witnessed by the agency and the results should be audited by the agency to ensure their veracity.

VALIDATION PLAN

The target audience for this document is the same as for the Concept of Operation. This document (or set of documents) describes how the performance of the system will be measured to determine whether or not the needs expressed in the Concept of Operations have been met. It will describe the measures of performance that will be used, the data that will be needed and the type of analysis that will be appropriate. This will include a description of the data and analysis that will need to be available from the proposed system, as well as other data collection and analysis that will be undertaken external to the system. Detailed guidance to help you with each chapter is included in Chapter G.

These documents plan, describe, and record the activities which validate that the system being built meets the user needs and scenarios developed in the Concept of Operations. Usually, for even moderately complex systems, the following three levels of validation documents are prepared:

• A plan to initially lay out the specific validation effort
• The user's/operator's manual and/or a validation plan that defines the detailed operational procedures
• A report on the results of the validation activities

A critical issue is assuring that all user needs and scenarios contained in the Concept of Operations are validated by this activity. This is best done by first tracing each need and scenario into a validation case and then into appropriate steps in the validation procedure. The Validation Plan may describe detailed procedures for data collection and analysis, although existing procedures defined elsewhere may be referenced. Validation is the responsibility of the agency, and cannot be delegated to the system supplier or vendor.

PROCUREMENT PLAN

The method of procurement of an ASCT system has a major impact on the ability of an agency to select the most appropriate system for its situation. 23 CFR 635.411 requires procurement of ITS systems and components to be competitive, unless certain conditions are met. However, competitive procurement does not mandate "low-bid" procurement. In fact, in procurement of complex IT and ITS system, "low-bid" is generally inappropriate. The requirement for competitive procurement is satisfied through an RFP or similar process that allows for careful evaluation of the extent to which each requirement is met and the suitability of the method used to satisfy the requirement. The evaluation process may also include consideration of costs to determine best value for money.

The requirements for a complex system may be classified as mandatory, desirable and optional. All should be considered in the evaluation. Very often, "optional" can be used to define requirements that will be important at a future date, but need not be included in the package purchased at this time. This ensures that future system expansion and migration paths are not precluded by the initial system design and capabilities.

Steps in the procurement process that may be applicable to your situation are described below.

Request for Information (RFI)

Once your requirements have been drafted, there may be some that you are not sure can be met by any, or a sufficient number, of the available systems. This is a point at which you need to consider whether it will be appropriate to purchase a commercially available system (with or without some customization) or develop a unique system. When this occurs, it is appropriate, and perhaps desirable, to release to vendors a formal request for information (RFI) focused on the specific requirements about which you are uncertain. Response to an RFI should not be a condition of future participation by a vendor in the procurement process.

Use of the RFI will allow you to decide whether or not your requirements can be satisfied by a commercially available system. If not, you can then decide whether to modify your requirements or develop a much more detailed set of requirements and a specification that would be appropriate for development of a unique system.

Industry Review of Requirements

More comprehensive feedback from vendors can be obtained by distributing a draft version of your requirements to vendors. This is appropriate when you have developed requirements that will involve some customization or may include assumptions about appropriate technology. Provided you have included a Concept of Operations and clear statements of need, this provides the opportunity for vendors to contribute in two effective ways.

Vendors will recognize requirements that assume a design that is different from theirs and this gives them the opportunity to redefine the requirement in a manner that does not preclude their system simply because of the wording or structure of the requirement. Vendors will also recognize high-cost customization or new development that would be necessary to satisfy a requirement, and this also gives them the opportunity to suggest alternatives that would substantially reduce the procurement cost or streamline the development.

Responses from vendors should be treated as confidential and not shared with other vendors. If a vendor expects a response to be shared, he is less likely to offer advice that he considers proprietary and part of his competitive advantage.

Request for Qualifications (RFQ)

A request for qualifications (RFQ) is appropriate when there are mandatory requirements placed on the capabilities and experience of the vendor. This is a means of reducing the agency's risk. It provides the opportunity to assess a vendor's financial stability, their track record in providing support and training, and proof that an advertised system is fully operational and successful. It should NOT be used simply to reduce the workload of the agency in evaluating responses to an RFP by limiting the number of vendors permitted to respond to a detailed specification or RFP. It should be used after requirements have been prepared and the agency is certain that vendors who show satisfactory qualifications will be able to also satisfy the technical requirements.

Request for Proposals (RFP)

A request for proposal (RFP) is the key vehicle for assessing the extent to which an ASCT system will satisfy the agency's requirements. Using a "best value" approach supported by systems engineering provides the most effective use of the system requirements in the procurement process. As illustrated in Figure 5, the requirements are referenced at every step in the process to help guide the selection. While it may contain mandatory, desirable and optional requirements, the vendor should be required to explain how their system satisfies each requirement, and not simply be permitted to provide a Yes/No or Pass/Fail response. Because of the complex nature of ASCT systems, few requirements can simply be submitted to a Pass/Fail test. There are often different ways in which systems may claim to satisfy a requirement, and not all methods will be equally suitable (or acceptable to an agency) in all situations. Each answer should be evaluated to determine firstly whether or not the Yes/No answer is accurate, secondly the extent to which it satisfies the requirement, and thirdly as to whether the method of satisfaction is acceptable to the agency

In addition, the method by which a vendor satisfies a requirement may have other implications, such as the effect on work practices, staff efficiency, or requiring additional equipment or software to be efficiently implemented. The method used to compare responses to an RFP should include a means of accommodating the assessment of compliance. The agency must also select an appropriate means of accommodating variation in cost between vendors. Two methods are commonly used:

• A simple cut-off point of acceptable cost, which would allow an agency to obtain the system with the greatest utility within a pre-determined budget ceiling.
• A best-value approach, which would relate some measure of utility to the proposed cost of each competing alternative. This allows the agency to pick the best value for money from two or more closely comparable systems.

Figure 5. Best Value Supported by SE Analysis

Low-Bid

The use of "low-bid" contracting is rarely a satisfactory method of procuring a complex software system such as ASCT as a complete design-bid-build package. However, it is often possible and appropriate to separate the intelligent portion of a system (such as central software and servers, and local controller software) from the physical components that can be clearly and concisely specified, and construction work that can be undertaken by a qualified contractor.

If this approach is adopted, it is generally appropriate to first use the RFP process to select the ASCT system and vendor. Then the specifications can be completed for the equipment and services to be procured under a low-bid contract with confidence that there will not be compatibility issues between field equipment and the ASCT system.

When purchasing a product, if it is tightly specified, then there is limited risk that product will fail to meet the purchaser's expectations. However, when procuring an ASCT system, much of what is being purchased are services, not simply a product. If an agency is restricted to a traditional "low-bid" process, a strategy that builds sound systems engineering into the rigid procurement process (illustrated in Figure 6) would include the following:

• Include System Requirements as part of the special provisions of a standard bid set based on PS&E
• Require submittals at an early stage of the contract to explain in detail the vendor's method of compliance with the requirements
• Require submittals at an early stage of the contract to fully demonstrate compliance with the key requirements
• Require a detailed acceptance test plan as an early submittal prior to any construction

It is critical that the compliance with the requirements is demonstrated as early as possible, preferably before any of the project budget is spent.

Figure 6. Low-Bid Supported by SE Analysis

Verification Plan

A very important component of the procurement plan is the verification plan. It must be prepared before a formal RFP is issued and, regardless of the procurement approach you have selected, before a contract is signed. The verification plan should set out the method by which each requirement will be verified as satisfied, who will undertake verification tests and when within the process each verification test will be conducted. The plan will also include a test and verification matrix that will identify which requirement is verified by each test.

It may be appropriate for compliance with some requirements to be demonstrated prior to selection of a vendor or system, particularly for mandatory requirements that cannot be demonstrated by reference to existing operation with other agencies. For example, if you require an ASCT system to operate with flashing yellow arrow for permissive left turns, and a candidate system has not previously demonstrated such operation successfully, it is appropriate to require this demonstration before finalizing your system selection.

The verification plan should be provided to potential vendors with the RFP. In general, the test procedures are not part of the verification plan at this stage. The procedures can only be written after the ASCT system has been selected and any customization designed. Note that it is usually most appropriate (and cost effective) to schedule tests to occur at various stages during the project and not leave all testing until after installation is complete.

Market Research Approach

A commonly used but often less successful approach is to begin with market research and employ a low-bid process, illustrated in Figure 7. A detailed set of specifications is prepared based on a list of features and capabilities identified during the market research. It is extremely difficult to avoid specifying unique technology using this approach and the only control over brand choice comes at the submittal stage. The unfortunate experience of many agencies has been that many requirements are only discovered during the acceptance phase at the end of the construction period. This approach has been used for many of the adaptive installations whose final operation has not been satisfactory to the implementing agency.

Figure 7. Market Research / Low Bid Approach

Procurement and Verification References

For further guidance on procurement practices for ASCT and ITS projects, consult the following references.

NCHRP 560: http://onlinepubs.trb.org/onlinepubs/nchrp/nchrp_rpt_560.pdf

Special Experimental Project 14 (SEP 14): https://www.fhwa.dot.gov/programadmin/contracts/sep_a.cfm

The Road to Successful ITS Software Acquisition: https://www.fhwa.dot.gov/publications/research/operations/its/98036/rdsuccessvol2.pdf

For further guidance on preparation of verification plans, see:

Systems Engineering for Intelligent Transportation Systems, A Guide for Transportation Professionals https://ops.fhwa.dot.gov/publications/seitsguide/section4.htm#s4.7

Systems Engineering Guidebook for ITS https://www.fhwa.dot.gov/cadiv/segb/

Sole Source Procurement

In the event that only one system fulfills the requirements, and when Federal funds are used, the acquisition process will be governed by 23 CFR 635.411.² This regulation governs the acquisition of proprietary materials. In this case, certification from the FHWA is required that one of the following conditions is true:

• The proprietary product uniquely fulfills the requirements imposed on the product. Evaluation of the documents you are creating will provide the necessary justification.
• The proprietary product is required to synchronize with existing systems.

A Public Interest Finding (PIF) for proprietary purchases is not required in these cases. A PIF is required when the proprietary product is not the only product that fulfills the requirements. It is recommended that if more than one product fulfills all requirements, they should be considered competitively.

Proprietary acquisition is also possible for limited experimental application. This might apply to an adaptive system being installed as a pilot project to determine suitability for broader implementation. In this case, you should develop an experimental plan showing what you hope to learn from the experiment that you do not know before the experiment is conducted. Experimental application cannot be used to circumvent the requirement for system engineering.³ If you have used proprietary acquisition to install a small pilot installation of adaptive operation and you wish to now expand adaptive operation to become a much larger system, a PIF will not be issued simply because you already have a small proprietary system, if other systems may fulfill your requirements.

previous | next