Reviews, Simulations and Tests

The following learning objectives are covered in this lesson:

• Recognize the importance of modeling and simulation in the defense acquisition process.
• Recognize the contribution of STEP (Simulation, Test & Evaluation Process) to the development of a system.
• Distinguish among various types of DT&E (e.g., Production Qualification Tests, Production Acceptance Test and Evaluation).
• Recognize the relationship between risk management and exit criteria.
• Identify the information required for a milestone review.

1. One way to effectively manage acquisition risk is through the use of exit criteria, which serve as a litmus test as to whether the program is on track to achieve its goals. In order for exit criteria to be meaningful, they must be unique to not only the program itself, but to each phase of the program. Exit criteria are proposed by the Program Manager and approved by the Milestone Decision Authority (MDA).

Exit criteria can take many forms. However, the criteria should be measurable and reflect progress made in high risk areas of the program. Examples include the achievement of technical capabilities as seen in test results or the maturity of a manufacturing process. Thus, exit criteria are event-driven and considered at program reviews throughout the life of a program. They are critical "show-stoppers;" failure to meet an exit criteria could prevent a program from making further progress.

2. Milestone reviews are conducted by the MDA to initiate technology development, authorize program initiation and entry into the SDD phase, and to commit to production and deployment. Information for milestone reviews may be required by statute or regulation. The specific information required for each milestone review can be found in Enclosure 3 of DoDI 5000.2.

3. The use of modeling and simulation (M&S) can be very helpful during the acquisition process. Used as a predictor of future capabilities, M&S can be an inexpensive way to test various capabilities. However, M&S should not be used as a substitute for good test data. Models and simulations can also be modified and reused later in the acquisition process, which should avoid costs in the long run.

The Simulation, Test and Evaluation Process (STEP) is a DoD initiative that attempts to integrate modeling and simulation into the test and evaluation process by combining the two into one process. It is important to note that neither one is used as a replacement or substitute for the other, but are used together to complement each process.

Simply put, the STEP process, which can be used throughout the system life cycle, can be broken down into four steps: Model-Test-Fix-Model. The key to this process is to fix problems as they are discovered; not at the end of the process - and then begin the STEP process all over again in an effort to isolate new problems that might have arisen.

While M&S can be very effective, simulations only provide predictions of a system's performance and effectiveness. Thus, by combining M&S data with the empirical, measurable data provided by T&E, the two processes enhance each other and should result in long term efficiencies and cost savings.

4. Developmental Testing and Evaluation (DT&E) can take many forms during the acquisition process, depending upon what stage of the life cycle the program is in.

• Component tests take place on individual system parts before being merged into the system as a whole. Component testing is conducted both on hardware items and on software items before they are integrated with system hardware.
• Integration testing is used to assess compatibility of individual hardware and software components as they are aggregated to form subsystems or systems.
• Environmental testing, sometimes referred to as the "shake-rattle-roll" part of the testing process, attempts to define how different components react under various conditions, such as temperature and shock.
• Production Qualification Testing (PQT) is conducted on initial production articles to verify the effectiveness of the manufacturing process.
• Production and Acceptance Testing and Evaluation (PAT&E) is conducted on production items to verify that these items have met contract requirements.
• Live Fire T&E provides an assessment of system vulnerability and/or lethality.
• Modification testing can be used during production, or following system deployment, to determine the need for or benefits of any system changes.