APB Breaches

The following learning objectives are covered in this Lesson:

• Identify when program deviations occur and the actions that should be taken by the acquisition manager.
• Relate the Acquisition Program Baseline (APB) to planning, control, and risk management in attaining cost, schedule and performance goals.

1. A program deviation occurs when the Program Manager has reason to believe that the current estimate for a given cost, schedule or performance parameter does not meet the threshold value specified for that parameter in the Acquisition Program Baseline. The PM must follow certain procedures whenever this occurs:

• The PM must immediately inform the Milestone Decision Authority (MDA) when a program deviation occurs.
• Within 30 days of the deviation, the PM must explain to the MDA the reason for the deviation and what steps need to be taken to bring the program back on track.
• Within 90 days of the deviation, one of the following scenarios must take place:
• The program is brought back on track; or
• A new APB is approved, changing only the parameters that were deviated; or
• An OIPT-level review is conducted to evaluate the PM's proposed baseline revisions, and feedback is given to the MDA, or in the case of a major program, to the Defense Acquisition Executive; or
• If it's not possible for at least one of these actions to take place within 90 days, then the MDA should hold a formal program review to determine the status of the program.

2. Cost, schedule, and performance parameters are interrelated, and a change in one parameter can affect the others. For example, the materials needed for a lighter aircraft may cost more and take longer to design and manufacture than materials in a heavier aircraft. In that case, performance would affect both cost and schedule parameters. Therefore it is important to involve all the key stakeholders when considering changes to the APB. 

Integrated Baseline Review

The following learning objectives are covered in this lesson:

• Identify the primary factors that the government should review to evaluate the contractor's PMB during an Integrated Baseline Review (IBR).
• Identify the three reasons for PMB changes, and recognize their impact.

1. The Cost Performance Index (CPI) and Schedule Performance Index (SPI) indicate the performance efficiency factors that the contractor has achieved to date. Anytime the CPI or SPI are running significantly below 1.0, rebaselining may be necessary in order to complete the program. Generally, a CPI or SPI falling 10% or more below 1.0 is considered significant. The To Complete Performance Index (TCPI) indicates the efficiency factor that the contractor must achieve from "time now" to meet the budget at completion (BAC) or estimate at completion (EAC).

A TCPI greater than 1.0 indicates the contractor must work more efficiently that they have in the past to stay within the BAC or meet the EAC. These performance indices may indicate the need to conduct an Integrated Baseline Review (IBR). The IBR assesses the validity of the program management baseline (PMB) and identifies the risks associated with executing to the current PMB. Participants in an IBR typically include the government PM and technical staff, along with the related contractor's staff. During an IBR, the primary factors that are evaluated include:

• The technical scope of the PMB
• Program schedule requirements
• Effective resource allocation to ensure that the work can be accomplished

2. There may be considerable risks associated with the current PMB indicating a need to rebaseline the program in order to make it executable. Changing the PMB can be caused by any one of the following three reasons:

• Contract changes : only applies to changes/contract modifications directed by the government, not the contractor.
• Internal re-planning : occurs when the contractor's original plan needs adjustment in response to problems or the opportunity to capitalize on efficiencies. The remaining work is then replanned by the contractor PM using the remaining budget and schedule.
• Formal re-programming : occurs when the remaining budget and schedule is unrealistic; the contractor requires more time and dollars; the PMB exceeds the contract target cost and an over target baseline (OTB) occurs and the budget is insufficient; and the original objectives cannot be met.

Software Problems

The following learning objectives are covered in this lesson:

• Apply a generic problem-solving model to an acquisition situation.
• Apply one or more selected qualitative tools (e.g., fishbone diagram) to resolve a problem.
• Identify developer practices essential for creation of high quality software.
• Identify the requirements for interoperability testing.

1.    One problem-solving technique is the cause and effect diagram or "fishbone" diagram.  By analyzing all the possible causes of a problem, the fishbone diagram focuses on determining the root cause of a problem, rather than on symptoms or solutions.  Typically, the fishbone diagram begins with a statement of the problem in a box on the right side of the diagram--the "head" of the fish.  Then categories of major causes are identified and drawn to the left--the "bones" of the fish.  These major causes are broken down into all the related causal factors that might contribute to the major causes.  Finally, the causal factors are examined and narrowed down to the most significant elements of the problem to determine the ultimate cause or causes.

2. The Software Program Managers Network has identified several software best practices based on interviews with software experts and industry leaders.  Here is a synthesized list of some of those characteristics, which are essential for the creation of high quality software: Adopt Continuous Program Risk Management

Risk management is a continuous process beginning with the definition of the concept and ending with system retirement. Risks need to be identified and managed across the life of the program.

Estimate Cost and Schedule Empirically

Initial software estimates and schedules should be looked on as high risk due to the lack of definitive information available at the time they are defined.

Use Metrics to Manage

All programs should have in place a continuous metrics program to monitor issues and determine the likelihood of risks occurring. Metrics information should be used as one of the primary inputs for program decisions.

Track Earned Value

Earned value requires each task to have both entry and exit criteria and a step to validate that these criteria have been met prior to the award of the credit. Earned value credit is binary with zero percent being given before task completion and 100 percent when completion is validated.

Track Defects against Quality Targets

All programs need to have pre-negotiated quality targets, which is an absolute requirement to be met prior to acceptance by the customer. Programs should implement practices to find defects early in the process and as close in time to creation of the defect as possible and should manage this defect rate against the quality targets. Meeting quality targets should be a subject at every major program review.

Treat People as the Most Important Resource

A primary program focus should be staffing positions with qualified personnel and retaining this staff through the life of the project. The program should not implement practices (e.g., excessive unpaid overtime) that will force voluntary staff turnover. The effectiveness and morale of the staff should be a factor in rewarding management.

Adopt Life Cycle Configuration Management

All programs, irrespective of size, need to manage information through a preplanned configuration management (CM) process. This discipline requires as a minimum:

• Control of shared information
• Control of changes
• Version control
• Identification of the status of controlled items(e.g., memos, schedules) and
• Reviews and audits of controlled items.

Manage and Trace Requirements

Before any design is initiated, requirements for that segment of the software need to be agreed to. Requirements need to be continuously traced from the user requirement to the lowest level software component.

Use System-Based Software Design

All methods used to define system architecture and software design should be documented in the system engineering management plan and software development plan and be frequently and regularly evaluated through audits conducted by an independent program organization.

Ensure Data and Database Interoperability

All data and database implementation decisions should consider interoperability issues and, as interoperability factors change, these decisions should be revisited.

Define and Control Interfaces

Before completion of system-level requirements, a complete inventory of all external interfaces needs to be completed. Internal interfaces should be defined as part of the design process. All interfaces should be agreed upon and individually tested.

Design Twice, Code Once

Traceability needs to be maintained through the design and verified as part of the inspection process. Design can be incrementally specified when an incremental release or evolution life cycle model is used provided the CM process is adequate to support control of incremental designs.

Assess Reuse Risks and Costs

The use of reuse components, COTS (Commercial Off-The-Shelf), GOTS (Government Off-The-Shelf) or any other non-developmental items (NDI) should be a primary goal, but treat any use as a risk and manage it through risk management.

Inspect Requirements and Design

All products that are placed under CM and are used as a basis for subsequent development need to be subjected to a formal inspection defined in the software development plan. The program needs to fund inspections and track rework savings.

Manage Testing as a Continuous Process

All testing should follow a preplanned process, which is agreed to and funded. Every test should be described in traceable procedures and have pass-fail criteria.

Compile and Smoke Test Frequently

Smoke testing should qualify new capability or component only after successful regression test completion. All smoke tests should be based on a traceable procedure and run by an independent organization (not the engineers who produced it). Smoke test results should be visible and provided to all project personnel.

3.    Interoperability problems can best be identified through the use of actual, live systems to mitigate risk.  Joint interoperability is defined as the ability of systems to provide services to and accept services from other systems and to use the services exchanged to enable them to operate effectively together.  The Joint Interoperability Test Command is responsible for verifying the interoperability of systems to the parameters outlined in the ICD, CDD, CPD and ISP. 

Reprogramming Funds

The following learning objectives are covered in this lesson:

• Select the appropriate public law (i.e., Misappropriation Act, Anti-deficiency Act, Bona Fide Need) that applies to the use of appropriated funds under specific circumstances.
• Given a funding shortfall, apply the rules governing reprogramming of appropriated funds in each appropriation category to resolve the problem.
• Identify the role of Operational Assessment (OA) in reducing program risk.
• Identify the risks and benefits associated with
•  
•  DT/OT.

1. Congress has passed laws to ensure the proper use of the funds they make available for defense acquisition programs:

• The Misappropriation Act states that funds appropriated by Congress can only be used for the programs and purposes for which the appropriation was made. Using Research, Development, Test and Evaluation (RDT&E) funds to pay for the procurement of items, for example, would violate the Misappropriation Act.
• The Anti-Deficiency Act prohibits the obligation of funds in excess of an appropriated amount or in advance of receiving an appropriation. In other words, you can't spend more funds than you have or before you have them. Incurring a contractual obligation without having the funds to cover it, for example, would violate the Anti-Deficiency Act.
• The Bona Fide Need Rule states that funds appropriated for a particular area can only be used during the period in which the appropriation is available for new obligations. If a research and development contract were awarded with FY03 RDT&E funds, and a new requirement arises in FY05 beyond the scope of that contract, then using FY03 RDT&E funds to pay for the new requirement would violate the Bona Fide Rule.

2. Although there are strict rules governing the use of appropriated funds, Congress recognizes that there are certain situations where some flexibility is needed. Reprogramming is the use of funds for purposes other than those intended by Congress at the time originally appropriated. Note that reprogramming only applies to funds that have already been appropriated by Congress.

Prior approval from Congress is required to move funds between appropriations, to increase the quantities of major systems procured, new starts, or for designated special interest items. However, most reprogramming actions in DoD are approved at the service or agency level, without the involvement of Congress, using below threshold reprogramming. Below threshold reprogramming allows the transfer of funds among programs within an appropriation category, subject to certain limitations. Up to $20 million of procurement funds can be transferred into a line item, and up to $10 million of RDT&E funds can be transferred into a program element, through below-threshold reprogramming.

3. An Early Operational Assessment (EOA) is typically conducted sometime before the Design Readiness Review held in the System Development and Demonstration phase (SDD). Using prototype systems, the EOA identifies potential operational effectiveness and suitability issues during system development. An Operational Assessment (OA) is conducted before Milestone C. Using engineering development models or pre-production systems, the OA provides operational effectiveness and suitability data before low rate initial production is begun.

4. Sometimes developmental and operational testing are combined to save resources, time and money. DT and OT are typically combined when the data, resources, objectives, test scenarios, and measures of effectiveness of both tests are similar and compatible. DoD policy encourages combined testing as long as the objectives of both types of testing are met. Combined testing eliminates redundant activities and raises operational concerns in time to make changes in the system design. However, combined tests require extensive coordination, are more difficult to design, and risk compromising test objectives.

Combining DT and OT does not remove the requirement to conduct initial operational test and evaluation (IOT&E), which is required by law for ACAT I and ACAT II programs. IOT&E uses production representative systems and typical user personnel in a scenario that is as realistic as possible. Successful IOT&E is required for the milestone decision authority to make the full-rate production decision. 

Design Changes

The following learning objectives are covered in this lesson:

• Identify how instability of user capability needs, design, and production processes impact program cost and schedule.
• Identify the purpose of specific technical reviews and their relationship to the acquisition process.
• Identify the roles, responsibilities, and methods for interface control and technical data management.
• Recognize how configuration management impacts all functional disciplines (e.g., test, logistics, manufacturing, etc.)
• Identify the impact on configuration management when commercial items are used in the system.
• Relate the different types of program unique specifications to their appropriate configuration baselines and technical review requirements.
• Trace the maturation of system design information as it evolves through the acquisition life cycle of a system.
• Identify the relationship between configuration baselines, specifications, and configuration management planning.
• Identify key acquisition best practices, including commercial practices that impact the relationship between government and industry.

1. Technical reviews are conducted throughout the acquisition life cycle to reduce program risk.  They are event-driven, not schedule-driven, and help determine whether to proceed with development or production.  Technical reviews are used to clarify design requirements, assess design maturity, and evaluate the system configuration at various points in the development process.  They provide a forum for communication across different disciplines in the system development process and establish common configuration baselines from which to proceed to the next level of design.

Types of technical reviews include:

• System Requirements Review (SRR), in which the system specification is evaluated to ensure that system requirements are consistent with the preferred concept and available technologies.
• Preliminary Design Review (PDR), in which the top-level design for each configuration item function and interface is evaluated to determine if it is ready for detailed design.
• Critical Design Review (CDR), in which the detailed Product Baseline is evaluated to determine if system design documentation is good enough to begin production (hardware) or final coding (software).
• Test Readiness Review (TRR), in which test objectives, procedures and resources are evaluated to determine if the system is ready to begin formal testing.

2. Configuration management is a systems analysis and control tool that is used in the systems engineering process to control the design of a product as it evolves from a top-level concept into a highly detailed design. Through configuration management, we ensure that designs are traceable to requirements, interfaces are well defined and understood, change is controlled and documented, and product documentation is consistent and current.

Configuration management involves development of program unique specifications and other technical data to document the design.  As design requirements are finalized at different levels of detail, configuration baselines are established to formally document those requirements and to define an item's functional and physical characteristics. The baselines progress from the overall system level (functional baseline), to the more specific configuration item level (allocated baseline), down to the detailed level (product baseline):
 

BASELINE	SPECIFICATIONS	UAV EXAMPLE
Functional ("system specification")	Overall system performance requirements, including interfaces	Night vision requirement
Allocated ("design to" specification)	Item performance specifications.  Performance characteristics of specific configuration items, including form, fit, function requirements.	Specific light level and resolutions that are required of a digital camera for the night vision capability.  Interface requirement for camera to attach to air vehicle.
Product ("build to" baseline)	Item detail specifications.  Process, procedure, material details, technical documentation	Camera shutter design details.  Video transport circuit detailed design. Drawing showing locking mechanism for camera body.

The Government must determine which baselines should come under Government control.  Generally speaking, the Government maintains control of the functional, or system-level baseline; either the Government or contractor can maintain the allocated baseline; and the contractor is usually responsible for the product, or 'build-to' level baseline and below.

3.Interface management involves the control and definition of the boundaries at which product subsystems come into contact with other components of the system.  Effective interface management involves identifying, developing and maintaining the external and internal interfaces necessary for system operation.  Interface management can become a configuration management challenge when a product is modified.

The contractor is usually responsible for design and control of internal interfaces, while the Government is responsible for external interfaces.  An Interface Control Working Group (ICWG) is often used to establish formal communication links between Government and contractor personnel involved in system interface design.

4. Once a system is fielded, configurationmanagement documentation becomes the basis for supporting the system, whether that support is provided by the contractor or by the Government.  Interoperability and maintenance issues can become very problematic if configuration management isn't done properly.  Even minor changes to a commercial item can create configuration challenges and impact logistics, testing, production and other functional areas.

The contractor will ultimately document the functional, performance, and physical characteristics of their product in a Technical Data Package (TDP).  Ensuring that the TDP is comprehensive and updated regularly is especially important if the Government is going to maintain or modify the system. 

Reviews, Simulations and Tests

The following learning objectives are covered in this lesson:

• Recognize the importance of modeling and simulation in the defense acquisition process.
• Recognize the contribution of STEP (Simulation, Test & Evaluation Process) to the development of a system.
• Distinguish among various types of DT&E (e.g., Production Qualification Tests, Production Acceptance Test and Evaluation).
• Recognize the relationship between risk management and exit criteria.
• Identify the information required for a milestone review.

1. One way to effectively manage acquisition risk is through the use of exit criteria, which serve as a litmus test as to whether the program is on track to achieve its goals. In order for exit criteria to be meaningful, they must be unique to not only the program itself, but to each phase of the program. Exit criteria are proposed by the Program Manager and approved by the Milestone Decision Authority (MDA).

Exit criteria can take many forms. However, the criteria should be measurable and reflect progress made in high risk areas of the program. Examples include the achievement of technical capabilities as seen in test results or the maturity of a manufacturing process. Thus, exit criteria are event-driven and considered at program reviews throughout the life of a program. They are critical "show-stoppers;" failure to meet an exit criteria could prevent a program from making further progress.

2. Milestone reviews are conducted by the MDA to initiate technology development, authorize program initiation and entry into the SDD phase, and to commit to production and deployment. Information for milestone reviews may be required by statute or regulation. The specific information required for each milestone review can be found in Enclosure 3 of DoDI 5000.2.

3. The use of modeling and simulation (M&S) can be very helpful during the acquisition process. Used as a predictor of future capabilities, M&S can be an inexpensive way to test various capabilities. However, M&S should not be used as a substitute for good test data. Models and simulations can also be modified and reused later in the acquisition process, which should avoid costs in the long run.

The Simulation, Test and Evaluation Process (STEP) is a DoD initiative that attempts to integrate modeling and simulation into the test and evaluation process by combining the two into one process. It is important to note that neither one is used as a replacement or substitute for the other, but are used together to complement each process.

Simply put, the STEP process, which can be used throughout the system life cycle, can be broken down into four steps: Model-Test-Fix-Model. The key to this process is to fix problems as they are discovered; not at the end of the process - and then begin the STEP process all over again in an effort to isolate new problems that might have arisen.

While M&S can be very effective, simulations only provide predictions of a system's performance and effectiveness. Thus, by combining M&S data with the empirical, measurable data provided by T&E, the two processes enhance each other and should result in long term efficiencies and cost savings.

4. Developmental Testing and Evaluation (DT&E) can take many forms during the acquisition process, depending upon what stage of the life cycle the program is in.

• Component tests take place on individual system parts before being merged into the system as a whole. Component testing is conducted both on hardware items and on software items before they are integrated with system hardware.
• Integration testing is used to assess compatibility of individual hardware and software components as they are aggregated to form subsystems or systems.
• Environmental testing, sometimes referred to as the "shake-rattle-roll" part of the testing process, attempts to define how different components react under various conditions, such as temperature and shock.
• Production Qualification Testing (PQT) is conducted on initial production articles to verify the effectiveness of the manufacturing process.
• Production and Acceptance Testing and Evaluation (PAT&E) is conducted on production items to verify that these items have met contract requirements.
• Live Fire T&E provides an assessment of system vulnerability and/or lethality.
• Modification testing can be used during production, or following system deployment, to determine the need for or benefits of any system changes.

Operational and Live Fire Tests

The following learning objectives are covered in this lesson:

• Identify which organizations develop, coordinate, or approve Critical Operational Issues (COIs).
• Identify which organizations develop, coordinate, or approve Critical Technical Parameters (CTPs).
• Recognize how Measures of Effectiveness (MOE) and Measures of Suitability (MOS) are used throughout the T&E process.
• Recognize the purpose and objectives of Live Fire Test and Evaluation.
• Distinguish among various types of DT&E (e.g., Production Qualification Tests, Production Acceptance Test and Evaluation).

1. Developmental test and evaluation is essential in determining a system's readiness for initial operational test and evaluation (IOT&E). The results of developmental testing are formally reviewed in an Operational Test Readiness Review (OTRR) prior to proceeding with IOT&E.

• Critical Technical Parameters (CTPs) are key parameters and developmental testing criteria that are derived from the CDD, and from technical performance measures as specified by the System Engineering Plan. The CTPs are developed, coordinated and approved by the T&E IPT within the Program Management Office. Examples of CTPs are an aircraft's cruising speed, range and altitude.

2. Two types of developmental testing become important as a system nears and enters production:

• Production Qualification Testing (PQT) is conducted on a small number of initial production items to evaluate the effectiveness of the manufacturing process.
• Production Acceptance Testing and Evaluation (PAT&E) is conducted on items as a form of quality assurance to ensure that contractual obligations are being met.

3. Operational test and evaluation is conducted to determine if a system will successfully meet the user's capability needs.

• Critical Operational Issues (COIs) indicate the operational effectiveness and operational suitability needs of a system. They are expressed in the form of a question, developed by an independent operational test agency, and broken down into quantifiable MOEs and MOSs. An example of a COI is: "Does the aircraft accomplish its mission in the battlefield environment?"
• Measures of Effectiveness (MOEs) are specific, objective measures of system performance that are closely related to mission accomplishment. An example of a MOE is: "Number of targets destroyed "
• Measures of Suitability (MOSs) are specific, objective measures of how well as system can be maintained and utilized by the end user. They are written and approved by an independent operational test agency. An example of a MOS is: "Aircraft Mean Time Between Failure (MTBF)."

4. In summary, COIs are the primary operational issues that must be answered by the testing program, while MOEs and MOSs may be thought of as the quantifiable measures that can be used to determine whether the COIs have been addressed successfully. In turn, CTPs provide developmental test data that help support the MOEs and MOSs.

5. Under Public law (Title 10, US Code 2366), Live Fire Test and Evaluation is required for certain major systems before full rate production can begin:

• Survivability testing is required for "covered" systems that are occupied by personnel and designed to provide the personnel some degree of protection in combat situations.
• Lethality testing is required for all major munitions and missile programs to determine whether the weapon can reliably disable or destroy its target.

Live Fire Test and Evaluation results are sent to the Director, Operational Test and Evaluation (DOT&E), acting as the OSD agent, who then reports them to Congress before a program can move forward beyond LRIP and on to full rate production. 

Using the Software Acquisition Capability Maturity Model (SA-CMM), an organization whose acquisition processes are well documented and standardized would be rated at least:

Answer:

• Level 3

Link:

• ELO: 26a. Identify key laws and software acquisition management policies and practices that are required for the acquisition of a DoD software-intensive system.

A _______ is an aggregation of program elements that reflects a macro-level force mission or support mission of DOD.

Answer:

• Major Program (MP) 

Wrong Answers:

• Budget 
• Program Objectives Memorandum (POM) 
• Future Years Defense Program (FYDP) 

Link

• ELO: 11a. Relate the following building blocks to the PPBE process: - Future Years Defense Program (FYDP) - Major Program (MP) - Program Element (PE).

If the Government has funded the entire development of an item, the Government is entitled to _______ rights.

Answer:

• Unlimited 

Wrong Answers:

• Government purpose 
• Limited 
• Restricted 

Link:

• ELO: 14d. Identify the policy and concepts involved in the acquisition of data rights

Which one of the following is a method of designing for producibility?

Answer:

• Minimize assembly requirements

Wrong Answers:

• Maximize the number of parts used
• Limit the functions performed by each part
• Use a wide variety of component parts 

Modeling and simulation, when properly planned for and implemented, has been shown to: (Select all that apply)

Answers:

• Shorten program schedules 
• Improve system quality 
• Help make better decisions 

Wrong Answer:

• Increase technical risk 

The capability needs for a new software-intensive telecommunications system are well understood, but funding is not available to meet all the capabilities at once. The user is willing to field a basic system initially and add the rest of the capabilities over a period of three years. Which development model is best suited for this project?

Answer:

• Incremental model 

Wrong Answer:

• Life cycle model 
• Spiral model 
• Waterfall model 

Productivity, rework, and technology impact are examples of which kind of software metric?

Answer:

• Process 

Wrong Answers:

• Quality
• Management
• Resource allocation 

If a computer monitor was initially designed for use in a home or office environment, and the Navy decided to modify it for use on board ship, an early operational assessment (EOA) of that monitor would likely be conducted.

Answer:

• True

The Army receives proposals from eight different companies in response to an RFP. Before the most highly rated proposals can be identified, the Contracting Officer needs to resolve some ambiguities in the proposals. What type of information exchange would the Contracting Officer use to get the information needed to establish the competitive range of proposals?

Answer:

• Communications 

Wrong Answers:

• Clarifications 
• Solicitations 
• Negotiations 

Every commercial item must undergo the same governmental testing as a newly developed item before it is acquired for use by the Department of Defense

Answer:

• False 

Technical Performance Measures should be selected for those parameters that:

Answer:

• Are expected to have the highest degree of risk

Wrong Answers:

• Will not be addressed at technical design reviews
• Are expected to exceed the objectives contained in the Capability Development Document (CDD)
• Will not be subjected to formal test and evaluation. 

The Navy awards a cost plus award fee (CPAF) contract for $265 million to design and develop electronic components for a communications system within 18 months. What reports, if any, would the contractor normally need to provide to the Program Management Office regarding their cost, schedule, and technical performance?

Answer:

• The contractor would submit a Contract Performance Report (CPR) formats 1 through 5 and an Integrated Master Schedule (IMS).

(Updated Answer by unknown comment):

• The Navy PM would require submission of an Integrated Program Management Report (IPMR) with formats 1 through 7.

Wrong Answers:

• The contractor would submit either a Contract Performance Report (CPR) formats 1 through 5 or an Integrated Master Schedule (IMS) but not both. 
• The contractor would submit a Contract Performance Report (CPR) format 1 only. 
• The contractor would submit a Contract Performance Report (CPR) formats 1 through 5 only. 
• The contractor would not be required to submit any earned value related reports to the Program Management Office.

Link

• ELO: 21a. Compare the purpose and content of the Cost Performance Report (CPR) formats

What developmental test would you conduct to see how an item would perform under tropical conditions of heat and moisture?

Answer:

• Environmental Testing 

Wrong Answers:

• Component Testing 
• Live Fire T&E 
• Modification T&E 

Link:

• ELO: 30a. Distinguish among various types of DT&E (e.g., Production Qualification Tests, Production Acceptance Test and Evaluation).

Critical Technical Parameters (CTPs) are developed and coordinated by:

Answer:

• The Program Management Office that is developing the system. 

Wrong Answers:

• The independent Operational Test Agency that is testing the system. 
• The contractor that is developing and producing the system. 
• The Director, Operational Test and Evaluation (DOT&E) who is providing oversight for testing of the system. 

Link:

• ELO: 32b. Identify which organizations develop, coordinate, or approve Critical Technical Parameters (CTPs).

The Integrated Baseline Review (IBR) assesses the validity of the performance measurement baseline (PMB) and the Integrated Master Schedule (IMS). Which of the following is NOT an IBR outcome goal?

Answer:

• Confirming that the contractor is compliant with ISO 9000 

Wrong Answers:

• Determining that the PMB and IMS are realistic and achievable
• Determining that all contract work scope requirements are baselined
• Ensuring that work scope schedules are consistent with contract delivery requirements 

Link:

• ELO: 20c. Identify the primary factors that the government should review to evaluate the contractor's PMB during an Integrated Baseline Review (IBR)

Which of the following is a valid reason for re-baselining the performance measurement baseline (PMB)?

Answer:

• Actual Reasons for PMB changes and impact:
• Contract Change: The Government procuring contracting officer (PCO), not the contractor, directs these changes or contract modifications.
• Internal Replanning: Existing plan needs adjustment due to problems or when opportunities to capitalize on efficiencies are discovered, Contractor's PM must make adjustments to the existing plan that do not affect the overall budget or schedule. (Within the scope, schedule and cost objectives or the overall program)
• Rebaselining: Formal reprogramming that occurs when the remaining budget becomes so unrealistic that more money and time are needed.

Wrong Answers:

• The cost and schedule earned value variance metrics exceed the OSD thresholds.
• The Government agrees to a minor administrative modification to the contract.

Possible Answers:

• The plan for completing the remaining contract effort is no longer achievable. 
• The contractor projects that the Budget at Completion will exceed the Estimate at Completion. 
• The Government agrees to a minor administrative modification to the contract. 

Link:

• ELO: 20d. Associate rebaselining, replanning and reprogramming as they apply to PMB changes.

Within _____ days after a program deviation occurs, the Program Manager must inform the Milestone Decision Authority of the action(s) necessary to get the program back on track.

Answer:

• 30

Link:

• ELO: 4b. Identify when program deviations occur and the actions that should be taken by the acquisition manager.]   

Budget Execution Summary

The following learning objectives are covered in this lesson:

• Given a scenario, track budget execution through the commitment, obligation, and expenditure process.
• Identify the use and importance of obligation and expenditure plans.
• Assess the impact of the failure to execute funds in accordance with program plans.
•  

1. In the budget execution process, the following steps are taken:

• Commitment - an administrative reservation of funds, made upon receipt of a request for spending. Commitment occurs upon certification that funds are available in the correct appropriation, in the correct fiscal year, and in the correct amount to cover the anticipated obligation.
• Obligation : a "legal reservation" of funds, tying the government to a liability, such as a contract for goods or services. Obligation occurs when a contract is signed or when orders are placed.
• Expenditure - a payment of some part or all of an obligation. Expenditure occurs when a check is issued, or when funds are electronically transferred, to a contractor in response to an invoice or bill for costs incurred, services rendered, or products delivered.
• Outlay - a payment by the U.S. Treasury to the contractor. Outlay occurs when a check is cashed or when funds are electronically transferred from the Government to the contractor. (In electronic funds transfer, expenditure and outlay happen simultaneously.)

2. A number of players are involved in the execution of funds. After the Comptroller commits the funds by certifying their availability, the Contracting Officer obligates the funds by awarding the contract or signing purchase orders. Then the contractor performs the work and submits a Material Inspection and Receiving Report to the Quality Assurance Representative from the Contract Management Office, if deliverables are received at the contractor's plant, or to the Contracting Officer's Representative (COR), if deliverables are received at the program management office. The Quality Assurance Rep or COR verify that the deliverables were received and accepted and inform the Administrative Contracting Officer (ACO). The contractor submits an invoice to the ACO.

The ACO certifies that the invoice is correct, then forwards the invoice to the finance office to make payment. The ACO also assures that the contractor gets paid in a timely manner. The Finance and Accounting Office in turn expends the funds by check or electronic funds transfer. Finally, the U.S. Treasury outlays the funds when the cash is provided to the contractor.

3. Failure to make timely payment to a contractor can cause serious cash flow problems for the contractor. In addition, poor expenditure or outlay rates are a bad reflection on a program and may jeopardize a program's current and future funding. To minimize this risk, the Program Management Office prepares a spending plan that projects and tracks obligations and expenditures on a month-by-month basis.

A spending plan is required for each Procurement line item, RDT&E program element, and Operations and Maintenance sub-activity group in the program. The PMO creates an obligation plan for each fiscal year of funding that is available for new obligations and an expenditure plan for each fiscal year of funding that has not been completely expended, even if the period of obligation availability has expired.

Spending plans serve as a tool to analyze program execution, an indicator of potential problems, and a predictor of future program performance. Generally, a history of poor obligation, expenditure, or outlay will cause a program to come under increased scrutiny, or worse, lose funding. When a program deviates from its spending plan, it risks becoming a source of funding for other programs through reprogramming and runs the risk of having its funding cut in future years. 

Contractor Performance Measurement Summary

The following learning objectives are covered in this lesson:

• Given performance data, select and compute appropriate performance status indicators.
• Given performance data, detect and analyze the impact of significant problem areas, based on the status indicators.
• Given performance data, calculate an estimate of cost at completion.
• Recognize the importance of Earned Value data in external reporting.

1. There are various performance status indicators used in earned value management to tell whether a program is on track or not.

Budgeted Cost of Work Scheduled (BCWS)

indicates the value of work planned to be accomplished or planned value.

Budgeted Cost of Work Performed (BCWP)

indicates the value of work accomplished or the earned value.

Actual Cost of Work Performed (ACWP)

indicates the cost of work accomplished or actual cost.

Schedule Variance (SV)

equals the difference between the work accomplished the value of work accomplished to the value of work planned to be accomplished. It is calculated by subtracting the budgeted cost of work scheduled from the budgeted cost of work performed or earned value:

SV = BCWP : BCWS

A negative schedule variance in dollars is unfavorable and indicates that less work was accomplished than planned, while a positive schedule variance shows that more work was accomplished than planned. The program's critical path schedule must be reviewed to determine the impact of these schedule variances to the program.

Cost Variance (CV)

indicates whether the work accomplished cost more or less than planned. It is calculated by subtracting the actual cost of work performed from the budgeted cost of work performed:

CV = BCWP : ACWP

A negative cost variance in dollars is unfavorable and indicates that more money was spent for the work accomplished than was planned. This has the potential to put the program over budget if the trend continues, and may require the government to provide additional money to complete the program. A positive cost variance is favorable and indicates that the work accomplished cost less than planned.

2. We can also identify performance trends to see whether performance is improving or worsening over time and at what rate. This can be done for the overall program or for a specific activity within the program.

Schedule Performance Index (SPI)

indicates the efficiency with which the work has been accomplished in comparison to a specific value of work planned. For example, we may be functioning at only 0.8 or 80% efficiency of what we had planned to accomplish. It is calculated by dividing the budgeted cost of work performed by the budgeted cost of work scheduled:

SPI = BCWP/BCWS

Cost Performance Index (CPI)

tells the cost efficiency. It compares the value of work that has been accomplished to the actual cost of the accomplished work. For example, if our CPI is 0.75, we are accomplishing only 75 cents worth of work for every dollar we spend. It is calculated by dividing the budgeted cost of work performed by the actual cost of work performed:

CPI = BCWP/ACWP

Ideal CPI for a project is 1.0. Any activity with a CPI of less than 1.0 will rarely be improved over time. In fact, a program's CPI performance of less than 1.0 is often non-recoverable.

Cumulative CPIs and SPIs are usually less than 1.0 for most programs. Current period SPIs and CPIs for individual tasks can exceed 1.0, and exhibit positive and negative elements. When cumulative performance (CPI and SPI) fall below 1.0, the government needs to discuss the performance status with the contractor as part of risk management. Earned Value industry guidelines specifically state that management reserve will NOT be used to offset negative variances.

3. Budget at Completion (BAC) is the sum of all authorized budgets for the contract scope of work. The project's scope of work forms the performance measurement baseline (PMB), which projects the cost to do the entire program. The BAC equals the sum of all the allocated budgets plus any undistributed budget (management reserve and profit/fee not included).We use the BAC to determine the percent of the program spent and completed.

Percent Spent

: indicates how much of the program funds have been spent to date relative to the total amount of the project's budgeted funds. It is calculated by dividing the actual cost of work performed by the total amount expected to be spent on the program, the budget at completion:

% Spent = ACWP/BAC

Percent Complete --

indicates how much of the total program has been completed to date relative to the total amount of work to be performed. It is calculated by dividing the dollar amount of work performed to date, the budgeted cost of work performed, by the total amount expected to be spent on the program, the budget at completion:

% Complete = BCWP/BAC

Percent Scheduled :

indicates where the program should be based on a point in time. It is calculated by dividing the dollar amount of work scheduled to date, the budgeted cost of work scheduled, by the total about expected to be spent on the program, the budget at completion:

% Scheduled = BCWS/BAC

If the Percent Spent is greater than the Percent Complete, the program is going to run out of funds before the end of the project if it continues on the current trend. Conversely, if the Percent Complete is greater than or equal to the Percent Spent, the project has sufficient funds if it continues on the current trend. For example, if the percent complete is 50% and percent spent is 66%, we know we have a problem because we are spending at a faster rate than the project's work is being completed.

Note: don't confuse Percent Spent and Percent Complete with the SPI and CPI. Percent Complete and Percent Spent indicate program status, looking at the entire program from beginning to end. SPI and CPI indicate efficiency trends and look at a program up to a certain point in time.

4. Estimate at Completion (EAC) is the 'current' estimate of what the program will cost when completed. The EAC is based on the actual cost of work performed to date plus an estimate of the work remaining. It is calculated by adding the actual cost of work performed to the estimated cost to complete the remaining work of the program.

EAC = ACWP + Estimated Cost to Complete

The EAC can be calculated as follows: EAC is equal to the ACWP plus the BAC minus the BCWP divided by a performance factor such as the product of the CPI and SPI.

EAC = ACWP + [(BAC : BCWP)/(CPI x SPI)]

Both the Government and the contractor calculate EACs. The contractor's EAC is often referred to as the Latest Revised Estimate (LRE).

5. To Complete Performance Index for the Budget at Completion {TCPI(BAC)} shows what efficiency is required to accomplish the remaining work within the contract budget. This index attempts to quantify the efficiency required to deliver the project within the authorized funds for the remainder of the project.

The TCPI is correlated with the cumulative CPI; it takes the cost efficiency experienced to date, as reflected by the cumulative CPI, and determines what level of performance efficiency will be required to complete the project within available budget. If the cumulative CPI is 0.8 or 80%, in order to stay within our budget, we must achieve a performance factor of 1.2, or work at an efficiency of 120% for all the remaining work in order to complete the project at the BAC. This means the contractor must work 40% more efficiently than its current cumulative CPI of 80%.

To calculate TCPI (BAC) we divide the value of the work remaining by the value of the budget remaining. In other words, we subtract the BCWP from the BAC then divide that difference by the difference between the BAC and the ACWP.

TCPI (BAC) = BAC - BCWP
                         BAC - ACWP

The TCPI can also be based on an EAC which is most often used to validate the contractor or government EAC. DoD analysts have determined that after 20% into a program, the cumulative CPI rarely improves. Therefore, achieving a TCPI that is greater than 5% of the CPI is unlikely; this means we may have to restructure the program in order to obtain an executable program.