System Test and Evaluation Plan and Procedure: The Department of Defense (DoD)
The objective of this study is to conduct a system test and evaluation plan and procedure for a the Department of Defense (DoD). Toward this end this study will review literature that informs this area of inquiry.
Concepts of Security Assurance and Trusted Systems
A system is “an integrated composite of people, products, and processes that provide a capability to satisfy a stated need or objective.” (Defense Acquisition University Press, 2011, p.3) is accomplished through integration of three primary activities:
(1) Development phasing that controls the design process and provides baselines that coordinate design efforts;
(2) A systems engineering process that provides a structure for solving design problems and tracking requirements flow through the design effort, and (3) Life cycle integration that involves customers in the design process and ensures that the system developed is viable throughout its life. (Defense Acquisition University Press, 2011, p.3)
The activities of system engineering are those of development phasing, systems engineering process, life cycle integration, baselines, systems engineering management, integrated teaming, and life cycle planning. All of these activities overlap one another as shown in the following illustration labeled Figure 1.
Figure 1 – Systems engineering activities
Source: Defense Acquisition University Press (2011)
The development phasing process involves specific stages including: (1) The concept level; (2) the system level; and (3) the subsystem component level. (Defense Acquisition University Press, 2011) Development phasing is illustrated as shown in Figure 2 in this study.
Figure 2 – Development Phasing
Source: Defense Acquisition University Press (2011)
Life cycle integration is reported to be achieved through “integrated development — that is, concurrent consideration of all life cycle needs during the development process.” (Defense Acquisition University Press, 2011) Life cycle functions are reported as the characteristic actions associated with the system life cycle.” (Defense Acquisition University Press, 2011) The primary functions in system engineering are: (1) development; (2) manufacturing production or construction; (3) deployment (fielding); (4) operation; (5) support; (6) disposal; (7) training; (8) verification. DoD is reported to have two fundamental requirements for program management; (1) the requirement that an integrated product and process approach be taken to design wherever practicable; and (2) requires that disciplined systems engineering process be used to translate operational needs and/or requirement into a system solution. (Defense Acquisition University Press, 2011)
II. Evaluation of Ways of Providing Assurance Throughout the Lifecycle
DoD reports that there is no “separate milestone decision required for a program to enter this phase of the system life cycle. The requirement for the Sustainment phase is implicit in the decision to produce and deploy the system.” (Defense Acquisition University Press, 2011) This phase is reported to overlap the production phase. in the Sustainment phase are focused on maintaining the system’s performance capability relative to the threat the system faces. If the military threat changes or a technology opportunity emerges, then the system may require modification.” (Defense Acquisition University Press, 2011) It is necessary that these modifications be “approved at an appropriate level for the particular change being considered. The changes then drives the initiation of new systems engineering processes, starting the cycle (or parts of it) all over again.” () The systems engineering process for DoD is shown in the following illustration.
Figure 3 — Systems Engineering Process
Source: Defense Acquisition University Press (2011)
III. Discussion of Validation and Verification
It is reported that component and breadboard validation in the laboratory environment: are integrated to establish that the pieces will work together. This is relatively ‘low fidelity’ compared to the eventual system. Examples in clued integration of ‘ad hoc’ hardware in a laboratory.” (Defense Acquisition University Press, 2011) Component and breadboard validation in relevant environment — Fidelity of breadboard technology increases significantly. The board validation in relevant basic technological components are integrated with reasonably environment. realistic supporting elements so that the technology can be tested in simulated environment. Examples include “high fidelity” laboratory integration of components. The systems engineering process involves the process input characterized by customer needs and objectives. The System Analysis and Control includes requirements analysis, functional analysis and allocation, and synthesis resulting in process output. Systems analysis and control is reported to be inclusive of “technical management activities required to measure progress, evaluate and select alternatives and document data and decisions.” (Defense Acquisition University Press, 2011) Stated as the purpose of systems analysis and control are the following:
(1) solution alternative decisions are made only following evaluation of the impact on system effectiveness, life cycle resources, risk and customer requirements;
(2) Technical decisions and specification requirements are based on systems engineering outputs,
(3) Traceability from systems engineering process inputs to outputs is maintained,
(4) Schedules for development and delivery are mutually supportive,
(5) Required technical disciplines are integrated into the systems engineering effort,’
(6) Impacts of customer requirements on resulting functional and performance requirements are examined for validity, consistency, desirability, and attainability, and,
(7) Product and process design requirements are directly traceable to the functional and performance requirements they were designed to fulfill, and vice versa.( Defense Acquisition University Press, 2011)
IV. Illustration of Evaluation Methodology and Certification Techniques
Ensuring the functionality and use of the product involves the following:
(1) Verification — This is the process of determining that a model implementation accurately represents the developer’s conceptual description and specifications that the model was designed to.
(2) Validation — This is the process of determining the manner and degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model, and of establishing the level of confidence that should be placed on this assessment.” (Defense Acquisition University Press, 2011)
(3) Accreditation — this is the formal certification “that a model or simulation is acceptable for use for a specific purpose. Accreditation is conferred by the organization best positioned to make the judgment that the model or simulation in question is acceptable. That organization may be an operational user, the program office, or a contractor, depending upon the purposes intended.” (Defense Acquisition University Press, 2011)
These functions are shown in the following illustration labeled Figure .
Figure 4 — Verification, Validation and Accreditation
Source: Defense Acquisition University Press (2011)
References
Systems Engineering Fundamentals (2011) SUPPLEMENTARY TEXT PREPARED BY THEDEFENSE ACQUISITION UNIVERSITY PRESS Secure Software Development: A Security Programmer’s Guide. For Belvoir, VA. Retrieved from: http://www.dau.mil/pubs/pdf/SEFGuide%2001-01.pdf
Grembi, J. (2008)
Computer Security: Art and Science. Bishop, Matt
