Assessment Objects: Mechanisms (e.g., hardware, software, firmware)

Activities (e.g., system operations, administration, management; exercises)

Definition:  The process of exercising one or more assessment objects under specified conditions to compare actual with expected behavior, the results of which are used to support the determination of security and privacy control existence, functionality, correctness, completeness, and potential for improvement over time.44

Supplemental Guidance:  Typical assessor actions may include, for example: testing access control, identification and authentication, and audit mechanisms; testing security configuration settings; testing physical access control devices; conducting penetration testing of key information system components; testing information system backup operations; testing incident response capability; and exercising contingency planning capability.

SCAP-validated tools can be used to automate the collection of assessment objects and evaluate these objects against expected behavior. The use of SCAP is specifically relevant to the testing of mechanisms that involve assessment of actual machine state. The National Checklist Program catalogs a number of SCAP-enabled checklists that are suitable for assessing the configuration posture of specific operating systems and applications. SCAP-validated tools can use these checklists to determine the aggregate compliance of a system against all of the configuration settings in the checklist (e.g., CM-6) or specific configurations that are relevant to a security or privacy control that pertains to one or more configuration settings. SCAP-validated tools can also determine the absence of a patch or the presence of a vulnerable condition. The results produced by the SCAP tools can then be examined by assessors as part of the security and privacy control assessments.

Attributes:  Depth, Coverage

•The depth attribute addresses the types of testing to be conducted. There are three possible values for the depth attribute: (i) basic testing; (ii) focused testing; and (iii) comprehensive testing.

-Basic testing:  Test methodology (also known as black box testing) that assumes no knowledge of the internal structure and implementation detail of the assessment object. This type of testing is conducted using a functional specification for mechanisms and a high-level process description for activities. Basic testing provides a level of understanding of the security and privacy controls necessary for determining whether the controls are implemented and free of obvious errors.

-Focused testing:  Test methodology (also known as gray box testing) that assumes some knowledge of the internal structure and implementation detail of the assessment object. This type of testing is conducted using a functional specification and limited system architectural information (e.g., high-level design) for mechanisms and a high-level process description and high-level description of integration into the operational environment for activities. Focused testing provides a level of understanding of the security and privacy controls necessary for determining whether the controls are implemented and free of obvious errors and whether there are increased grounds for confidence that the controls are implemented correctly and operating as intended.

-Comprehensive testing:  Test methodology (also known as white box testing) that assumes explicit and substantial knowledge of the internal structure and implementation detail of the assessment object. This type of testing is conducted using a functional specification, extensive system architectural information (e.g., high-level design, low-level design) and implementation representation (e.g., source code, schematics) for mechanisms and a high-level process description and detailed description of integration into the operational environment for activities. Comprehensive testing provides a level of understanding of the security and privacy controls necessary for determining whether the controls are implemented and free of obvious errors and whether there are further increased grounds for confidence that the controls are implemented correctly and operating as intended on an ongoing and consistent basis, and that there is support for continuous improvement in the effectiveness of the controls.

•The coverage attribute addresses the scope or breadth of the testing process and includes the types of assessment objects to be tested, the number of objects to be tested (by type), and specific objects to be tested.45 There are three possible values for the coverage attribute: (i) basic; (ii) focused; and (iii) comprehensive.

-Basic testing:  Testing that uses a representative sample of assessment objects (by type and number within type) to provide a level of coverage necessary for determining whether the security and privacy controls are implemented and free of obvious errors.

-Focused testing:  Testing that uses a representative sample of assessment objects (by type and number within type) and other specific assessment objects deemed particularly important to achieving the assessment objective to provide a level of coverage necessary for determining whether the security and privacy controls are implemented and free of obvious errors and whether there are increased grounds for confidence that the controls are implemented correctly and operating as intended.

-Comprehensive testing:  Testing that uses a sufficiently large sample of assessment objects (by type and number within type) and other specific assessment objects deemed particularly important to achieving the assessment objective to provide a level of coverage necessary for determining whether the security and privacy controls are implemented and free of obvious errors and whether there are further increased grounds for confidence that the controls are implemented correctly and operating as intended on an ongoing and consistent basis, and that there is support for continuous improvement in the effectiveness of the controls.