A Systems Approach to Limited Notice Performance Testing

The U.S. Department of Energy (DOE) has conducted comprehensive multi-topic safeguards and security assessments of its plants, laboratories, and sites since the mid-1980s. These assessments cover a broad spectrum of the security program by assessing the various topical areas with the goal of providing stakeholders a system-level evaluation of a program’s health/effectiveness. Since inception, multi-topic assessments have proven to be a successful, ongoing means to identify areas where program improvements may be needed. However, because of the notice given, and subsequent efforts that plant, laboratory, and site personnel put into preparing for a DOE visit, assessments are typically conducted when organizations are “at their best”. While the assessments remain beneficial, it was realized after a well-publicized security incident at a high security DOE site in July 2012, that assessing organizations at their best may not provide stakeholders a full depiction of response readiness. Thus, at the direction of the Secretary of Energy, the Office of Security Assessments (EA-22) teamed with various DOE stakeholders and determined that a limited-notice testing program was the most safe and viable option to address limitations in the multi-topic assessment program. This approach entails leveraging existing performance-testing programs facilitate planning and execution of performance tests under real time conditions, with minimal advanced notice to tested personnel, thereby providing stakeholders an assessment of sites’ preparedness to respond to various threats under day-to-day conditions. Among the most significant observations from the limited notice program was the benefit of assessments aimed at testing entire security systems, rather than individual components. The primary purpose of this paper is to discuss the benefits of using this “systems approach” to examine the dynamics of the interactions between the processes and assessing the performance of the system when there may be no inherent weaknesses in individual system elements. The secondary purpose is to communicate the lessons learned through test development and analysis to organizations that may be considering adopting a similar testing approach.