September 8, 1995 The Honorable Thomas P. Grumbly Assistant Secretary for Environmental Management Department of Energy Washington, D.C. 20585 Dear Mr. Grumbly: The Defense Nuclear Facilities Safety Board's (Board) staff recently completed an in-depth review on the status and use of technical procedures at the Department of Energy s Hanford Site. The Board visited the Hanford Site on August 22 - 23, 1995, and took that opportunity to discuss the status of technical procedures at the site with both the Richland Operations Office (DOE-RL) and the Westinghouse Hanford Company (WHC) management. The enclosed staff report was also discussed by the Board, DOE-RL and WHC. It is provided for your use in follow-on actions as appropriate. Although both the DOE-RL and WHC proposed a plan to achieve improvements, the Board notes that despite more than eighteen months' effort, technical procedure deficiencies remain a severe problem endemic to the Hanford Site. If you have any questions on this matter, please let me know. Sincerely, John T. Conway Chairman c: The Honorable Tara J. O'Toole Mr. Mark B. Whitaker Mr. John D. Wagoner Enclosure DEFENSE NUCLEAR FACILITIES SAFETY BOARD July 17, 1995 MEMORANDUM FOR: G. W. Cunningham, Technical Director COPIES: Board Members FROM: Timothy J. Dwyer SUBJECT: Review of Procedures at the Hanford Site 1. Purpose: This memorandum discusses the status of technical procedures across the Hanford Site. A continuing review of Hanford facility procedure programs is being conducted by Defense Nuclear Facilities Safety Board (Board) staff members Timothy J. Dwyer and Paul F. Gubanc, and (at various times) outside experts David S. Boyd, Edward O. Dietrich and, prior to July 1994, John H. Straub. The subject review was initiated at a meeting with Westinghouse Hanford Company (WHC) on January 31, 1994. Subsequent portions of the review have been conducted on February 25, 1994, March 29, 1994, April 7-8, 1994, May 5, 1994, May 27, 1994, August 1-3, 1994, August 15-18, 1994, December 15, 1994, and March 20-23, 1995. On June 22, 1995, WHC issued a compilation of reports based on their March-April 1995 peer review of facility procedures across the Hanford Site. 2. Summary: Richland Operations Office (DOE-RL) involvement in correcting the known Hanford Site procedure problem remains minimal, despite their own program indicating the problem persists. Senior WHC management has taken a major step toward recognizing the problem by commissioning an extensive Procedure Assessment Review (PAR), but the potential benefits of the review have been diminished by: inconsistent review team performance; underreporting of findings; apathy of mid-level management toward the review; and failure at all levels of management to recognize the operational importance of technically adequate procedures. As a result, Hanford Site procedures remain, for the most part, technically deficient and, in some cases, inadequate. Further, Board staff observations revealed that failure to use procedures and violations of approved procedures remain problems endemic to the Hanford Site. 3. Background: Technical procedures at the Hanford Site have been a recognized conduct of operations deficiency for several years. Both the Board's staff and DOE Headquarters (for example, DOE/DP-0126, Operating Experience Review - Conduct of Operations at Department of Energy Facilities, August 1994), and DOE-RL's own Conduct of Operations Reviews have identified significant facets of this deficiency to both WHC and the DOE-RL on numerous occasions. The magnitude of this deficiency was brought to light in a series of incidents culminating in the "rock-on-a-rope" occurrence at the Hanford Tank Farms in the summer of 1993. That occurrence brought about an administrative hold for the Tank Farms themselves and prompted the first major procedure review and upgrade efforts across the site. In September 1994, WHC briefed the Board on their "Technical Procedure Review" process and identified target milestones for improving procedures at Hanford. One of the final milestones involved a follow-up peer review (the PAR) to be conducted across the Hanford Site in March-April 1995. The Board's staff and outside expert's observations of the PAR effort, coupled with over 18 months of detailed review by the Board's staff, form the basis for the conclusions presented in this report. 4. Discussion: The culmination of the WHC Procedure Upgrade Program was the PAR, initially scheduled to occur from March 20-29, 1995. This review, which both DOE-RL and the Board's staff were explicitly invited to observe, was intended to "determine the useability and adequacy of each facility's technical procedures by field verifying the technical procedures [emphasis added] which are used in each facility . . . ." The importance of this event cannot be overstated. Until this review, WHC management personnel (and occasionally DOE-RL personnel) had reviewed and assessed various facility's procedure development processes, procedure verification and validation processes, procedure approval processes, procedure change processes, and procedure upgrade processes, but almost no one [Board staff excepted] had focused on the procedures that were the end product of these processes. a. DOE-RL Management. DOE-RL management rarely participated in this PAR, although some DOE-RL working level engineers and Facility Representatives attended selected kickoff or summary meetings. DOE-RL representatives have previously stated that DOE-RL expectations concerning procedure development, control, and employment have never been agreed upon within DOE-RL nor formally communicated to the contractor. Per these representatives, DOE-RL line management has "always listed procedure upgrades as necessary." However, no concerted DOE-RL effort to remedy this situation has materialized. Facility-level procedure reviews are only undertaken by DOE-RL personnel as part of the DOE-RL biennial Conduct of Operations reviews. In practice, the procedures portion of these DOE-RL reviews just involves: 1) determining the activities and effectiveness of the (WHC) facility procedures champion; 2) determining the procedure-related activities of the (DOE-RL) facility representative; and 3) watching a couple of procedures being used. DOE-RL personnel also stated that DOE-RL had reviewed each facility's WHC-CM-3- 5 Section 12.5 Technical Procedure Standard Compliance Action Plan. These Compliance Action Plans were developed to correct deficiencies discovered via WHC facility-led self-assessments. No documentation resulting from the DOE-RL reviews has been provided. The Board's staff was unable to determine the purpose, effectiveness, or actions resulting from this DOE-RL activity. b. WHC Management. WHC management participation in this PAR varied greatly. The fact that the review took place at all was a major step forward in senior management commitment to procedure improvement. The PAR manager was perceived by other WHC managers as having the backing of the WHC Executive Vice President, through the Executive Vice President's position as Chair of the WHC Operations Excellence Council. Nevertheless, the effectiveness of the PAR, and the procedure upgrade program as a whole, has suffered from indifference on the part of some WHC managers, from Vice Presidents to Facility Managers to Facility Procedure Managers. For example, several facility management teams made no preparations for the scheduled PAR team visits to their facilities. Efforts by the review manager to staff his review teams were hindered by the fact that several managers/supervisors refused to release selected participants from their duties to participate in the review program and necessitated the selection of alternate team members. Several facility managers declined efforts to schedule closeout (exit) meetings between the review teams and facility management. It is evident that, despite the effort to date, WHC Management still does not understand or, in some cases, even recognize the problem. Two examples are instructive: PUREX and Plutonium Finishing Plant (PFP) facility management determined, based on PAR activities, that it would be necessary to "revalidate their procedures." This will be at least the third time in recent history that facility management attempts to fix the same problem by performing the same ineffective corrective action. In a presentation to the Board on June 7, 1995, WHC Management briefed the completion of the PAR, noting that some minor problems were identified, but that "deficiencies were being addressed." However, in the presentation immediately preceding this briefing, the WHC Characterization Project Operations Manager had informed the Board that all Hanford Tank Farms sampling activities had been suspended, in part, as a result of procedural inadequacies. c. Summary of Previous Board Staff Technical Procedure Reviews. Eight major Hanford Site facilities were visited at least once over the course of 1994. None of these major facilities had satisfactory procedures programs. Although, by the end of the year, a few facilities (i.e., East Tank Farms and 242-A Evaporator, with regard to operating procedures only) had taken steps in the right direction. Sitewide, facility management still failed to recognize the scope of the problem. In particular, several continuing deficiencies were found to be common among all of the facilities reviewed: a lack of engineering leadership and management involvement in developing procedures, leading to a lack of sound technical basis documentation concerning the development of procedures, leading to technical procedures that cannot be performed as written, fostering an operator and field supervisor culture that does not include verbatim procedure compliance. Further details are outlined in Attachment 1. d. WHC Procedure Assessment Review Process. Four PAR teams were constituted and assigned to participate in an aggressive multi-facility program. Each team was to consist of three members, a mix of WHC personnel with technical and auditor backgrounds. Additionally, one person of each three-man team had to have experience in operations, and one had to have experience in maintenance. Preparations for the PAR program included the following: An overall Technical Procedure Assessment Plan was developed and promulgated; A Technical Procedure Review Checklist (TPRC) was developed for team use; Members of each team attended a three-hour training session designed to familiarize them with the TPRC requirements and management expectations for review conduct; and An internal WHC memorandum (dated February 2, 1995) informed each affected facility manager of the schedule, scope, and objective of the assessment program. The memorandum stated that the assessment program included " . . . all technical procedures. Technical procedures include operations, maintenance, surveillance, test, fire protection, health protection, and other types of procedures . . . . A minimum of three procedures of each category will be walked down by the review team." Observations of PAR team performance in the field revealed significant shortcomings with the process. Pressure due to the aggressive schedule forced several team leaders to abandon efforts to observe/walkdown the "minimum of three procedures of each category" early on. Further, it was apparent that the TPRC was not being used as a guide by the review teams or completed during the reviews; some team leaders seemed more interested in the facility procedure processes and facility Procedure Upgrade Plans than in an evaluation of the use and usefulness of the available procedures. In addition, selection screening for team member personnel was inadequate. In one case, a late change of review team personnel was required when it was discovered that none of the team members were qualified to enter a radiologically-controlled area to observe/walkdown procedures. In another case, review team members were precluded from observing/walking down procedures in the field due to lack of appropriate security clearances. Although this was eventually resolved by assigning facility personnel as escorts, at one point it fell to the Board's Hanford Site Representative to officially escort the PAR Team Based on the Board's staff questioning of the PAR process, WHC management redirected team leader efforts. Instructions passed on included: 1) use the TPRC; 2) drive the review to complete verification and validation of at least three procedures of each type listed in the TPRC, ignoring schedule constraints; and 3) emphasize field observation or walkdowns of the procedures. Further, WHC management recognized that the Health Physics procedures in use across the site were not under the control of the individual facilities. A separate review team was therefore created to examine these procedures, although only minimal effort was made to coordinate this review and none of the team members were excused from their primary duties in order to conduct the review. Performance of the review teams themselves was highly variable. Several of the teams included members did not develop any findings during the period they were being observed conducting the review--despite the fact that numerous discrepancies were identified by the Board staff members observing them. The pressures of the aggressive schedule precluded team leaders from adequately observing and critiquing the activities of their team members. As a result, no feedback was available to compensate for team member weaknesses [except as provided by the Board staff]. Further, the team leaders themselves varied from highly organized and effective to ill-prepared and ineffective leaders or reviewers. Based on the Board's staff observation of and comments on all four initially constituted review teams in action, WHC terminated reviews by three of the teams upon completion of their initial assignments and tasked the strongest team to conduct the remainder of the PAR effort. Review team findings, as presented to the Operations Excellence Council, were soft- pedaled. Further, the review team's candor in documenting procedural problems is being adversely affected by the knowledge that their funding for the [continuing] effort will come from the facilities being reviewed. e. WHC Procedure Assessment Review Results. The compilation of reports issued by WHC on June 22, 1995 offers the following summary: Each facility has a formally defined procedure review and approval process, although compliance with WHC standard WHC-CM-3-5, Section 12.5 varies. [In some cases, Board staff found that facilities were unaware of the existence of this standard.] Many of the procedures reviewed were technically inadequate in areas such as human factors and inconsistent with the WHC standard. However, no procedures were judged to result in unsafe conditions or personnel injury. Many procedures do not contain sufficient information for use by lesser experienced personnel. Only minimal formal training exists for procedure writers and validators. Two instances of worker failure to use procedures were reported. This summary grossly understates the scope of the procedure problem at the Hanford Site as observed by the Board staff. In some cases, Board staff found that facilities were unaware of the existence of WHC standard WHC-CM-3-5. Many of the procedures reviewed were technically inadequate, not just in human factors, but in technical content. In most cases, no documentation of the technical basis for a procedure was available. At least two instances in which verbatim compliance with procedures would result in unsafe conditions were observed by the first review team at their first facility. Further, the suspension of Tank Farms sampling activity was attributed to inadequate procedures--in particular, an inability to verify that safety requirements identified in the sampling safety basis documents were properly incorporated into the procedures. With few exceptions, all technical procedures examined by the Board's staff included extensive text not related to process technical content. These sections did not appear to serve any technical purpose--they merely added bulk and diluted the actual technical requirements of concern. Also, it is inconceivable that a procedure could be classified as technically adequate yet not contain sufficient information for use by lesser experienced personnel. The fact that formal training for procedure developers and validators/verifiers is minimal is exacerbated by the fact that many of these personnel lack any formal engineering training at all. Although many of these personnel are classified as "plant engineers," this title is merely a contractor pay category and does not require that individuals have an engineering degree or background. Failure to use procedures remains a conduct of operations problem endemic to the Hanford Site, although some improvement has been noted. However, the significance of this reported finding pales in comparison to the numerous deliberate violations of procedures observed during the review by both the PAR teams and the Board's staff and unreported in the WHC summary. At least one such incident was observed at each facility visited by the Board's staff during procedure performance/walkdowns-- multiple incidents were the rule. In several cases, the violation was countenanced by a review team member or facility management. 5. Future Staff Actions: The Board's staff will continue to monitor WHC procedure upgrade efforts closely. In the short term, this will involve observation of the WHC technical procedure assessments scheduled as a follow-on to this effort, starting with B-Plant/WESF (identified as the worst facility by WHC and assigned the first follow-on visit) in September 1995. Attachment 1 Detailed Examples from Board Staff Hanford Site Faci lity Procedure Reviews [NOTE: All procedures were selected at random from the facility-provided list of currently effective technical procedures.] T Plant/Solid Waste Division Nine (9) Procedure History Files reviewed; five (5) procedures were observed in progress or during walkdowns. No formal qualification requirements were specified for procedure writers. Qualification requirements and involvement of cognizant engineers could not be determined. No technical basis documentation was available for any procedures reviewed. All five (5) observed/walked down procedures contained technical discrepancies that would preclude completion of the task in accordance with the procedure. In two (2) cases, operators also disputed technical criteria/specifications in the procedures as erroneous, but lack of basis documentation precluded accurate resolution of the dispute. Caution-like tags were found to be necessary (by facility management) on at least four (4) major steam valves to preclude operator manipulation without use of procedures. East Tank Farms, West Tank Farms, and the 242-A Evaporator Fifteen (15) Procedure History Files reviewed; seven (7) procedures were observed in progress or during walkdowns. No formal qualification requirements were specified for either cognizant engineers or procedure writers. No technical basis documentation was available for any procedures reviewed. All seven (7) observed/walked down procedures contained technical discrepancies, at least three (3) of which would preclude completion of the task in accordance with the procedure. Three (3) violations of procedures were observed. K Basins (including East and West) Twelve (12) Procedure History Files reviewed; three (3) procedures were observed in progress or during walkdowns. No formal qualification requirements were specified for either cognizant engineers or procedure writers. No technical basis documentation was available for any procedures reviewed. All three (3) observed/walked down procedures contained technical discrepancies that would preclude completion of the task in accordance with the procedure. One (1) violation of the DOE Radiological Control Manual was observed (operator found it to be necessary to complete the procedure). Plutonium Finishing Plant (PFP) Ten (10) Procedure History Files reviewed; eight (8) procedures were observed in progress or during walkdowns. No formal qualification requirements were specified for procedure writers. Cognizant Engineer qualification requirements could not be determined. No technical basis documentation was available for any procedures reviewed. Six (6) observed/walked down procedures contained technical discrepancies that would preclude completion of the task in accordance with the procedure. Two (2) violations of procedures were observed. B Plant/Waste Encapsulation and Storage Facility (WESF) Twelve (12) Procedure History Files reviewed; seven (7) procedures were observed in progress or during walkdowns. No formal qualification requirements were specified for cognizant engineers. No technical basis documentation was available for any procedures reviewed; facility management disputed the requirement to maintain such documentation. All seven (7) observed/walked down procedures contained technical discrepancies, at least two (2) of which would preclude completion of the task in accordance with the procedure. Two (2) violations of procedures were observed. One (1) was also a violation of the DOE Radiological Control Manual. The other was a deliberate violation observed in the presence of facility management - no action was taken to stop or correct the violation. PUREX Ten (10) Procedure History Files reviewed; four (4) procedures were observed in progress or during walkdowns. No formal qualification requirements were specified for either cognizant engineers or procedure writers. No technical basis documentation was available for any procedures reviewed; five (5) of the procedures were overdue for periodic review. All four (4) observed/walked down procedures contained technical discrepancies. One (1) deliberate violation of procedures (failure to use) was observed.