DEFENSE NUCLEAR FACILITIES SAFETY BOARD August 20, 1992 MEMORANDUM FOR: G. W. Cunningham, Technical Director COPIES: Board Members FROM: David C. Lowe THROUGH: Andrew G. Stadnik, Savannah River Site Team Leader SUBJECT: Savannah River Site Defense Waste Processing Facility Vitrification Plant Trip Report (July 8-10, 1992) 1. Purpose: This trip report documents the Defense Nuclear Facilities Safety Board (DNFSB) technical staff and outside experts July 8-10, 1992 review of operations, training and qualification, and implementation of technical programs at the Savannah River Site (SRS) Defense Waste Processing Facility (DWPF) Vitrification Plant. DNFSB technical staff included David Lowe, Daniel Ogg, Matthew Moury, James McConnell, Joseph Sanders, and Victor Williams, and outside experts Richard Thompson and Doug Volgenau. 2. Summary: A summary of the DNFSB staff review team's major conclusions and concerns are provided below: a. The Department of Energy (DOE) Office of Environmental Restoration and Waste Management (DOE-EM) has scheduled its Operational Readiness Review (ORR) for cold chemical runs to commence September 1, 1992. Based on the DNFSB staff team's observations of the immature state of conduct of operations, operator and supervisor qualification status and knowledge level, and status of technical programs, September 1, 1992 appears to be optimistic. The conduct of an ORR to the standards conducted at SRS K-Reactor, Rocky Flats (Building 559), or the Waste Isolation Pilot Plant (WIPP) would not turn out to be an evaluation of "readiness." b. Westinghouse Savannah River Company (WSRC) management has not implemented a critical self-assessment program to evaluate the state of development and maturity in a wide range of programs. Rather, the emphasis seems to be heavily focused on hardware systems and testing in order to achieve an accelerated cold chemical runs startup schedule. In the absence of timely self-assessment, WSRC management lacks awareness of the condition of operator training and qualification, conduct of operations, design baseline determination, and hardware preventive maintenance. 3. Background: The DWPF Vitrification Plant is currently undergoing startup testing in preparation for cold chemical runs. The current WSRC plan is to have all preparations completed for cold chemical runs by September 1, 1992. The previous schedule had cold chemical runs starting on November 20, 1992. DWPF management and personnel are focused towards being ready for cold chemical runs by the revised accelerated date. DOE Savannah River Field Office (DOE-SR) and WSRC recognize that this revised schedule is ambitious and may not reflect the time required to complete all of the pre-startup testing, and allow sufficient time for the completion of the required WSRC and DOE ORRs and the correction of any deficiencies identified by these reviews. 4. Discussion: The DNFSB staff team conducted a review of operations, training and qualification, and implementation of technical programs. This review consisted of DOE-SR and WSRC technical briefings and discussions; interviews of DWPF operators, supervisors, cognizant systems engineers, and a DOE-SR Facility Representative; observation of a classroom training session; facility walkthroughs and observations of shift turnover briefings and planning meetings; and documentation reviews. The DNFSB staff team reviewed several programs underway to support starting cold chemical runs on September 1, 1992. These included efforts to complete startup testing, reconstitute baseline data, confirm as-built status, enhance configuration control, human engineer the design of the process alarm system, improve training and complete interim qualifications, improve conduct of operations, and complete the WSRC ORR for the cold chemical run phase. While progress has been made in several of these areas, it appears that the primary focus of DWPF management is to complete only those actions viewed as necessary to get the hardware systems ready for cold chemical runs. a. DOE Technical Vigilance: DOE-SR surveillances have been conducted in various areas, and it was noted that they appear to be of higher quality than those conducted at other DOE facilities. There are six DOE Facility Representatives assigned to DWPF and one vacant position. The DOE-SR facility representative training and qualification program has recently been finalized, so Waste Management is now adjusting their program to be in compliance with the DOE-SR guidance. The first facility representatives are expected to be qualified in the December 1992 timeframe, after the DOE-EM ORR. b. Operational Readiness Reviews: WSRC has an ongoing review process that has been labeled an ORR. This ORR began in 1989 and is ongoing with reporting milestones that coincide with the major DWPF start-up milestones. The ORR organization is staffed with a full-time core staff as well as full-time temporarily assigned personnel (presently 8 ORR team members are Westinghouse Hanford employees involved in a technology exchange program) with the number of positions variable depending on the milestone the ORR is assessing. The ORR team leader reports to line management above the DWPF Program Manager, which provides adequate independence. Functionally, the ORR provides input to a Readiness Review Board (RRB) which reviews the ORR team's effort. The ORR review procedure is organized into 24 modules. Some of these modules wee assessed up to two years ago under different management, which applied a different safety philosophy and utilized different standards. These reviews cannot be relied upon to ensure operational safety for future milestones. The current WSRC ORR process for DWPF is worthwhile, but it seems to best fulfill the role of Performance Assurance instead of an Operational Readiness Review. The DOE ORR for cold chemical runs will be conducted by DOE-EM and is planned for the September 1992 timeframe. The ORR plan and criteria and review approaches (CRAs) were expected to be complete in late July 1992, but are still not available. c. Shift Organization: A DWPF shift consists of 33 personnel: Shift Manager, Shift Technical Advisor (STA) (new position-not implemented), 4 supervisors, and 27 operators. There are currently 19 operator qualification areas, but this will be reduced to 4 areas (control room, crane, vitrification support, and balance of plant) prior to radioactive operations. The manner in which this consolidation of qualification areas is conducted will be the subject of future DNFSB staff reviews. WSRC recently decided to include an STA on each shift, with STA selection scheduled in the November 1992 timeframe. The DNFSB staff team considers this as a very positive development and will continue to track the implementation of the STA program. d. Operator Training, Qualification, and Certification: The DWPF operations training program is ambitious and contains sound ingredients. It includes fundamentals training, job specific skills and knowledge requirements, conduct of operations training, and training in emergency response. WSRC indicated that much of this training had been completed. Additionally, WSRC implemented a fifth-shift for training in January 1992 in order to support continuing training and initial qualification/certification. Operator qualification consists of completion of a qualification card, which includes classroom training (fundamentals and watchstation specific), basic simulator familiarization training for control room operators, job performance measures (JPMs), and written and oral examinations. A provisional qualification program to support cold chemical runs is planned, but no operators or supervisors have completed this program. The provisional qualification written examination is not a comprehensive examination, but consists of the written examinations taken at the completion of each of the classroom courses. The oral examination (chaired by the Operations Manager) is intended to be a comprehensive examination with the initial oral examinations scheduled to take place in July-August 1992 timeframe. WSRC indicated that they were in compliance with DOE Order 5480.20, Personnel Selection, Qualification, Training and Staffing Requirements at DOE Reactor and Non-Reactor Nuclear Facilities, because the Training Implementation Matrix (TIM) was prepared. However, following the reviews it was apparent that WSRC was not in compliance with many of DOE Order 5480.20 requirements. A review of supervisor and operator training and qualification records indicated the following: 1. The DWPF training and qualification records are not documented in an "easily auditable format" as required in DOE Order 5480.20 Chapter I.16. Individual records reviewed were incomplete and did not contain the items required by DOE Order 5480.20, but they did contain many unnecessary papers. 2. Many of the operators assigned to shift positions had not been certified or qualified to accomplish the job tasking associated with that position. Some operators had been qualified for a milestone that had been completed, but there was no documentation to support their assignment to a shift position once the milestone had been completed. WSRC plans require provisional qualification before commencement of cold chemical runs. Currently, WSRC is engaged in an ambitious schedule of startup testing, but there is no process to certify the qualifications of people assigned to a position on shift. 3. Startup testing is in progress and a number of these tests involve operations similar to those expected when DWPF is fully operational. But, no provisional qualification or certification process is currently in-place. 4. Written examinations were not challenging and did not adequately examine an individual's knowledge level. The exams consisted of multiple choice with some short answer questions. Additionally, process and shift position written exams did not interrelate fundamental engineering principles with operator knowledge requirements. 5. A fifth-shift was added in January 1992, ostensively to conduct training needed to support operator development. Records of the training actually completed and who received particular training are incomplete, and this information is not recorded in the individual training records. 6. Interviews and discussions with operators indicated that fifth-shift supervisors were frequently required to give OJT checkouts during the training shift and were not able to receive training. Overall, it did not appear that the fifth- shift was being effectively utilized to maximize the intended training benefit. 7. Drill scenarios and records of drills conducted were reviewed. Most of the drills conducted consisted of table top discussions or walkthroughs. It is not clear that drills are being used by WSRC management to critically assess implementation of conduct of operations, operator training, and procedure adequacy. For example, a chemical spill drill was conducted with "no deficiencies" noted, but some areas requiring improvement were listed. Several "strengths" were also listed, including: communications and conduct of operations. It appears that this was the first, and only, spill drill that has been conducted at DWPF. It is difficult to imagine that the first spill drill conducted could be evaluated as having "no deficiencies". Additionally, drill scenarios were short, incomplete, and did not lend themselves to the conduct of a safe and productive drill program. 8. Control room operator training includes familiarization training using a control room console simulator. This simulator currently can be used only for general familiarization training. It does not include all of the interlocks associated with the actual process distributed control system (DCS) and does not accurately simulate dynamic plant response. The simulator comprises enough hardware to replicate two control room operator stations (our of four in the control room). Overall, while the simulator is valuable for operator familiarization, it cannot be used by control room teams to gain operating experience. Development of this level of sophistication is under consideration, but cannot be available until the 1994 timeframe. e. Supervisor and Operator Interviews: Interviews of DWPF control room and field operators, a control room supervisor, and a shift manager were conducted. In general, weaknesses were noted in the principles of conduct of operations, maintenance of plant status, the importance of maintaining plant configuration, and engineering and process fundamentals. Additionally, some operators could not relate fundamentals learned in classroom training to operational situations. The following specific observations are provided: 1. A shift manager interviewed appeared to have significant supervisory experience in the nuclear industry, but his knowledge of the DWPF process was inadequate. He could not draw a one-line drawing of the salt processing cell and provide an adequate technical explanation of the function of the salt processing cell. 2. A salt processing cell operator did not know temperature or pressure limits to be observed in the salt process cell despite completion of the salt processing cell technical course less than one month earlier. 3. A melter cell operator did not know the criteria for automatic switch-over to the backup off-gas system, or when manual switch-over is required. 4. Some field operators did not know the proper method to check shut and check open valves when conducting a valve lineup. f. Conduct of Operations: There is evidence that conduct of operations is being implemented and practiced at DWPF, but it is far from being an ingrained philosophy. Conduct of operations training has been given to some, but not all, operators. The plans for providing this training seem to demonstrate a lack of appreciation for the importance of providing the base philosophy and building blocks to all operators and support organizations early in the facility startup process. This inconsistent application of DOE Order 5480.19, Conduct of Operations Requirements for DOE Facilities, is indicated by the following examples: 1. The status of plant equipment and systems is difficult for the shift manager to determine. The plant status board provides basic equipment in-service/not-in-service indications, but determining equipment lineups requires an in-depth effort to recover system by system information from document control. Procedures reviewed revealed that some have signature requirements, some do not, and none have any independent line- up verification requirement. This is an example of a system that does not reflect good conduct of operations and operating personnel who do not understand that their pocedures are not matured to current standards. 2. Control room operators on-shift were permitted to be relieved by unqualified operators for necessary breaks. Since each shift is twelve hours in length, breaks are necessary. This practice was planned to be discontinued on July 24, 1992. Also, there is no provision to deal with the difficulty of manning a control room console for an extended period of time. 3. There is no consistency among control room operators in operation of their respective panels, particularly regarding the proper display to be selected for a given operating condition and operator response to alarms. A major reason for this insconsistency is the lack of control room console operations training and a full-scope control room simulator. The philosophy of operations for DWPF does not appear to be fully developed. 4. An oncoming supervisor had a personal problem that prevented his standing shift duty. The previous control room supervisor attended the shift turnover briefing and then was allowed to depart. Therefore, that supervisory position remained vacant for the shift. This situation was approved by the Shift Manager with no compensatory measures implemented. WSRC management appears to understand these shortcomings and plans to upgrade the conduct of operations. However, the past and current inconsistent application of the principles of conduct of operations will require considerable effort and retraining to correct. g. Technical Training: The DWPF technical training program for cognizant engineers is focused to support cold chemical runs. It includes classroom instruction on configuration management, computer software change control (self-paced), event root cause analysis, safety envelope, and unresolved safety question determination. Thee is no technical training on DWPF process fundamentals. Some of the course material has not been developed and post cold chemical run training is being formulated. The training program does not include oral examinations and the written examinations are not challenging. h. Instrumentation and Control: DOE-SR and WSRC have determined that DWPF control room operators are inudated with too many alarms. There are approximately 4,600 process alarms which activate in the control room. This number has already been reduced from approximately 13,000 by routing computer system alarms to the computer center. Alarms are assigned to two priority classes (Priority I and II), but this distinction only affects the color of the alarm panel screen display. All alarms have the same audible indication and are processed in the same manner. Alarms are displayed chronologically without regard to priority. In addition, the computer console can display only 16 alarms at one time. A total of 64 alarms (4 pages) is accessible from the computer console memory. To view additional alarms, the control room operator must scan a hard copy print out. If more than 64 alarms are received in a short period (an event that has happened at DWPF) the earliest alarms, which could indicate the underlying problem, will have already "scrolled off" the alarm indicating screens before the operator can respond. One control room supervisor stated that his crew spends about 20% of their time responding to alarms, the vast majority of which were not significant. If this condition continues, the operators may become desensitized to alarms which could result in delaying critical operator actions in the event of an actual emergency. WSRC is currently considering plans to improve the alarm system. Several committees have been formed to study the problem, but no formalized course of action has yet been approved. On proposed solution involves streamlining the number of alarms to approximately 700 and grouping them into three distinct categories. The three categories would be advisroy (no action required, information only), alerts (action required), and alarms (alarm response procedure required). The DNFSB staff team believes that reducing the number of alarms, and improving the display and processing of thos alarms should be a very high priority. i. Configuration Management: The DWPF Configuration Management (CM) Manager is matrixed to the DWPF Program Manager from the Site CM division of the Engineering and Technical Department. The CM Manager (Level 4) reports to the Department Manager (Level 3) who is one of seven departments under the Engineering Program Manager (Level 2). It is noted that the CM Manager is the coordinator for implementing CM over seven departments from his Level 4 position. Aggressive support and continual interest from senior line management is crucial to his ability to implement CM at DWPF. Earlier Design Basis and Design Baseline documentation for DWPF are incomplete and are not documented to current standards. Lapses in control of design documents and quality control records are acknowledged by DWPF personnel. Selected record reviews of early assessments (1988-1989) concluded in many cases "documentation not available, or irretrievable". The DWPF Configuration Management Plan was approved by the DWPF Program Manager in January 1992. It is based on a draft Nuclear Information and Tecords Management Association (NIRMA) Technical Guideline. The plan is a five phased technical approach with the Configuration Management (CM) Plan being the product of Phase I (Definition). Phase II (Evaluation) is ongoing from February 1, 1992 and is scheduled to be completed by October 1, 1992. This phase evaluates the existing configuration control processes and compares the current physical plant configuration to the configuration of the facility as depicted in the configuration control documents. Phase III (Implementation) is the most costly and time consuming phase and involves the reconstitution of the technical baseline documentation to accepted standards, and development or upgrades to the existing configuration control processes. Phase IV (Validation) is the validation of the established CM Program and Phase V (Feedback) is an ongoing process of refining the configuration control processes and configuration items. Based on past CM shortcomings, estimated costs are $20-30M and will require 3 to 3.5 years to accomplish the plan with the majority of the funding for Phase III. Phase III will be implemented using the following nine functional areas: Configuration Baseline, Design Change Control, Plant Status Control, Material Control, Document Control, Information Systems, Regulatory Compliance, Independent Review, Training and Administration. The development of the CM Program will take advantage of existing CM processes to the maximum extent possible in order to reduce the cost impact. A formal program is now in place to control changes to the design documentation using a Design Change Control Process. This program appears fundamentally sound to ensure configuration control is maintained. However, design changes can be initiated by anyone and if approved, the resultant design work is not independently reviewed for technical adequacy (calculations, standards, dimensions, etc.). The design work is only reviewed by the Change Control Board which only considers whether the design meets the intended functional requirements. The current sources of existing design basis information are scattered throughout various documents including: product specifications, critical function reports, process descriptions, design calculations, procurement specifications, project directive ltters, Savannah River Technology Center (SRTC) technical data reports, Process Requirements, and the Safety Analysis Report (SAR). During Phase III, System Description Documents (SDDs) will be developed to provide a "single, living source in a concise format". The SCC pilot program will begin in August 1992. All SDDs are scheduled for completion by December 1993. The SDDs format will be based on design basis documentation requirements in the commercial nuclear industry. Based on recent experience at K-Reactor in developing system description documentation, the completion of quality SDDs by December 1993 appears optimistic. The cognizant system engineers will be tasked with developing the SDDs. However, discussions with cognizant system engineers indicated an absence of a formal method/procedures or schedule for completion of this effort. The construction contractor (Bechtel National, Inc.) is presently under contract to reconstitute the baseline documentation. The facility is currently scheduled for radioactive operations in June 1994. Completion of Phase III is scheduled for October 1994 and Phase IV for April 1995. In view of the extended DWPF operating life, a more aggressive effort may be appropriate in order to complete the baseline documentation before commencing radioactive operations. The Safety Class Structures and Equipment have only recently been finalized by WSRC, and have not been approved by DOE-SR. The Safety Classification does not consider radiological consequences for on- site personnel and DWPF workers. Changes to this approach, to be consistent with commercial nuclear standards, may impact the CM plan implementation schedule. j. Software Configuration Management: DWPF management appears to be showing increased awareness of the importance of software configuration management and quality assurance (QA). Recently, a manager was assigned to direct software QA and his staff includes an Electrical Engineer with programmable logic controller experience. A separate manager, with Distributed Control System (DCS) experience, was recently selected to coordinate software CM. WSRC's improvements in this area are fairly recent and the program for software CM at DWPF is not fully implemented. While the DNFSB staff team is encouraged by changes evident so far, additional progress towards implementation of software CM at DWPF is still required. WSRC personnel assigned to software CM and QA appear to be qualified for the job, but they have had insufficient time to implement a program to achieve compliance with software CM and QA standards. Based on technical discussions, WSRC appears to have an adequate appreciation for the importance of software CM and QA. The DNFSB technical staff will periodically assess the implementation of software CM and QA at DWPF. k. Process Engineering: DWPF recently established a Process Engineering group to be the engineering lead on process-related issues and to provide a technical interface to the Savannah River Technology Center (SRTC), which operates the Integrated DWPF Melter System. A manager has been selected, but no staff have been assigned. The DNFSB staff team considers the establishment of a process engineering capability at DWPF a positive development and we will continue to monitor its effectiveness.