DEFENSE NUCLEAR FACILITIES SAFETY BOARD
August 8, 1995
MEMORANDUM FOR:G. W. Cunningham, Technical Director
COPIES: Board Members
FROM: Timothy J. Dwyer
SUBJECT: Review of DOE 1994 Technical Personnel Hiring Data
1. Purpose: This memorandum discusses a review of 1994 technical personnel hiring data from across the Department of Energy (DOE) defense nuclear facilities complex. This review was conducted by members of the Defense Nuclear Facilities Safety Board's (Board's) staff, Timothy J. Dwyer and Robert F. Warther, and outside experts John F. Drain and Ralph W. West, Jr. This review did not involve any evaluations of technical personnel performance in the field.
2. Summary: Throughout calendar 1994, DOE continued its efforts to improve the technical capabilities of the DOE staff, in response to Board Recommendation 93-3. In parallel with this effort, the Secretary of Energy authorized the hiring of nearly 1200 personnel. A few highly talented, well-qualified individuals were hired; however, many senior technical positions within DOE were filled by individuals drawn from a DOE technical population considered by several high level reviews to be deficient in personnel of exceptional technical qualities. Thus, DOE failed to take advantage of the unique opportunity this hiring authority provided to substantially raise the technical capabilities of the DOE staff.
In short, the data indicates that Board efforts to encourage DOE to ". . . raise the technical expertise of the Department [DOE] . . ." through allocation of technically qualified individuals to federal positions in the defense nuclear complex were not effective in 1994.
3. Background: The technical capabilities of DOE and contractor personnel have been an ongoing concern of the Congress, the Board, and numerous independent review groups for a number of years. The United States Senate Report accompanying the Board's enabling legislation states that the "Board is expected to raise the technical expertise of the Department [DOE] substantially. . . ." The Board has repeatedly stated, in its annual reports, that ". . . the most important and far-reaching problem affecting the safety of the DOE defense nuclear facilities is the difficulty in attracting and retaining personnel who are adequately qualified by technical education and experience to provide the kind of management, direction, and guidance essential to safe operation of DOE's defense nuclear facilities." Several other prominent organizations, including the National Research Council, the "Ahearne Committee," and the DOE itself have also weighed in on this matter with similar concerns. In this vein, Recommendation 93-3 was issued in June 1993 to recommend improvements in the recruitment, retention, education, and training of DOE's technical personnel. In fact, this Recommendation specifically called on DOE to ". . . establish the attraction and retention of scientific and technical personnel of exceptional qualities [emphasis added] as a primary agency-wide goal."
DOE's Implementation Plan for Recommendation 93-3 was submitted and accepted in November 1993. Throughout calendar 1994, DOE labored to implement the various portions of this plan. In parallel with this effort, the Secretary of Energy authorized the various defense complex DOE Offices (most notably the Office of Environmental Management (EM), and to a lesser degree the Office of Environment, Safety and Health (EH)) to hire a total of nearly 1200 personnel. Therefore, in early 1995, the Board's staff requested that DOE provide data that would permit an evaluation of DOE's effectiveness at attracting highly qualified scientific and technical personnel. It is emphasized that the documentation provided by DOE was the only source material used in this review. No evaluations of personnel performance in the field were conducted, nor were any interviews, reference checks, or other information gathering techniques employed.
4. Discussion.
a. Overall 1994 DOE Technical Hiring Data. The data provided by the DOE to conduct this review consisted of 467 Standard Form 171s (SF-171s) and their associated Position Descriptions (PDs), in some cases, augmented by their Vacancy Announcements, each set representing one individual who had filled a previously vacant technical DOE billet (either as a new hire, lateral transfer, or promotion) during calendar 1994. This data was not representative of all DOE hiring in 1994; rather, it concerned only technical personnel hired at the following DOE defense complex offices: the Office of Defense Programs (DOE- DP), DOE-EH, DOE-EM, the Albuquerque Operations Office (DOE-AL), the Idaho Operations Office (DOE-ID), the Nevada Operations Office (DOE-NV), the Oakland Operations Office (DOE-OAK), the Oak Ridge Operations Office (DOE-OR), the Ohio Operations Office (DOE-OH), the Richland Operations Office (DOE-RL), the Rocky Flats Operations Office (DOE-RF), and the Savannah River Operations Office (DOE-SR). Data on 17 technical DOE defense complex senior executive service (SES) billets filled in 1994 were also included as part of the 467 SF-171s/PDs, but were treated separately. Of the 467 SF-171s/PDs submitted, 22 were discarded upon initial review as not related to technical billets (or containing insufficient data for classification or use in the review), yielding an overall sample size of 445.
The types of billets defined as "technical" included those identified as chemical engineers, civil engineers, electrical engineers, mechanical engineers, nuclear engineers, facility representatives, fire protection specialists, occupational safety specialists, radiological protection specialists, technical program/project managers, etc. The final data set included billets ranging from the GS-5 to the SES levels.
Figure 1 provides an overview of the number of technical billets filled (less SES's) in 1994 at each of the locations for which data was provided. Also depicted is the subset of that number already employed in DOE prior to accepting the new technical billet.
It is significant to note the DOE had difficulty collecting and providing this data. Initial DOE figures concerning 1994 hiring totaled 771 individuals, 291 technical and 480 non- technical. DOE later revised these figures to 470 technical, and 505 non-technical (975 total). Based on a review of the SF-171/PD data provided, the number of technical billets filled in the DOE defense complex in 1994 was 445; the fidelity of the DOE figure for non- technical billets (505) is questionable and is most likely valid only as a floor value.
Based on the data collection difficulty encountered, it is concluded that no mechanism existed for DOE senior managers to receive feedback on the efficacy of DOE technical personnel hiring efforts. This lack of feedback is indicative of a failure to adequately manage the process. Despite Board admonitions to DOE senior managers that they become personally involved in the process of hiring scientific and technical personnel of exceptional qualities, and that the intention to fill 1200 billets in the DOE defense complex provided a rare opportunity to raise the technical expertise of their Offices, it is obvious that DOE did not take full advantage of this chance to improve the technical capabilities of its staff.
Further, despite the fact that DOE had the authority to recruit 200 exceptionally qualified senior personnel through an excepted service program, and received authorization in October 1994 for 200 additional technical excepted service billets, no excepted service positions were filled in 1994.
b. Evaluation of the Technical Qualifications of 1994 DOE Technical Hires. The 445 SF-171/PD sets were evaluated to determine the degree to which the SF-171 of each individual hired satisfied the specific Eligibility Requirements, Ranking Factors, and Duties and Responsibilities of the PD (and Vacancy Announcement, where available) under which the individual was hired. It is important to emphasize the fact that the standard used to evaluate each SF-171 was the same PD used by DOE to determine that the individual in question was the best-qualified candidate for the job.
For each SF-171/PD set, a grade was assigned, ranging from one to five. A grade of one signified that, based solely upon the SF-171 data, the individual did not meet the criteria of the associated PD, and accordingly, was not qualified for the assignment. A grade of three signified that the individual was probably qualified to the minimum criteria associated with the PD. A grade of five signified that, as described in the SF-171, the individual met or exceeded all criteria associated with the PD, and appeared to be an excellent match for the billet described. (See Attachment A for further details on the method of evaluation used.) This data was collected for all 445 1994 DOE technical hires and is depicted in histogram form in Figure 2.
Note that the data approximates a normal (Gaussian) distribution: the data mean is 3.08, with a standard deviation of 1.04. In fact, it is strikingly similar to the smooth curve (Figure 2) that plots the normal distribution obtained for 445 data points with a mean score of 3.00 and standard deviation (1.14) fixed such that scores outside the range (of one to five) are limited to approximately one percent of the sample size. The significance of the similarities between the two plots rests on the fact that the smooth curve represents a hiring process in which the desired outcome is selection of an individual who probably meets all of the criteria of the PD (i.e., a score of 3), with selection of an individual who does not meet the criteria (e.g., a score of 1) or selection of an individual who is well- matched to the criteria (e.g., a score of 5) occurring with a frequency dictated by a random process centered on 3. Simply stated, analysis of the qualifications (relative to their PDs) of 1994 DOE technical hires reveals that the hiring process tended towards selection of the minimally qualified candidate, and selection of a highly qualified technical candidate occurred with no greater frequency than that which would be expected of a random selection process.
A more telling comparison can be made by considering how much improvement is required of DOE to begin raising the technical expertise of the DOE staff substantially. This would require that DOE not hire any technical personnel who would score below minimally qualified (i.e., hire only for scores of three to five). To accomplish this, DOE would have to develop a hiring process that actively screens applicants to not select unqualified candidates (scores of one or two). Had this criteria been applied, fully 30 percent (134 of 445) of the 1994 DOE technical hires would not have been selected. Note that, from the 1994 DOE data, only 37 (less than 10 percent) of the 445 SF-171s/PDs were scored as highly qualified technical matches (score of 5) for the position in question.
The observed distribution of qualification scores for individual candidates did not improve even for the more senior positions filled in 1994. Figure 3 depicts the qualification scores obtained if SES data only is extracted from the data set. Once again, overlaid on the data is the normal curve that represents a hiring process in which the assignment of technical SES applicants to available technical SES billets selects the minimally qualified candidate. The comparison indicates that even at senior management levels, 1994 DOE performance did not result in hiring a significant number of technical candidates highly qualified with respect to their PDs. (This subject is discussed further in Section 4.d of this memorandum.)
c. Evaluation of the Educational Background of 1994 DOE Technical Hires. The 445 SF-171s were further reviewed to evaluate the educational background of the 1994 DOE technical hires. In this review, the objective was to develop a snapshot of the technical educational background of the entire data set; in the aggregate, the quality of the educational background of a workforce is generally indicative of their technical capabilities. (Technical is defined to include the fields of Engineering, Science, or Mathematics.) An independent standard was chosen as the yardstick by which to score educational quality: The Gourman Report. The Gourman Report is an objective evaluation of degree program curricula at individual colleges/universities, based on such factors as faculty, standards, admission requirements, and, most importantly, student performance following graduation.
Using both the undergraduate and graduate versions of this document, the educational background (including both curriculum and school) presented on each SF-171 was converted to a standardized quantitative score. These scores varied on a scale of zero (very poor at the undergraduate level, unacceptable at a graduate level) to five (very strong to excellent) for each level of degree (i.e., Baccalaureate, Masters, Doctoral). (See Attachment B for further details on the method used to evaluate educational quality.)
The first striking piece of data that falls out of this analysis is shown in Figure 4. Over 10 percent (49 of 445) 1994 DOE technical hires did not have at least a Baccalaureate technical degree. Nineteen had no degree. This is significant in that the Office of Personnel Management (OPM) minimum standard for hiring a GS-5 employee is four academic years beyond high school leading to a Bachelor's degree; a GS-7 should have a Bachelor's degree with at least two years of Superior Academic Achievement, or one year of graduate education.<1> Yet all 19 were hired at the GS-7 level or higher -- nearly two-thirds were hired at the GS-12/13/14 levels. It is clear that these personnel, plus the other 30 lacking a technical educational background, will be limited in their ability to provide technical expertise to DOE.
Further information developed through this review of educational backgrounds is depicted in Figure 5. To understand the data presented, it must first be pointed out that The Gourman Report does not rate schools in the range 0.0 to 2.0. Any school that would be scored below 2.0 is considered "not adequate" and is defaulted to a score of 0.0. Thus, there is a natural gap between 0.0 and 2.0. For purposes of this review, the following values have been placed in this gap: the range 0-0.5 has been used to count individuals lacking a four year technical degree; and the range 0.51-1.0 has been used to count individuals whose degree was obtained at a school not scored by The Gourman Report (in most cases, because the school was not in the U.S.). As can be seen, the distribution of education quality scores at the baccalaureate level is spread rather uniformly across the spectrum from 2.5 to 5.0. This is indicative of a hiring selection process that is not screening for highly qualified technical educational backgrounds. Using the scoring categories employed in The Gourman Report (see Translation of Grading Scales in Attachment B), educational programs must be scored 3.6 or higher to be merely "good." "Strong" technical educational programs are scored as 4.0 or higher. The absence of a bias toward the last two columns (4.01-4.5 and 4.51-5.0) in the 1994 DOE data indicates that DOE did not place adequate emphasis on scientific and technical educational backgrounds as criteria for personnel selection.
The data depicting the educational quality of 1994 DOE technical hires educated to the Masters level generally reveals the same absence of bias towards the higher end of the scale, although there is definitely an increase in the proportion of "very strong" educational backgrounds at this level. It is only at the Doctoral level that the educational quality data reveal a bias toward selection of highly technically qualified individuals; however, as was seen in Figure 4 this population represents less than four percent (16 of 445) of the 1994 DOE technical hires. Furthermore, the universe of available doctoral programs is already naturally skewed toward higher rated schools.
d. Evaluation of the Source of 1994 DOE Technical Hires. The 445 SF-171s were examined to determine the source of the 1994 DOE technical hires. That is, for each individual, the job which he left to accept the DOE position in 1994 was categorized as either in DOE, in some other government agency, or outside of government (i.e., recruited from industry or school). For those SF-171s source categorized as in DOE, a further distinction was drawn as to whether the transfer to the new position was lateral or involved a promotion.
As depicted in Figure 6, overall, approximately 45 percent (203 of 445) of the 1994 DOE technical hires were already employees of DOE when they accepted their new position. Approximately 30 percent (136 of 445) transferred from other government agencies, and nearly 25 percent (106 of 445) were recruited from outside of government. Thus, nearly half of the 1994 DOE technical hires were drawn from a pool considered by several prominent review groups as lacking in scientific and technical excellence (see sources cited in the Background section of this memorandum, as well the letter from the Secretary of Energy to the President, December 20, 1991).
The longer-range aspects of this observation can be illustrated by examining the data relating to the senior levels of management in DOE. Figure 7 depicts the distribution of hiring sources for 1994 DOE technical hires at the GS-14, GS-15, and SES levels. As can be seen, more than 80 percent (127 of 158) of the 1994 senior level technical hires were drawn from this DOE population. Thus, even if (contrary to the findings presented in Sections 3.b and 3.c of this memorandum) the DOE technical personnel hiring process was successful at bringing in scientific and technical individuals of exceptional capabilities, they would be under the leadership of senior managers responsible for the culture that engendered Board Recommendation 93-3.
5. Future Staff Actions: The Board's staff will continue the evaluation of DOE technical personnel hiring through analysis of quarterly DOE data from calendar year 1995. The Board's staff is also considering evaluating DOE's staff (and contractors' staff) in their performance of safety functions at defense nuclear facilities.
Individual DOE Position Descriptions were reviewed to ascertain the following:
Individual SF-171s were then compared to the associated individual as-hired Position Descriptions, taking into account the criteria specified above. Grades were assigned to each Position Description/SF-171 pair based on the following criteria:
5 This individual is an excellent match for the Position Description, meeting or exceeding all criteria. As a hiring authority, this individual would be scheduled for an immediate interview, regardless of transportation requirements/schedule conflicts.
4 This individual is a good match for the Position Description, and appears to meet all criteria. As a hiring authority, this individual would be scheduled for an interview with the next group of candidates to be evaluated.
3 This individual probably meets all criteria in the Position Description. As a hiring authority, this individual might be scheduled for an interview if he was local (i.e., no transportation costs) and no schedule conflicts exist.
2 This individual requires additional training or experience in order to meet all criteria in the Position Description. As a hiring authority, this individual would more properly be considered for the position two or three years from now.
1 This individual does not currently meet all criteria and will not meet all criteria in the near future. As a hiring authority, this individual would likely never be considered for this position.
It is important to note that this review did not take into account any information other than that presented on the documentation available, and that both the Position Description and SF-171 were accepted at face value. No data verification, reference checks, or interviews were conducted in evaluating the candidates. Further, the Position Descriptions supplied were highly variable in quality, especially between sites. Some Position Descriptions clearly specified the duties of the position, and the level of education and experience required to fill the position. Others were so vague as to be only of marginal value.
To account for the fact that multiple reviewers were conducting the individual SF-171/Position Description evaluations, all reviewers jointly evaluated, discussed, and scored the first five SF- 171/Position Description pairs to arrive at consensus grading criteria. The next 17 SF-171/Position Description pairs were individually evaluated and scored by each reviewer, with post-evaluation discussions to aid in achieving significant correlation. [As this accounted for approximately five percent of the data sample, and correlation factors of approximately +.90 between reviewer grades were realized, it was determined that multiple or joint grading would not be required for the remainder of the data.]
As the evaluation process proceeded, 12 additional randomly selected SF-171/Position Description pairs, as well as a mid-point group of 8 SF-171/Position Description pairs, received multiple scorings, using this individual evaluation system, to ensure that correlation between the reviewers remained high. [Because the SES data was treated separately, these 20 data sets amounted to approximately ten percent of the remaining data sample.] In all cases for which multiple, individually evaluated scores were developed, the final score assigned to an SF-171/Position Description pair was the mean of the individual evaluator scores, rounded to the nearest whole number (upward from .50 on).
All SES SF-171/Position Description/Vacancy Announcement data was evaluated jointly, to arrive at consensus grades.
Evaluation of the quality of the technical education of 1994 Department of Energy (DOE) Technical Hires was based upon a recognized independent rating system published by National Education Standards. Undergraduate and Graduate Programs are rated separately in two books:
The GOURMAN REPORT: A Rating of Undergraduate Programs in American & International Universities, Seventh Edition [Revised], National Education Standards, Los Angeles, CA, 1989; and
The GOURMAN REPORT: A Rating of Graduate and Professional Programs in American & International Universities, Fifth Edition [Revised], National Education Standards, Los Angeles, CA, 1989.
As stated in the reports themselves, The Gourman Report is an objective evaluation designed to synthesize complex data into a "deceptively convenient" numerical rating, on a scale from zero (0, very poor) to five (5, excellent). Fourteen specific criteria are taken into consideration in the evaluation process, including:
·Faculty qualifications, experience, intellectual interests, attainments, and professional productivity (including research);
For each 1994 DOE Technical Hire, information concerning schools granting degrees, and the fields in which the degrees were granted, were obtained from the SF-171's. Using this information, a quantitative score was developed according to the following criteria:
UNDERGRADUATE DEGREE SCORING, using The GOURMAN REPORT: A Rating of Undergraduate Programs in American & International Universities:
No Degree = 0.0
Non-Technical Degree (not Engineering, Science, or Mathematics) = 0.5
Technical Degree, but less than four years (e.g., A.S.) = 0.5
Technical Degree, but school not listed in The Gourman Report = 1.0
Four Year Technical (Engineering, Science, or Mathematics) Degree = see below
Enter the Gourman Report using the major field of study to find the appropriate List of Leading Institutions. SCORE the school value in the list, usually in the range 4.0 to 5.0, occasionally in the range 3.0 to 5.0.
- if not found -
Enter the Gourman Report using the school to find the appropriate section of the Rating of Undergraduate Schools in Engineering on the Approved List of the Gourman Report. SCORE the major field of study value in the list.
- if not found -
If the degree is in Engineering, enter the Gourman Report using the school to find the appropriate section of the Rating of Schools in Engineering. SCORE the overall engineering program value in the list, not to exceed the floor value for that major field of study previously found in the List of Leading Institutions.
- if not found, or if the degree is in Mathematics or Science -
Enter the Gourman Report using the school and SCORE the school value in the Overall Academic Rating of American Undergraduate Institutions, not to exceed the floor value for the appropriate major field of study previously found in the List of Leading Institutions.
- if not found -
This school is not rated in Gourman Report, use default scoring of 1.0 [see list above]
GRADUATE DEGREE SCORING, using The GOURMAN REPORT: A Rating of Graduate and Professional Programs in American & International Universities:
No Degree = not scored
Non-Technical Degree (not Engineering, Science, or Mathematics) = 0.5
[Note that Engineering Management/Business Administration/Law Degrees
were treated separately; no score was developed for this examination of
technical graduate degrees.]
Technical Degree, but school not listed in The Gourman Report = 1.0
Technical (Engineering, Science, or Mathematics) Degree = see below
Enter the Gourman Report using the Graduate or Professional field of study to find the appropriate List of Leading Institutions. SCORE the school value in the list, usually in the range 4.0 to 5.0, occasionally in the range 3.0 to 5.0.
- if not found -
If the graduate degree is in Engineering, enter the Gourman Report using the school to find the appropriate section of the Rating of Graduate Schools in Engineering. SCORE the overall engineering program value in the list, not to exceed the floor value for that field of study previously found in the List of Leading Institutions.
- if not found, or if a Mathematics or Science Graduate Degree -
Enter the Gourman Report using the school and SCORE the school value in the Rating of United States American Graduate Schools: Academic and Selective, not to exceed the floor value for the appropriate major field of study previously found in the List of Leading Institutions.
- if not found -
This school is not rated in Gourman Report, use default scoring of 1.0 [see list above]
Translation of Grading Scales Used in the Gourman Reports:
For UNDERGRADUATE ENGINEERING PROGRAMS:
| very strong | 4.51 to 4.99 |
| strong | 4.01 to 4.49 |
| good | 3.61 to 3.99 |
| acceptable plus | 3.01 to 3.59 |
| adequate | 2.00 to 2.99 |
For UNDERGRADUATE PROGRAMS OVERALL:
| strong | 4.41 to 4.99 |
| good | 4.01 to 4.40 |
| acceptable plus | 3.51 to 3.99 |
| adequate | 3.01 to 3.50 |
| marginal | 2.01 to 2.99 |
For GRADUATE ENGINEERING PROGRAMS:
| very strong | 4.51 to 4.99 |
| strong | 4.01 to 4.49 |
| good | 3.61 to 3.99 |
| acceptable plus | 3.01 to 3.59 |
For GRADUATE PROGRAMS OVERALL:
| very strong | 4.51 to 4.99 |
| strong | 4.01 to 4.49 |
| good | 3.61 to 3.99 |
| acceptable plus | 3.01 to 3.59 |
| adequate | 2.51 to 2.99 |
| marginal | 2.01 to 2.49 |
