ARTICLE : EVALUATING PUBLIC SECTOR TRAINING PROGRAMS

This article discusses the importance of training program evaluation in increasing the effectiveness of agency training efforts. The article describes a framework available to the personalist to plan, design, and conduct and implement training program evaluations that more clearly identify the effectiveness of training programs.

The concept of training evaluation has received widespread recognition as beneficial, but the practice of evaluation has lagged behind (Bell and Kerr, 1987). Few reports of actual program evaluation have been published; compared to the number of training programs, few evaluations have been conducted. Lack of training program evaluation is even more evident in the public sector, and is possibly the least developed aspect of the training process in public agencies. Yet it is arguably the most important. Despite the hundreds of articles, books, and seminars devoted annually to the topic, training evaluation remains largely misunderstood, neglected, or misused. Too often, training is done without any thought of measuring and evaluating how well the training worked. Yet, the training process is not complete until and unless evaluation has taken place, for it is evaluation which informs training and gives it meaning.

If this is the case, why then is the evaluation of training in public agencies so underdeveloped? There are several reasons for this. Primary, among them, as Brethower and Rummler (1979) suggest, is that evaluation of training means different things to different people. There does not seem to be a consistent definition of what training evaluation is among personnelists. A second, more fundamental, reason, which persists even when there is consensus on the definition of training evaluation is the fact that the serious evaluation of training in public agencies is a difficult, tedious, time-consuming task which most personnelists would prefer not to engage in. A third reason for not conducting evaluations is that training administrators responsible for training simply tend to assume that training will work. A final reason is that a personnelist who champions a training program may feel threatened by the prospect of an objective evaluation of the program's effectiveness.

Recent research has shown that more than one-third of the members of the American Society of Training and Development responding to a survey reported that evaluation was the most difficult aspect of their jobs (Galagan, 1983). In another survey, Lusterman (1985) found that over two-fifths of the responding organizations reported significant changes when training effectiveness was evaluated. An even more recent survey of training directors found that 90% claimed that even though they believed the evaluation of training to be important, they did not conduct evaluations because their organizations did not require them (Bell and Kerr, 1987). Unfortunately, because of the perceived difficulties and the inability to identify specific outcomes as a result of training, post-training evaluation and follow-up often are ignored (Rao and Abraham, 1986).

To demonstrate training's importance, personnelists must not only present excellent programs but also must demonstrate that their programs get results: improved job performance, more efficient use of resources, and satisfactory returns on the training dollars invested. It is the contention of this paper that personnelists can prove the value of training when they systematically plan and implement program evaluation. Without a systematic plan a lack of emphasis on the determination of the worth of a training program can mean danger in training efforts in the long run. That is, failure to systematically evaluate training leaves open the potential for growth in training without accountability. This may lead to the continuation or even proliferation of ineffective programs or, in times of budget cutbacks, the perception by top administrators that training programs are superfluous and should be cut. Again, if personnelists are to eliminate the current roller coaster approach to agency support for training, systematic evaluation must become a part of every training program--whether or not key agency stakeholders require it.

Training Evaluation:. A Definition

Evaluation of training compares the post-training results to the objectives expected by administrators, trainers, and trainees (Mathis and Jackson, 1991). Hamblin (1970) defines evaluation of training as "any attempt to obtain information (feedback) on the effects of a training program, and to assess the value of the training in the light of that information". Put another way, evaluation tries to answer the question: did training work, and if not, why not? Hamblin further contends that the primary purpose of evaluation is to improve training by discovering which training processes are successful in achieving their objectives. Similarly, Swierczek and Carmichael (1985) identify the goals of evaluation as:

1. To improve the training program.

2. To provide feedback to program planners, managers, and participants.

3. To assess employee skill levels.

In addition, personnelists evaluate training for professional reasons because evaluation is one way in which they can assess their own effectiveness as trainers. From an administrative standpoint, personnelists evaluate in order to justify the time and money spent on training. The evaluation of training is, therefore, an integral part of the personnelist's "bag of tricks".

Given the diversity of agency training needs, there is no single method most appropriate for evaluating training efforts. The circumstances dictating the need for training, different methods used in training, and the different purposes for evaluation all make plain the need for multiple approaches in training evaluation. Regardless of need, method, or purpose the personnelist must carry, out a systematical identification and organization of important factors related to planning and executing the training evaluation process.

Having provided a definition of training evaluation, which will be revisited later in this paper, it is important to discuss in more detail training evaluation objectives benefits.

Training Evaluation Objectives and Benefits

The primary and overriding objectives of the evaluation of agency training programs should be to collect data that will serve as a valid basis for improving the training system and maintaining quality control over its components. It must be emphasized that all components of the system and their interaction are the objects of scrutiny and that personnelists should ensure that training programs are designed with a prior consideration given to evaluation. That is, public sector trainers should be committed to evaluating the effectiveness of their programs. Several potential benefits result from evaluating agency training programs:

1. Improved accountability and cost effectiveness for training programs which might result in an increase in resources;

2. Improved effectiveness (Are programs producing the results which they were intended?);

3. Improved efficiency (Are the programs producing the results for which they were intended with a minimum waste of resources?);

4. Greater credibility for the personalists to include information on how to do a better job now or in future programs or to redesign current or future programs;

5. Stronger commitment to and understanding of training by key administrators so they can make up for deficiencies and confirm/disconfirm subjective feelings about the quality of agency training;

6. Formal corrective feedback system for developing strengths and weaknesses of training participants. Trainees that understand the experience more fully and are more committed to the program;

7. Managers better able to determine whether to send potential recruits to future training programs;

8. Quantifiable data for agency researchers and training program developers interested in training research;

9. Increased visibility and influence for public sector training program sponsors;

10. Increased knowledge and expertise in the development and implementation of training programs that produce the results for which they were intended.

This is not an exhaustive list of the objectives and benefits of a training program evaluation, however, personnelists who are responsible for training must continually ask themselves what are the objectives of evaluation; and what do they want to gain by conducting an evaluation?

A priori consideration of evaluation gives the personnelist at least five important advantages:

1. The ability to identify relevant audiences, interested in training evaluation, early in the process to ensure that evaluation feedback addresses their interests and information needs.

2. The development of an evaluation process that compliments the training program. Evaluative methods can be carefully incorporated to minimize any disruptive effects on the training program.

3. The ability to construct a research design that allows for valid conclusions about the program's effectiveness. This includes finding appropriate pre-measures, selecting appropriate groups or individuals to train, identifying comparison groups, and isolating extraneous variables prior to beginning training.

4. The ability to delineate material, data, and human resource requirements for evaluation and incorporating these as part of the training program, not simply as an appendix to the training program.

5. The ability to modify the training program based on feedback gained through ongoing evaluation. Corrective feedback is crucial when modifying or upgrading subsequent stages of the training program.

Thus, personnelists committed to evaluation can enjoy benefits and advantages that have long been sacrificed in training designs without evaluation. Determination of the audiences of the evaluation can improve the likelihood of not falling into one of the many pitfalls which can effect the potential success of a program evaluation. In addition, although it may be impossible to design the "perfect" evaluation, major errors can also be avoided. The next section presents some of the evaluation pitfalls or mistakes that personnelists must be aware of in evaluating training efforts.

Pitfalls in Training Evaluation

Too often, training program evaluations have failed. Mainly these failures can be attributed to inadequate planning or designing, lack of objectivity, evaluation errors of one sort or another, improper interpretation of results and inappropriate use of results. Poor systems of training program evaluation produce anxiety, resentment, budget reductions, and efforts to sabotage the program. But what is of even greater importance, poor training evaluation programs do not provide firm data for improving and controlling the quality of the training system. Following are some common pitfalls or mistakes in training program evaluation (Tracey, 1971, 1984; Russ-Eft and Zenger, 1985; and Sims, 1990). Some of these pitfalls or mistakes can be easily overcome through good planning, while others are more difficult. However, personnelists must at least recognize the problems that occur when such mistakes or pitfalls occur.

Poor planning. To be effective, a training program evaluation must be carefully planned. Some of the common deficiencies in planning are these:

1. Failure to work out the details of the program, failure to include data-collection instruments, specific procedures to be followed, and the scheduling of surveys, interviews and observations.

2. Failure to train evaluators in the principles and techniques of evaluation, which includes the use of data-gathering instruments.

3. Failure to make clear to all concerned the purposes of the evaluation program and the uses to be made of evaluations and recommendations.

Lack of objectivity. Although it is impossible to guarantee that training program evaluations will be completely objective, there are some steps that can be taken to make certain they will be more objective.

1. Select evaluators who are capable of making objective judgments.

2. Train evaluators.

3. Design appropriate data-gathering instruments.

4. Look at all the components of the training situation as an integrated system.

5. Focus on important details -- avoid "nit-picking."

Rate terrors. When scales are used to evaluate quality of performance or materials, observers often differ in their ratings. These differences are rater errors, although this may not be the most accurate term to use for all these disparities. Some errors are caused by faults in the design of rating instruments; others, by the raters. Some errors occur only with certain groups of observers; and some occur only with individual observers. Other errors occur only when certain behaviours of individuals are rated. Some observers make errors when rating all persons; some when rating certain groups; and others when rating certain individuals. Some of the more typical rating error categories are: central tendency, halo effect, and recency.

Improper interpretation of data. The collection of data on the training program is one thing; interpreting that data is quite another. Here, the meaning and impact of the data are judged. if this step is not handled properly, the value of the information collected will be completely lost. Results from any evaluation must be interpreted cautiously, recognizing the extraneous variables that may have affected the findings. This is particularly true for those personnelists claiming to have identified productivity improvements resulting from a training program. Here are some of the main pitfalls in interpretation of data from training programs:

1. Assuming that consensus among one category of data on a single training system element guarantees a valid and accurate judgment.

2. Concluding that an observation or judgment made by only one observer or group of trainees for example, is inaccurate or invalid.

3. Taking comments or responses to open-ended questions at face value, and not considering the nuances of language and the problems of semantics.

4. Failing to take into consideration the perspective of the individuals providing the data.

Not reporting evaluation results in terms that are meaningful to the intended audience. Training evaluations in government agencies often yield results that are of little value to decision makers. This problem results because the evaluation collects the wrong information, uses technical jargon to describe the results or presents the results after critical decisions have been made. Evaluations of training programs conducted within agencies must focus on the direct outcomes of that training--behavior change. Personalists must realize that the basic aim of any evaluation should be to ensure that relevant information is made available to decision makers (the audience) at proper times and in appropriate forms. By doing so, evaluation findings and generalizations may influence future decisions and policies.

Overgeneralization of findings. A problem related to the previous one is generalizing the findings of an evaluation in one agency to what might be expected in other agencies. Only by conducting repeated evaluations in many different agencies at many different locations can an accurate picture emerge. "Meta-analysis" (Glass, 1976) provides one means of examining different organizations. Meta-analysis is merely a grand term to describe the conduct of a summary analysis of a large collection of previous studies. Although the personalist may need to be rigorous in their selection of high-quality studies and cautious in their interpretation of the results, such summaries can prove useful in identifying important trends in their evaluations.

Inappropriate use of evaluation results. When tabulated, data collected during the training program evaluation have the aura of complete objectivity and truth. Sometimes the results of evaluation are used for purposes other than that originally intended. This is a major error. Some of the inappropriate uses to which evaluative data have been put are as follows:

1. Using data and reports on a single part of a training program to make decisions on the whole program.

2. Using data and reports designed for evaluating the whole training program as a basis for denying or granting funding for future training programs.

3. Using otherwise unsupported and invalidated data as a basis for causing significant changes to a training program or system to be made.

The personalist should be aware of potential evaluation mistakes or pitfalls and keep in mind four general steps when organizing evaluations or new or existing training programs. First, determine why evaluation is being conducted. Second, identify resources needed to conduct the task. Third, design the evaluation process with particular emphasis on the role of the personalists and other training participants. And-fourth, implement evaluation even though this step most certainly will not always be smooth and efficient. Within this general framework, the evaluation of training by the personalists can be thought of as being planned and executed at three separate, interacting levels adapted from Nicholas (1977) work in organization development and Sims (1990) in training: the training program level (TPL), the training session or component level (TSL), and the micro-training level (MTL)igure 1).

At the program level, planning involves establishing the broad strategy of the training program evaluation based on overall program goals. Personnelists and key agency decision makers work together to define the intent of training evaluation based upon perceived needs for the program or as determined by the personnelists in a formal needs analysis (diagnosis).

The training program evaluation plan requires specification, even if loosely, of the individual training sessions that will comprise it. This is necessary so that those responsible for implementing the evaluation program can establish a timetable of activities. The training evaluation materials must be obtained, and participants must roughly know how to schedule their time for the program evaluation. For each training component or session, evaluation objectives should be established to indicate how it will contribute to the overall training program evaluation. Once the individual training components have been loosely planned, it should be clear from statements of objectives of each session how the goals of the evaluation will be achieved.

During each component of the training program, the personnelist uses one or more "micro training levels." A micro training design level is a set of structured training activities used to achieve some training goal The personnelist combines micro training designs such as those accepted in the training field and their own original designs to form sequences of activities to achieve program objectives. Each MTL has objectives which are compatible with objectives at the TSL and are the operants through which objectives at the TSL are achieved. Selection of MTLs and accompanying evaluations depend on the purposes they are to serve, and on the characteristics and constraints of the situation such as time limitations, costs, number of participants, and the level of training evaluation.

The considerations for developing evaluations of training programs should include some formal diagnosis of an agency's training needs and the training program that will be evaluated and are summarized in Figure 2. This diagnosis should specify training program evaluation objectives, evaluation criteria, and resources and constraints that will be encountered in planning and implementing the evaluation process. Wherever evaluation is called for, the personnelist must ensure that objectives are made in terms of clear statements that provide some amount of measurability. Criteria are those specific measures that establish whether or not objectives are met. Resources and constraints include not only money, personnel, equipment, time and space, but also, attitudes, norms and values of the agency toward training. From the possible training evaluation techniques, techniques are selected that will most likely achieve program objectives within given constraints and existing resources. This is a "systems analysis" approach applied to developing a training program evaluation plan. The result is an action plan indicating roughly the objectives and procedures at all three training evaluation levels (program, session, and micro).

For training program evaluations that involve multiple components, such as indicated in Figure 1, specification of evaluation procedures for later sessions will most likely initially be vague; these are developed into more detail by the personnelist as their time of implementation draws closer. The important point is that thought is given by the personnelist as to how the micro designs and training sessions tie together to meet the overall objectives of the program. The pyramiding of objectives enables the personnelist to test assumptions concerning the ability of procedures at lower levels to meet objectives at higher levels (i.e., can the MTLs and TSLs evaluation activities reasonably be expected to achieve overall program objectives).

The Design of Training Program Evaluations

There is a need for more carefully designed training program evaluations and personnelists should be concerned with developing two major components of a training program. First, the training content or activities to be included in the program should be identified. Second, an outline or program for training evaluation should be developed. Simply put, the training program evaluation plan is created by (1) defining the agency's training needs, (2) deciding what has to be evaluated, (3) developing the training program with objectives and criteria clearly laid out to enable evaluation, and (4) developing an evaluation plan based on the objectives, criteria, and activities of the training program. Figure 3 presents a model for designing a training program with an accompanying evaluation program. The process on the right is the development of the evaluation part of the plan. As in the design of the action plan, the design of the evaluation part of the training evaluation is a systems analysis approach with antecedents of training program objectives, criteria, resources and constraints, and training evaluation techniques.

The left side presents steps necessary for designing the evaluation plan. Since the focus of this paper is training evaluation, the right side of the model is of immediate concern. For interested readers, training content development has been discussed more explicitly by others (see McGehee and Thayer, 1961; Goldstein, 1974; 1986, and 1989; Wexley and Latham, 1981).

Purpose of Training Evaluation

The first step in planning training evaluation is determining the purpose for evaluating the program; what do you want to know about the training program? Each kind of question necessitates consideration of how the evaluation program should be designed to provide answers. Stuffie-beam et al. (1971) discuss three purposes for evaluation relevant to training program evaluation. First, evaluation can be used to identify differences in behavior. Individuals or groups may be compared to other individuals or groups, to ideals or standards (as in performance appraisal), or with themselves at different moments in time (a time-series study). This is a comparative evaluation. Evaluation can also investigate causes of behavior. The variables, within a training program, responsible for changes within individuals or groups can be identified through experimental manipulation. This is explanatory evaluation. Finally, a predictive analysis can evaluate how well training performance data correlate with other variables such as changes in individual job performance and/or system performance. The purpose for training evaluation will impact each step in developing both the content and evaluation components of the training program.

Criteria for Training Evaluation

The criteria used in a training evaluation program will depend on its purposes. Whatever the purposes of the evaluation, the personnelist can make use of five general categories, adapted from Suchman (1967):

1. Evaluation of effort assesses input, regardless of output. The questions "What did you do?" and "How well did you do it?" are addressed.

2. Evaluation of performance focuses on the results of the program. This requires clear statements of objectives; it asks "Did any change occur?" and "Were objectives achieved?"

3. Evaluation of adequacy determines how effective the program was. For example, trainees in a program may exhibit considerable practical application change, but the number of trainees may be inadequate in determining the benefits of the program for the agency.

4. Evaluation of efficiency is concerned with alternate ways of achieving the same ends that are more efficient in terms of time, money, human resources, materials, and convenience.

5. Evaluation of process focuses on the operation of a training program, with emphasis on how and why it works or does not work.

Besides Suchman, numerous criteria have been proposed (Schuler and Huber, 1990). The many options include changes in productivity, reported attitude on the job (e.g., satisfaction with supervisor, job satisfaction, stress, role conflict), changes in job knowledge, cost savings, benefit gains, and attitudes toward training (Burke and Day, 1986a and 1986b; Fisher and Weinberg, 1988; Fitzenz, 1988; Ford and Wroten, 1984; Kaman and Mohr, 1984; Kfipatrick, 1983; and Russ-Eft and Zenger, 1985). While several criteria have been tested through the years, at least four components commonly have been included in training evaluation: reaction to training, learning changes, transfer of training, results (Kilpatrick, 1983).

Employee reactions. Employee reactions to training are evaluated by conducting interviews or administering questionnaires to trainees. Here, personnelists are interested whether trainees liked the program, whether they thought the instruction was clear and helpful, and/or whether they believe that they learned the material. Even though positive employee reactions are necessary for training to be successful, positive employee reactions do not necessarily mean that training will lead to changes in knowledge or performance (Alliger and Janak, 1989).

In fact, trainee reactions constitute the lowest level of training evaluation (Birnbrauer, 1987) and often can be misleading. Although reaction information can be useful for fine-tuning some aspects of the training program such as scheduling, mode of teaching, etc., data collected on the following three criteria generally provide stronger information for evaluating the program.

Employee learning. Instead of using employee reactions as the criterion in evaluating training performance, actual employee learning can usually be measured (Bell and Kerr, 1987). Employee learning criteria are used when personnelists wish to determine how well trainees acquire knowledge, skills, or abilities (KSAs) taught in training. Tests on the training material are commonly used for evaluating learning and can be given both before and after training to compare scores. Of course, learning enough to pass a test does not guarantee that the trainee can do anything with what was learned. However, learning tests should be considered since they are objective, reliable, and easy-to-administer tools for evaluating training effectiveness.

Application or transfer of training. Transfer of training concerns whether behavioral or performance changes taught in training are expressed on the job. Can trainees now do things they could not before (e.g., negotiate, conduct an appraisal interview)? Do they demonstrate new behaviors on the job? Has their performance improved? Data useful for evaluating training transfer can be obtained through interviews of trainees and their co-workers and observations of job performance. Transfer of training is perhaps the most critical goal of training. Since training represents a step toward improving job performance and/or organizational effectiveness, a lack of transfer in KSAs to the job may indicate a poor training effort.

Results or bottom line measures. Results are evaluated by measuring the effect of training on the achievement of agency objectives. For example, "Did the agency actually save money following the training program?" or "How well were the training dollars invested?"

The difficulty with this measurement is pinpointing whether training caused the changes in results. Other factors may have had a major impact as well. Correlation does not imply causation. Still, the American Society for Training and Development (1988) notes that training must be treated as an investment, with similar potential payoffs as other organizational efforts. Cascio (1989) describes a training utility model that considers both the expected payoffs from training and an organization's capital investment decisions. He also provides computational formulas for examining important factors contributing to the cost-benefit ratio. Interested personnelists may do well to begin exploring training utility issues in Cascio's (1989) chapter.

Training Program Evaluation Resources and Constraints

When any training evaluation is undertaken a careful analysis of available resources and possible constraints should be included. Resources are the material and personnel needed for developing the training evaluation plan and subsequent implementation. Constraints are limitations or restrictions on what can be included in the evaluation plan or implementation.

Both are considerations in selecting among various methodologies and procedures to make up the evaluation plan. For example, time is a constraint for most training evaluations. The time available for collecting, analyzing data, and reporting results is limited. This influences not only procedures for data collection and analysis, but also the type of data collected. Time may not permit development of tailored survey instruments for collecting reaction data and personnelists may turn to alternatives such as off-the-shelf questionnaires. Such a decision may reduce evaluation time, however, the evaluator may be constrained to measure what the instrument purports to measure, rather than variables of interest. Some of the resources/constraints to training program evaluation that must be considered include:

1. Funding--the dollars allotted to cover training evaluation planning and implementation.

2. Time--limits imposed in developing and executing the evaluation. Time limits may be thought of as a sequence of "milestones," such as completion of pretest and post-test data collection (for example, testing knowledge or skill usually by paper-and-pencil), completion of data analysis, dissemination of results to appropriate audiences, etc.

3. Human resources--trained personnel such as statisticians, computer specialists, research methodologists, and other personnelists.

4. Organizational climate--the trust and openness of administrators, employees, or trainees in providing and receiving evaluative feedback information.

5. Availability of data--availability and quality of organizational information such as records of individual, group, department and organization performance, reports, personnel records; availability of employees for providing new data through surveys, interviews and observation.

6. Details of the training evaluation action plan--objectives, timetable, procedures, participants, location; possible use of strategies which overlap evaluation strategies, such as survey feedback.

7. Audiences--kind and number of key players interested in the evaluation; information needs and interests.

8. Technical ability and feasibility--availability and feasibility of using standardized instruments, computerized analysis and storage of data; logistics in collecting and disseminating results; competencies and abilities of persons involved.

9. Ethical concerns--privacy considerations, employee and organizational confidentiality, obtrusiveness, or harmful aspects of data collection and reporting.

To a large extent, these are interdependent factors to which the personnelist must attend during the training program planning analysis.

Training Program Evaluation Design Options

Upon identifying important training outcome criteria, personnelists must select an evaluation design to measure changes in these variables. There are many ways to design and evaluate training programs to determine their effects. The three most common are shown in Figure 4. The level of rigor of the designs increases from Post, Pre/Post, to Pre/Post Measure With a Control Group (Mathis and Jackson, 1991).

Designing a good evaluation effort involves knowing when to collect evaluation measures and which groups to collect them from. Together, these factors determine the experimental design used to assess the impact of training. More specifically, the training evaluation design refers to the steps taken to ensure:

a. that a change occurred (e.g., employee productivity increased, accidents declined, etc.);

b. the change can be attributed to the training program; and

c. that a similar change could be expected if the training were done again with other employees.

Of course, the ability to make such statements will depend on the experimental rigor incorporated in the training evaluation process. Conclusive statements about the effectiveness of training can be made only when the personnelist strictly adheres to experimental principles such as manipulation of variables, random assignment, control of extraneous and/or confounding variables, and equivalence of groups. Unfortunately, conducting experimentation in the field has proven to be a difficult, almost overwhelming, task. Many agencies generally demand that all employees in a department be trained, not only those randomly selected. It is also difficult for personnelists to control the many variables that can affect a worker's job behavior at a given time (e.g., interaction with co-workers, supervisor, personal relationships, promotions, etc.). However, previous training evaluations have been able to overcome these difficulties by using several highly effective designs for evaluating training (See Cook and Campbell, 1979; Kilpatrick, 1983; Smith and George, 1984; Wehrenberg, 1983; Hoffman, 1984 for more detailed discussions of the available designs depicted in Figure 4).

Regardless of the design selected, personnelists must adhere to certain basic experimental principles. First, when possible, both pretest and posttest data should be collected on relevant criteria. Second, selection of participants should be randomized when possible. If it is not possible, solid quasi-experimentation should be employed. Third, reliability of data collected should be monitored throughout the evaluation. Fourth, when statistical analyses are performed, characteristics of the sample and data should conform to the assumptions of tests used. Finally, the evaluation process (i.e., training development, data collection and implementation) should be conducted in a manner to allow valid inferences about the effectiveness of training,

The final success of the training program evaluation depends on how well the personnelist can overcome problems through a methodologically sound design and implementation scheme. The evaluation plan should be developed through selection of alternatives assessed against the objectives of the evaluation and existing constraints and resources. A familiarity with available resources, imposing constraints, and methodological alternatives will allow accurate, useful, and practical training program evaluations.

Multiple Levels of Training Program Evaluation

Thus far, this paper has described strategies for evaluating the overall or general outcomes of training. For purposes of discussion in this section training evaluation can be delineated at three levels: micro training evaluation (MTE), training session evaluation (TSE), and training program evaluation (TPE). MTE focuses on smaller aspects of the training program such as timing and scheduling of training activities, presentation of training information, the outcomes of individual exercises, and etc. For most training efforts, a rigorous MTE will not be conducted, since simple observation may be all that is needed to determine the effectiveness of such aspects of the program. However, factors such as scheduling and presentation method can greatly affect the other levels at which training evaluation can occur.

TSE focuses on the quality, processes, and outcomes of individual training components or sessions. Here, the personnelist wants to ensure that sessions have desired effects. Data collection can be as informal as interviewing trainees to assess their perceptions of the session, or as formal as testing trainees and interpreting scores to determine what was learned. Criteria such as learning and reaction are appropriately measured at this stage of the program. Evaluation at the training session level can provide valuable information useful for: 1) identifying areas where trainees may require additional training and/or 2) modifying existing training practices. Modifying the training based on employee reaction measures is fairly straightforward. A boring speaker can be replaced or a video rated as irrelevant dropped, depending on the feedback received.

At the training program evaluation level the criteria discussed earlier (employee learning, application of training, results, and employee reaction) are evaluated in relation to the entire agency. The methods and strategies described earlier would apply. Where MTE and TSE may be conducted in a matter of hours or days, TPE can continue for months or years following the training sessions to determine these effects. At this level, where evidence of outcomes or results is more mediated and distal than at lower levels, training evaluation is more difficult to carry out and usually is more complex in design.

The three levels of training evaluation are represented in Figure 5. This guide can serve as a "mental checklist" for personnelists when developing training programs to better understand the various components of the plan and how they relate to one another. A training program's objectives, criteria, procedures, and activities (methods) can be entered into columns 1 and 2. Column 1 contains the objectives of the training program and column 2 contains the methods that will be used to obtain training objectives. When evaluation is desired, the objectives and criteria for evaluation should be entered into column 3. Column 4 will contain the evaluation techniques used to accomplish the column 3 objectives. Column 4 includes the methodology and techniques for data collection, analysis, and reporting.

Personnelists should also derive expected outcomes of the program in terms of the objectives and criteria in columns 1 and 3. Specifying expectations is useful for two reasons: 1) it forces personnelists to judge whether the intended training program outcomes can reasonably be expected, and 2) it forces them to assess whether the training evaluation plan measures achievement of desired results. The final training program evaluation plan should enable personnelists to visualize the total training program process and anticipate further problems or design considerations. Thinking about expected outcomes also forces personnelists responsible for training to closely monitor steps in the training process to determine whether results are occurring and outcomes are feasible.

In conclusion, an agency training program should address the following questions to ensure adequate evaluation of training outcomes:

1. Does the evaluation design fit the objectives of the training program?

2. Does the design address important issues such as trainee needs and expectations, learning style, and trainee culture (expectations about authority, how hard to work, etc.)?

3. Does the evaluation method meet standards discussed by public sector trainers?

4. Does the structure provide a framework where emergent issues can be addressed? Can the design be modified to address trainees' felt needs without sacrificing objectives?

5. Can the design be carried out in the time allotted?

6. Does the design provide a mix of activities that appeal to different learning styles such as listening, discussing, interacting?

7. Is the material logically and psychologically sequenced?

8. Is there redundancy in information presented in training? Should there be?

9. Does the evaluation design allow for on-going development of a learning climate?

Summary

Carefully designed and planned evaluations are keys to better agency training. However, one problem with many agency training programs is that they are not designed with a priori consideration given to evaluation. Evaluation should be an integral part of training and training program objectives and procedures should be compatible with evaluation purposes and procedures. In addition, personnelists need to realize that evaluations of training programs are conducted to answer certain questions, and that there may be different questions at different times in the history of a program. The information needs and interests of key agency members and interested parties should be identified early, rendering training program evaluation results credible.

Today, there is an increasing recognition that evaluation is critical to training. The "systems" approach to planning, designing, and implementing a training evaluation presented in this paper should improve the personnelists ability to conduct evaluation efforts. The personnelist must develop an orderly, goal-oriented, and systematic approach to training evaluation that considers a range of options useful for overcoming the resource shortages and organizational constraints often confronted in agency training.

In any training program instance the personnelist must first determine that it (training program) does have some beneficial effects. Several questions that should be answered include: What outcomes does it really have? How long-lasting are these effects? Which people and behaviors change? The final series of questions to be asked focuses on improving the product or program. They include: How can the delivery be improved? What improvements can be made in the content or the schedule? Answers to these questions can lead to improvements in the training program. These questions can only be answered by carefully planned, designed, and implemented evaluations. Thus, agency training programs will only progress if personnelists do the appropriate research and do it right.

While the problems facing personnelists in training evaluation allow for no easy solutions, evaluations can nonetheless be rewarding. Basically, all parties gain through evaluating training--the agencies, programs, the training departments and the personnelists. Such assessments and evaluations yield important information, but only when care is taken in planning, designing, and conducting the evaluation. As Brooks (1977) writes, "if proper care is given to the manner in which findings are obtained and interpreted, the positive contributions [of the evaluator] will far outweigh the negative implications deriving from the imperfection of his tools" (p. 62). By continually measuring the effects of training, improving programs based on evaluation data, and keeping decision makers informed of and involved in the evaluation process, the full value of evaluation as a potentially powerful organizational tool can be realized by agencies.

a. Evaluation objectives and criteria are specified only for those levels, sessions, and micro-designs where evaluation is to be done.

b. Evaluation procedures are specified only for those levels, training sessions, and micro-training designs where evaluation is to be done.

c. At the training program level, procedures roughly outline what is to take place at the session level: number of sessions, trainable, etc.

d. At the training session level, procedures roughly outline what is to take place during each session: kinds of micro-training designs, participants, etc.

e. At the micro-training design level, procedures define in detail step-by-step procedures, activities and timing for each.

Adapted from "A systems analysis approach for planning evaluations of OD Interventions" by John M. Nicholas, 1977, pg. 358-362, Academy of Management Proceedings.

DIAGRAM: Figure 1: Interacting Levels of Training Evaluation

DIAGRAM: Figure 2: Considerations in Developing a Training Evaluation Program Plan

DIAGRAM: Figure 3: Designing a Training Program Evaluation

DIAGRAM: Figure 4: Training Evaluation Designs

References

Alliger, G. M., and Janak, E..(1989). Kilpatrick's levels of training criteria: Thirty years later. Personnel Psychology, 42, 331-342.

American Society for Training and Development. (1988). Gaining the competitive edge. Alexandria, VA: American Society for Training and Development.

Bell, J.D., and Kerr, D.L. (1987). Measuring training results: Key to managerial commitment. Training and Development Journal, 41(1), 70-73.

Birnbrauer, H. (1987). Evaluation techniques that work. Training and Development Journal, 41(1), 53-55.

Brethower, S., and Rummler, G.A., (1979, May). Evaluating training. Training and Development Journal 14-22.

Brooks, M. (1971). The community action program as a setting for applied research. In F. Caro (Ed.) Readings in evaluation research (pp. 53-62). New York: Russell Sage Foundation.

Burke, M. J., and Day, R. R. (1986a). A cumulative study of the effectiveness of managerial training. Journal of Applied Psychology, 71, 232-45.

Burke, M. J., and Day, R. R. (1986b, August 21). Cost-effective training techniques. Bulletin to Management, p. 284.

Cascio, W. F. (1989). Using utility analysis to assess training outcomes. In I. L. Goldstein (Ed.), Training and development in organizations (pp. 63-88). San Francisco, CA: Jossey-Bass Publishers.

Cook, T. D., and Campbell, D. T. (1979). Quasi-Experimentation: Design & analysis issues for field settings. Boston, MA: Houghton Mifflin Company.

Fisher, H. E., and Weinberg, R. (1988, January). Make training accountable: Assess its impact. Personnel Journal, 67, 73-75.

Fitzens, J. (1988, March). Proving the value of training. Personnel, 65, 17-23.

Ford, J. K., and Wroten, S. P. (1984, Winter). Introducing new methods for conducting training evaluation and for linking training evaluation to program redesign. Personnel Psychology, 651-66.

Galagan, P. (1983). The numbers game: Putting value on HRD. Training and Development Journal, 37 (8), 48-51.

Glass, G.V. (1976). Primary, secondary, and meta-analysis research. Educational Researcher, 5 (10), 3-8.

Goldstein, I. L. (1974). Training: Program development and evaluation. Monterey, CA: Brooks/Cole.

Goldstein, I. L. (1986). Training in organizations: Needs assessment, development, and evaluation (2nd ed.). Monterey, CA: Brooks/Cole.

Goldstein, I. L. (Ed.) (1989). Training and development in organizations. San Francisco, CA: Jossey-Bass Publishers.

Hamlin, A.C. (1970, November). Evaluatiotraining. Industrial Training International, 33.

Hoffman, F.O. (1984, February). A responsive training department cuts costs. Personnel Journal, 63, 48-53.

Kaman, V. S., and Mohr, J. D. (1984, October). Training needs assessment in the eighties: Five guideposts. Personnel Administrator, 29, 47-53.

Kilpatrick, D. L. (1983, November). Four steps to measuring training effectiveness. Personnel Administrator, 28, 19-25.

Lee, C. (1989, October). 1989 Industry Report. Training, 35, 49-64.

Lusterman, S. (1985). Trends in corporate education and training. The Conference Board, Report No. 870.

McGehee, W., and Thayer, P. W. (1961). Training in business and industry. New York: Wiley.

Mathis, R. L. and Jackson, J. H. (1991). Personnel/Human Resource Management. St. Paul, MN.: West Publishing Co.

Nicholas, J.M. (1977). A systems analysis approach for planning evaluations of OD interventions. Academy of Management Proceedings, 358-362.

Rao, T.V., and Abraham, E. (1986). Human resource development. Management Labour Studies, 2, 73-85.

Russ-Eft, F., and Zenger, J. H. (1985, April). Common mistakes in evaluating training effectiveness. Personnel Administrator, 30, 57-62.

Schuler, R. S., and Huber, V. L. (1990). Personnel and human resource management (4th ed.). West Publishing Company. St. Paul, MN.

Sims, R.R. (1990). An experiential learning approach to employee training systems. Westport, CT.: Quorum/Greenwood Press.

Smith, H. W., and George, C. E. (1984, August). Evaluating internal advanced management programs. Personnel Administrator, 29, 118-31.

Stufflebeam, D., Foley, W., Gepart, W., Guba, E., Hammond, R,, Merriman, H., and Provus, M. (1971). Educational evaluation and decision making. Itasca, IL: Peacock Publishers.

Suchman, E. (1967). evaluative research. New York: Russell Sage Foundation.

Swiercaek, F. W. and Carmichael, L. (1985, January). The quantity and quality of evaluating training. Training and Development Journal, 95-99.

Tracey, W.R. (1984). Designing training and development systems. Washington, D.C.: American Management Association.

Tracey, W.R. (1971). Designing training and development systems. Washington, D.C.: American Management Association.

Wehrenberg, S. B. (1983). Evaluation of training: Part I. Personnel Journal, 62, 608-10.

Wehrenberg, S. B. (1983). Evaluation of training: Part II. Personnel Journal, 62, 698-702.

Wexley, K.N., and Latham, G. P. (1981). Developing and training human resources in organizations. Glenview, IL: Scott, Foresman.

~~~~~~~~

By Ronald R. Sims

Ronald R. Sims is Associate Professor of Business Admnistration at the College of William and Mary. His reserach interests are primarily in the area of training and development and the application of experimental learning to school and work settings.

---

Copyright of Public Personnel Management is the property of International Personnel Management Association-USA and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Source: Public Personnel Management, Winter93, Vol. 22 Issue 4, p591, 25p, 1 chart, 4 diagrams.
Item Number: 9409080662