Evaluating rater trainings with double-pretest one-posttest designs: An analysis of testing effects and the moderating role of rater self-efficacy.
Klaus Moser University of Erlangen-Nürnberg Verena Kemter University of Erlangen-Nuremberg Kerstin Wachsmann University of Erlangen-Nuremberg Nora Z. Köver University of Erlangen-Nuremberg Roman Soucek University of Erlangen-Nürnberg
(December 2016) LASER Discussion Papers - Paper No. 1 (edited by A. Abele-Brehm, R.T. Riphahn, K. Moser and C. Schnabel)
Correspondence to: Prof. Dr. Klaus Moser, Lange Gasse 20, 90403 Nuremberg, Germany, Email:
[email protected].
Abstract Previous research on performance appraisal in Human Resource Management has found that frame-of-reference (FOR) training can improve rating accuracy. However, both the time-consuming development of training materials and the almost exclusive use of experimental evaluation designs limit the dissemination of FOR training as well as a better understanding of how and when it works. Therefore, unlike past research, the present studies used more general rater training materials and examined improvements in rater accuracy by means of a double-pretest one-posttest design. Study 1, using a student sample (the majority being part-time employees; N = 58), demonstrated the effectiveness of the rater training over and above a testing effect. Study 2 with participants from the workforce population (N = 45) replicated these results. In addition, in Study 2, pre-training rater self-efficacy moderated rater training effectiveness such that the rating accuracy of trainees with low self-efficacy improved more than did accuracy scores of trainees high in rater self-efficacy. We conclude that an effective FOR training can be conducted with less organization-specific materials, though both testing effects must be controlled for and individual differences (i.e. rater self-efficacy) between trainees have to be taken into account.
Zusammenfassung Bisherige Forschung hat gezeigt, dass Leistungsbeurteilungen durch Bezugsrahmentrainings deutlich verbessert werden können. Im Unterschied zu bisherigen Untersuchungen, die vor allem auf Unterschiede zwischen trainierten und untrainierten Beurteiler konzentrierten (= experimentelles Evaluationsdesign), analysiert die vorliegende Studie Verbesserungen der Akkuratheit auf der Grundlage des Vergleichs von Vortest- und Nachtestergebnissen innerhalb eines Doppel-Pretest-Ein-Posttest-Designs. Es fanden zwei Trainingssitzungen statt. Die Zweite Sitzung fand eine Woche nach der ertsten Sitzung statt und beinhaltete den zweiten Pretest, das eigentliche Training, und die Nachtesturteile. Die wahren Werte und die Prinzipien, die die Grundlage für das Training bildeten, beruhten auf Urteilen von vier Expertenteams. Zur Evaluation des Trainings wurden diverse Akkuratheitsmaße berechnet (Cronbachs 1955 Akkuratheitskomponenten, D-Index, Distance Accuracy). Erwartungsgemäß zeigten sich bei der Posttest-Akkuratheit besondere bei den wichtigsten Akkuratheitskomponenten (Cronbachs Differential Accuracy und Stereotype Accuracy), wobei sich überraschenderweise die Akkuratheit vom ersten zum zweiten Pretest zunächst verschlechterte.
Copyright statement Moser, K., Kemter, V., Wachsmann, K., Köver, N. Z., & Soucek, R. (in press). Evaluating rater trainings with double-pretest one-posttest designs: An analysis of testing effects and the moderating role of rater self-efficacy. The International Journal of Human Resource Management. doi:10.1080/09585192.2016.1254102 Author note Writing of this paper has been partially supported by a grant from the Deutsche Forschungsgemeinschaft to the second author.
