Angovian Methods for Standard Setting in Medical Education: Can They Ever Be Criterion Referenced?

Brian Chapman


This paper presents a discussion of Angovian methods of standard setting – methods which are widely used with the intent of defining criterion-referenced absolute standards for tests in medical education.  Most practitioners, although purporting to pursue absolute, criterion-referenced standards, have unwittingly slipped into focussing on norm-referenced concepts of ‘borderline’ students and their predicted ability to answer assessment items in a test.  This slippage has been facilitated by a shift in language from the original concept of ‘minimally acceptable’ persons to the modern concept of ‘borderline’ persons.

The inability of university academics to predict accurately the performance of ‘borderline’ graduate-entry medical students is illustrated by presentation of data obtained from three successive cohorts of a small regional medical school during the years 2010-2012.  Other data are presented to show how student performance, both ‘borderline’ and general, can be significantly altered by switching from didactic lectures to tutorials preceded by task-based active learning.

A protocol, based on a stricter interpretation of what is meant by a ‘minimally acceptable’ person, is suggested for moving towards a more criterion-referenced standard for a test based on the curriculum’s learning objectives.  Nonetheless, the fallibility of criterion-referenced standard-setting processes means that norm-referenced relative standards may need to be brought into play to deal with anomalous grade results should they arise.

The ideal of defining an absolute criterion-referenced standard for a test, using the most commonly implemented Angovian method, is probably as least as unattainable for graduate-entry medicine as it has been previously shown to be for secondary school science.


Angoff method, standard setting, assessmen, criterion referencing, norm referencing

Full Text:



Amin, Z., Chong, Y.S. and Khoo H.E. (2006), Practical Guide to Medical Student Assessment, World Scientific Publishing Co., London.

Angoff, W.H. (1971), ‘Scales, Norms, and Equivalent Scores’ in Educational Measurement, 2nd edn, ed. R. L. Thorndike, American Council on Education, Washington, USA, pp.508-600.

Bejar, I.I. (1983), ‘Subject matter experts’ assessment of item statistics, Applied Psychological Measurement, 7:303-310.

Cizek, G.J. and Bunch, M.B. (2007), Standard Setting: A Guide to Establishing and Evaluating Performance Standards for Tests. Thousand Oaks, CA: Sage Publications.

Ebel, R.L. (1972), Essentials of educational measurement, 2nd edn, Prentice-Hall, Englewood Cliffs, NJ.

Glaser, R. and Nitko, A.J. (1971), ‘Measurement in learning and instruction, in Educational Measurement, 2nd edn, ed. R. L. Thorndike, American Council on Education, Washington, USA, pp.625-670.

Glass, G.V. (1978), ‘Standards and criteria’, Journal of Educational Measurement, 15:237-261.

Hills, J.R. (1971), ‘Use of measurement in selection and placement’ in Educational Measurement, 2nd edn, ed. R. L. Thorndike, American Council on Education, Washington, USA, pp.680-732.

Impara, J.C. and Plake, B.S. (1997), ‘Standard Setting: An Alternative Approach’, Journal of Educational Measurement, 34:353-366.

Impara, J.C. and Plake, B.S. (1998), ‘Teachers’ Ability to Estimate Item Difficulty”: A Test of the Assumptions in the Angoff Standard Setting Method’, Journal of Educational Measurement, 34:353-366.

Jaeger, R.M. (1989), ‘Certification of student competence’, in Educational measurement, 3rd edn, ed. R.L. Linn, American Council on Education/Macmillan, New York, pp.485-514.

Jalili, M., Hejri, S.M. and Norcini, J.J. (2011), ‘Comparison of two methods of standard setting: the performance of the three-level Angoff method’, Medical Education, 45:1199-1208.

Kane, M.T. (1994), ‘Validating the performance standards associated with passing scores, Review of Educational Research, 64:425-461.

Lorge, I. and Kruglov, L.K. (1953), ‘The improvement of the estimates of test difficulty, Educational and Psychological Measurement, 13:34-46.

Mills, C.N. and Melican, G.J. (1988), ‘Estimating and adjusting cutoff scores: Features of selected methods’, Applied Measurement in Education, 1:261-275.

Nedelsky, L. (1954), ‘Absolute grading standards for objective tests’, Educational and Psychological Measurement, 14:3-19.

Norcini, J.J. (2003), ‘Setting standards on educational tests’, Medical Education, 37:464-469.

Shepard, L.A. (1994, October), Implications for standard setting of the NAE evaluation of NAEP achievement levels. Paper presented at the Joint Conference on Standard Setting for Large Scale Assessments, National Assessment Governing Board, National Center for Educational Statistics, Washington, D.C.

Williams, P. (2008), ‘Assessing context-based learning: not only rigorous but also relevant’, Assessment & Evaluation in Higher Education, 33:395-408.

Yudkowsky, R., Downing, S.M. and Popescu, M. ‘Setting standards for performance tests: a pilot study of a three-level Angoff method’, Academic Medicine, 83:S13-S16.

Zieky, M.J. (1995), ‘A historical perspective on setting standards. In Proceedings of the Joint Conference on Standard Setting for Large Scale Assessments. Washington, DC: National Assessment Governing Board and National Center for Educational Statistics, pp.1-38.

Zieky, M.J. and Livingston, S.A. (1977), Manual for setting standards on the Basic Skills Assessment tests, Educational Testing Service, Princeton, NJ.


  • There are currently no refbacks.

e-ISSN: 1694-2116

p-ISSN: 1694-2493