Comm Eye Health Vol. 14 No. 40 2001 pp 65 - 66. Published online 01 December 2001.

Evaluation of training

Detlef Prozesky MBChB MCommH PhD

Professor, Community Based Education, Faculty of Medicine, University of Pretoria, PO Box 667, Pretoria 0001, South Africa

Related content

‘Quality assurance’ and ‘appraisal’ are words we often see nowadays. Training institutions in many countries are trying to improve the quality of the training they provide. This is not new – good teachers have always looked critically at the way they teach, in order to do it better.

What is ‘evaluation’? Broadly, it means looking carefully at something that we are worried about, and then making a judgement about it. We usually do this because we want to improve the thing we are looking at. The following diagram shows this process more clearly:

We call this the ‘evaluation cycle’. Once you have seen if your plan of improvement is working you identify a new problem in your teaching, and evaluate that again.

This is clearly a kind of research. Perhaps you feel: ‘I’m not an evaluator! I can’t do this kind of research!’ Another way of looking at evaluation is that you use your common sense to judge what you (and other teachers) do in your work. If you think back you will see that you have often done it – but this article should help you to do it more systematically. Also, you don’t have to evaluate everything at once. You can choose a small part of your work which seems to be giving particular trouble, and start by looking at that. What might this be?

Evaluation cycle
Evaluation cycle

Things teachers evaluate

The curriculum: what we teach

Curricula get out of date easily, and people just go on year after year teaching the same material.

Here are some areas you might want to look at:

  • The overall curriculum: is it complete? Does it contain all the knowledge and skills that the students are going to need to perform their job?
  • The overall curriculum: is it overloaded? Are you teaching a lot of ‘nice to know’ and ‘nice to do’ material, instead of concentrating on the ‘must know’ and ‘must be able to do’ material?
  • The content of individual lessons: do they contain what the curriculum planners intended them to? Do they emphasise priorities, and leave out the rest? Does the teacher present the material in a sequence which helps students to understand it more easily?

The lesson process: how we teach

  • The teaching methods: are they appropriate for the domain of the material you are teaching? Do students learn skills by seeing a demonstration, and then practising the skill personally?
  • How well do teachers use these methods? Are lectures well prepared and skilfully delivered, and do they interest and involve the students? Are practicals well organised, with checklists, and do all students get feedback about their performance?
  • About the teaching aids that are used in class: is their quality good? Are they well used – do they help the learning process?
  • About the handouts and written documents that are used: do they focus on priorities? Are they clearly written, using simple language? Are they suitably illustrated?

The assessment: how we test our students

  • Is the assessment valid. Is it suitable for the ‘domain’ of the subject matter (for example, do we assess skills by observing students perform them)? Does it mostly contain the ‘must know’ and ‘must be able to do’ material? Does it cover most of the important topics?
  • Is the assessment reliable? Are there good marking schedules and checklists, to guide the examiners so that they give fair marks?
  • What is the ‘assessment curriculum’? – in other words, does the assessment make students learn those things which we consider to be the priorities?

These are just some of the possibilities. Of course, you will decide from your situation what you should be looking at.

Instruments to collect information for evaluation

Once we have identified a problem we need to collect more information about it. How do we do this? There are a number of ‘instruments’ that we commonly use, to collect data for evaluation:

Document study

Here we examine written curricula, timetables, lesson plans, visual aids, handouts, exam papers and so on. We compare them to a standard that we have set beforehand. This can be done in an unstructured way (by reading them and gaining an overall impression), or more structured (by making a checklist beforehand, of things we are looking for in the document). One special kind of document study is the ‘readability test’, where we check how easily students are able to read and understand the handouts and textbooks we give them

Observing practice

Here we sit in during classroom and practical teaching, and observe what is going on. Once again we can do this is an unstructured way (by writing down what happens, and analysing it afterwards) or a structured way (by having a checklist of things we would like to see, and checking if these happen). We can also ask colleagues or even students to observe us, as we teach.

Questionnaires

The ‘Student Happiness Questionnaire’

It is common practice to present students with a questionnaire at the end of a part of a curriculum, or at the end of a term. The students are asked what they liked or disliked about the content; the teaching; the assessment; the practical arrangements. This can provide useful information about problems. However, it must be used with caution:

  • Students may be wrong – for example, since they lack a wider understanding they may believe (wrongly) that some of the course content is unimportant
  • Teachers may believe that this is all the evaluation you need to do. If it is routinely done, year after year, teachers tend to ignore the findings

We use these when we want to know people’s opinion about an aspect of a training course – practical arrangements, the relevance of the material that is taught, what happens in class and so on. Again, questionnaires can be unstructured (asking the respondents to write general comments on how they feel about the topic) or structured (giving questions with pre-prepared answers, from which they have to choose the one they prefer). There are some special kinds of questionnaires we use:

  • The ‘student happiness questionnaire’
  • Diaries: we ask teachers or students to keep diaries of their experiences on the course.

Interviews (with individuals) and discussions (with groups)

These are useful when we want information from people about aspects of our courses and teaching, but in more depth and detail. We carefully prepare some questions, and put them to the persons concerned. Then we record exactly what they say (by hand or with a tape recorder) and analyse the information afterwards. What were the main points that the respondents raised?

Ready-made data collection instrument

Other teachers who have gone before us can help us with evaluation. It is often possible to find ready-made data collection instruments, which deal with a variety of common problems in teaching. A good source of these is the booklet, 53 Ways to Evaluate Your Teaching by Gibbs, Gibbs and Habeshaw (published by Educational Technologies, Bristol, UK). Such instruments have been tested and refined, and should provide us with useful information. Of course we don’t use them uncritically – they usually need to be adapted a little to fit our own situation.

Who should evaluate?

Who is best placed to evaluate teaching practice? Do you do it yourself (an ‘insider’), or do you get someone else to do it for you (an ‘outsider’)? Do you evaluate your own practice, or that of your colleagues? The advantages of doing it yourself, about your own work, is that you understand it thoroughly – the background, the players, the details. The disadvantage is that you are used to looking at your work in a certain way, and it is difficult to see it objectively – so an outsider coming with a fresh view may be more useful. Outsiders usually want to be paid though!

Three points

When someone asks you to do an evaluation you must be opportunistic. Of course you are going to collect specific data with instruments you have prepared. However, you should use every opportunity to get additional information. Talk to everyone you meet (and write down what they say); look at notice boards and classroom walls (making notes of relevant information), go into the course filing cabinet and read relevant documents. In this way you gain a deeper understanding, which helps you to make the right judgements.

One of the aims of evaluation is to find and clarify problems. The problem is that many people find it difficult to accept that they have been making mistakes. You, therefore, have to present your judgements – your feedback – in a sensitive way. Start by listing all the good things that you found (and you will find them). Then, once you have affirmed the persons you are evaluating, you can mention the deficiencies in a polite and non-judgmental way.

Here are two books on evaluation which I have found very helpful:

  • Harris D and Bell C. (1986). Evaluating and Assessing for Learning. London: Kogan Page.
  • Hopkins D. (1989). Evaluation for School Development. Milton Keynes: Open University Press.

And finally ..

Here is a thought. The great philosopher Socrates said that the best teachers are like stinging flies. They make their students uncomfortable, by asking the really important, really difficult questions – then make the students find the answers to those themselves. What do you think?