Conventional wisdom on evaluation

A great deal has been learned about educational evaluation through research and practice over several decades, resulting in some broadly accepted tenets which can be summarised as follows.

General issues

Evaluation is an enormous field involving a wide range of possible purposes, processes, techniques and tools, though the basic procedure is the same as for any other kind of study. It involves selection of a focus, the collection and organisation of data, analysis and interpretation of the data; and the reporting of findings.
The essential purpose of educational evaluation is to determine the extent to which the aims and objectives of an educational experience/program have been met.
Evaluation is a continual and natural process which goes on all the time informally: however, there are particular times when it is important to use more formal methods for gathering and analysing data relating to specific questions.
As an important component in the teaching process, evaluation is essentially the responsibility of the teacher. It can feed into performance appraisal systems, but should not substitute for them.
Formative evaluation is used in a diagnostic sense to assess the strengths and weaknesses of the educative process with a view to improving learning experiences and, ultimately, outcomes.
Summative evaluation is generally carried out at the end of a period of teaching and has a judgemental focus, though it can also be formative in the sense that it recommends action for improvement for the next teaching period.
Evaluation of some sort or other should be continually in progress. Throughout the process of development and delivery of courses, from situational and needs analysis to institutionalisation, some form of evaluation is needed.
The context in which an evaluation takes place and the instruments that are used will significantly influence the nature of responses.
The use of a variety of techniques provides the best possibility of gaining accurate and comprehensive data on the quality of programs. By using a combination of methods, the weaknesses of one can be overcome by the strengths of others.
Most students readily participate in evaluation processes and are amenable to the use of different methods to gain their views.
There can be marked differences in response rates according to the method of evaluation used and the stated purposes of the evaluation.
Students tend to respond best when they perceive that teaching staff are genuinely interested in the results to improve their teaching, rather than to fulfill administrative requirements.
Lack of confidentiality and anonymity strongly influences the openness and honesty of some student responses.
Using data in a way not originally explained and agreed to by the contributors would generally be viewed as unethical behaviour.

Using questionnaires

It is desirable to use questionnaires as only one of a number of evaluation instruments and strategies. Reliance on consumer opinions alone provides insufficient evidence about the quality of a course or unit.
The context in which a questionnaire is presented influences responses.
The timing of presenting questionnaires also influences responses. Students may be more positive after they have completed their unit than during the semester.
Questionnaires completed in class generate a much higher response rate than those completed in a student's own time.
Questionnaires completed in a student's own time often contain more detailed and considered responses.
Over reliance on questionnaires can reduce the validity of results.

Deciding which questions to ask

To inform educational change, it is preferable to use a small number of well-chosen, carefully constructed questions rather than to use numerous vague or superficial questions.
Evaluative procedures should be designed to answer specific questions in a particular way. It is useless to ask questions that are too broad to translate into a meaningful evaluation study.
Questions chosen for evaluation should enable the evaluation objectives to be met.
Each educational environment has unique characteristics that give rise to particular evaluation questions relevant only to that environment.
Evaluation findings cannot necessarily be extrapolated to other situations because of the unique characteristics of educational environments.

Evaluation and technology

When evaluating the use of technology in educational environments, the normal principles of good evaluation apply.
Because of the costs involved in using technology, there is an acute need to evaluate the development and implementation processes in ways that provide timely feedback to ensure that improvements can be made before it is too late.
The use of technology broadens the nature of what should be evaluated because in a technological environment, there are many more variables that affect the educational process.
Technology can greatly facilitate the gathering of evaluation data. For example, there is a permanent record of online communications that can be a rich source of evaluation data if ethics approval has been obtained. Online survey forms can be embedded in the technology and responses stored on databases.
Many students particularly like computer-based surveys because they are usually quick and convenient to complete and return.
Development and implementation of technology should be an exploratory, evaluative process.