What are CEQ, SETU and AUSSE, and why are they important?

Student Evaluation of Teaching and Units

While it has been shown that the original course experience questionnaire (CEQ) was a useful summative measure of student experience at the level of aggregation of whole-of-program and broad field of study, it was not intended as an instrument to examine the quality of individual units of study or performance of staff repeatedly within a program (Ramsden, 1991). So, in addition to participating in the national CEQ survey and perhaps administering their own CEQ-style graduate course experience survey(s), many universities also administer student questionnaires relating to individual units of study. These questionnaires have a range of names - Units of Study Evaluation (USE)(Institute for Teaching & Learning, 2009), Student Evaluation of Teaching Effectiveness (SETE) (Emery, Kramer & Tian, 2003), Student Opinion Survey (SOS) (Bedggood & Pollard, 1999), Student Evaluation of Teaching and Subjects (SETS) (Neumann, 2000), Student Perceptions of Teaching (SPOT) (Hicks, 1999), or in the case of Deakin, Student Evaluation of Teaching and Units (SETU). Collectively, they are referred to as student evaluation of teaching (SET) (Millea & Grimes, 2002). It should be noted that, while the data from SET surveys can inform the teaching evaluation process, of itself, administering questionnaires to students is not evaluation; evaluation of teaching is the on-going process of discovering the strengths and weaknesses of your professional work with students and acting upon this information (Ramsden & Dodds, 1989).

As the array of names suggest, these instruments have a range of stated purposes, including measurement of perceived worth/value of units of study, measurement of perceived performance of the teacher, to assist in the disaggregation of course experience-type survey data, etc., or a combination of these reasons. In the case of Deakin, the SETU instrument aims to collect student perceptions about both the delivery and content in units of study. The inclusion of items evaluating teacher performance in some SET instruments is premised on research that showed that effectiveness of student learning was influenced by teacher behaviours (teacher enthusiasm, preparation and organisation, presentation skills, clarity of objectives, etc.) (Sheehan & DuPrey, 1999). The frequency of application of SET instruments varies; in an analogy to statistical quality assurance, some institutions administer them bi- or triennially; at Deakin, SETU is currently administered to every undergraduate and postgraduate unit in every semester of offer (Deakin University, 2009).

Unlike the CEQ, the evidence that unit-based SET instruments are valid and reliable measures of teaching quality is more equivocal. There is evidence that well designed SET questionnaires can be made reliable (the same instrument administered under the same conditions yields the same results) (Langbein, 1994). Validity refers to the ability of the instrument to accurately measure what it purports to measure without being influenced by factors that are expected to be irrelevant to teaching quality (Langbein, 1994). For many simple SET questionnaires it is not possible to establish reliability (Bedggood & Pollard, 1999), and many external factors beyond the control of academic staff have been found to influence SET results (and hence validity), including discipline, course level and whether the unit is mandatory or not (Emery et al., 2003). In addition, while some SET ratings have been shown to exhibit a positive correlation with student outcomes, the correlation is modest (Miller, 1998).

To have confidence in making important judgements based on survey data, we must first be confident that the respondent group is a representative sample of the population under consideration. The range of recommendations for what is a valid minimum number of respondents and/or valid minimum response rate in SET surveys varies dramatically in the literature. Another concern is that many studies have shown that students who respond to voluntary SET instruments are different in their study habits and academic achievement to non-responders (Richardson, 2005). It is important that any SET results reported are statistically justifiable (Miller, 1998).

All of these limitations of typical SET instruments mean that the results must be interpreted with caution. The literature describes a range of formats for reporting SET results, from simple presentation of the raw data through to sophisticated schemes such as indicating performance range (within one of lower 25%, mid 50% or top 25%) for each scale item against results from comparable study units based on grouping according to the known influence factors of discipline, class size and year level (Neumann, 2000). Where a SET instrument contains a range of items, there is an opportunity to examine comparative strengths and weaknesses, as well as overall student satisfaction. It is important to consider the range of external factors that may have influenced SET results, including class size, available resources, whether the unit was elective or compulsory, etc. (Institute for Teaching and Learning, 2009).

In the same way that the CEQ contains both quantitative and qualitative measures of student course experience, it is recognised that SET instruments containing only a fixed set of items that produce strictly quantitative results provide a very limited picture of unit teaching. It is desirable that students have the opportunity to also provide open-ended written feedback, commonly (including the current version of SETU at Deakin) this takes the form of asking students to comment on the 'best' and 'worst/most in need of improvement' aspects of the unit (Miller, 1998). This qualitative feedback can be extremely valuable in understanding the reasons why students have given a particular quantitative SET rating.

Reflecting the modern understanding of the multi-faceted nature of 'quality' and the finding that it is the entirety of the university experience that contributes to the student 'course experience', virtually all authors examining the value of the SET process recognise that SET data are only one of many sources of information that should be called upon when assessing the quality of teaching in units. This is reflected in Deakin's procedure on Evaluation of Teaching and Units (Deakin University, 2009). Other equally valuable sources include objective measures of student learning (such as unit marks), reflective self-assessment of teaching performance, peer assessment of teaching and student focus groups. In summary, although remaining contentious, SET instruments are in wide use and for a range of purposes. With thoughtful questionnaire design, valid response rates, and careful interpretation of the results, SET data can be one useful input of the teaching and learning quality improvement process (Richardson, 2005).

Activity

If you are involved in university teaching, you have probably encountered one or more SET survey instruments – at Deakin, this would probably be SETU.  What other methods of obtaining feedback from students about teaching could you use?

Forward to: The Australasian Survey of Student Engagement