SETU and units

Using SETU results

As with the course experience questionnaire (CEQ), the student evaluation of teaching and units (SETU) questionnaire provides both quantitative and qualitative data based on student perceptions of their experiences of a study unit that may be used as one source to inform investigations about the quality of that unit. The SETU data provided is in a relatively raw form, and like all survey data must be interpreted with some care. To be useful, the respondent group must be based on a representative sample of the unit enrolment - a reasonable number of responses (SETU data is not reported for units that receive a very low number of responses), a reasonable overall response rate and a reasonably equal rate of responses between campuses provide some confidence that the survey sample is representative. Whereas CEQ data comes from a large pool of respondents considering their entire course/program, SETU data comes from a single unit in a single semester, so consideration of the results must take into account the unique context of each unit and how this may have influenced students when rating each SETU item. Common sense and the professional insights of the unit chair will both be valuable in this regard. SETU results should be triangulated with other sources of teaching evaluation data such as self-review, peer review, unit results, etc. (Wagenaar, 1995).

The generally broad variation in student expectations of university can lead to significant variation in student responses to SETU. A key factor in interpreting SETU data is to look for 'consistency' of results (or lack thereof). This can take several forms:

• a small standard deviation on a mean item rating indicates that most respondents were in general agreement; if the standard deviation is large, then there was a wide spread in the student ratings (even though the same mean value can be reported in both cases);
• a large positive difference between the proportion of agreement and disagreement indicates that substantially more students agreed with an item than disagreed with it; a negative difference means that more students disagreed than agreed (note that the size of the difference result is influenced by the number of students selecting the neutral rating for an item);
• if the written open-ended comments reveal common, repeated themes, these are worthy of further investigation;
• if SETU results for a teacher are similar in two or more units, this would tend to confirm the student ratings relating to teaching; and
• if SETU results are consistent with other forms of evaluation, this would tend to confirm the student ratings.

Related to data consistency is data 'discrepancy'. Again, this can take several forms:

• especially if the total number of respondents is low, a small number of ratings that are significantly different from the majority can unduly influence the reported mean - though the compressed rating range (1-5) of the SETU instrument somewhat mitigates against this;
• a small number of adverse comments that seem at odds with the majority of written feedback are probably not worth losing a lot of sleep over - a CEQ-style 'odds analysis' (best aspects/needs improvement) may help to interpret and prioritise the themes present in the open-ended SETU comments;
• a rating in one item that is significantly different from the others in the SETU results for the same unit might be an indicator worth investigating further; and
• a large standard deviation on a mean item rating may be caused by a bimodal distribution of students responses, that is, the distribution of the ratings for the item may be split between a group rating the item highly and another group rating lowly - this may be indicative of sub-groups in the class which have significantly different backgrounds, both of which need to be considered in the delivery strategies for the unit.

SETU results provide one avenue for identifying areas for teaching improvement. The particular SETU item(s) that is of concern provides a focus point for further investigation, planning and implementing of changes to unit materials and/or teaching practice. As with the CEQ, there is value in collecting SETU longitudinally to look for trends over time. Statistically significant changes in SETU results over time are one method of confirming that particular interventions made previously have had a positive impact on student perceptions. There are many places for you to seek advice on the interpretation of SETU results and for the planning and implementation of changes to your units and teaching. In no particular order, try the following:

• your academic peers (from Deakin or elsewhere) - those teaching similar units and/or students to you;
• a mentor you have found helpful with academic matters;
• the director/academic leader of the program(s) you teach in;
• your Associate Head of School for Teaching and Learning;
• your Head of School;
• your Faculty Associate Dean for Teaching and Learning; or
• the Institute for Teaching and Learning (http://www.deakin.edu.au/itl/).

While SETU data is primarily related to specific units, it may have additional wider uses. At the School level, it may be one useful input in considering the allocation of teaching duties to ensure academic staff have an enthusiasm for the areas they teach in. Taking into consideration that SETU ratings can be influenced by systematic factors (class size, year level, discipline, etc.), SETU results can be one way of identifying good teachers/teaching teams who can provide exemplars of good teaching practice that may be transferable between units.

A wealth of information, both general and discipline-specific, is available as guidance for those seeking to improve their teaching generally, or in specific areas. The following sections offer some general advice for each of the current ten SETU items, and some of the optional items that are frequently used. Where appropriate, use has been made of published, evidence-based frameworks, though those presented are not the only ones available. If you would like additional advice, please contact the Institute of Teaching and Learning.

Activity

Have you ever previously received/used any SETU data for the unit(s) that you contribute to?  If yes, how did you use it?  If no, how could you use SETU data?

Forward to: This unit was well taught / I was satisfied with the quality of teaching from this teacher in this unit