CEQ and programs - The longer term

Using CEQ results

The data produced by the Course Experience Questionnaire (CEQ) are divided using the Department of Education, Science and Training (DEST) field of education' categories, which do not always align conveniently with Deakin courses/programs, and in some cases, will require the best available standard category to be used to derive a CEQ data set for a program. As with all surveys, the number of respondents and response rate are crucial for data validity and confidence in inferring anything meaningful from the CEQ data. The guidelines for the disclosure of CEQ data require an institutional response rate of at least 50 percent before institutional data are published. What is considered a minimum number of responses and minimum response rate for producing valid data in student evaluation of teaching (SET)-style surveys varies widely in the published literature. It goes without saying that important decisions should only be made on the basis of data that are statistically reliable (Graduate Careers Australia, 2006). Many universities (including Deakin), whilst having good overall institutional CEQ response rates, find that at the course level the response rate and/or number of respondents is too small to draw reliable (or any) conclusions. Where numbers of responses are very low, it has been suggested that pooling data from two or more years may create a more useful data set (Aungles & Karme, 2000). However, this obviously reduces the currency of the data, and is a questionable approach if the characteristics of the course and/or delivery have varied substantially during the pooling period.

Results from the CEQ are a whole course/programme summative evaluation and, like all quality measures, should be used as one of a range of performance indicators providing directions for further investigation; they should be considered indicative rather than conclusive (Trembath, 1999). Publicly available CEQ information permits a comparison against the results from other institutions for the same field of education areas. However, it is known that CEQ results are systematically influenced by a range of factors, including institutional characteristics, so it may be more useful to benchmark' results against a smaller range of institutions with similar characteristics. Sustained trends in the results for CEQ items over time can be a valuable starting point for investigations seeking to understand root causes for particular student perceptions (or changes in them). The longitudinal use of the CEQ in Australia means that there is now a significant data time series available for use (Wilson, Lissio & Ramsden, 1997).

CEQ data has been shown to be systematically influenced by a range of factors, including field of study, so direct inter-faculty or even intra-faculty comparisons may not be appropriate (Wilson et al., 1997). While the scale construction of the CEQ permits such comparisons, it is necessary to take into account systematic differences between fields of education, such as teaching methods, work load, methods of assessment, etc. Nevertheless, where a particular field of study has good outcomes in a particular CEQ scale or item, it may be useful to investigate good teaching and learning practices in use that may generally applicable and contribute to positive student experiences. The construction of the CEQ does not permit the results from different scales to be compared. It is not sensible to suggest that a higher mean on the Good Teaching scale compared to the Student Support scale indicates that students thought their teaching was better than the student support they received.

Ramsden, one of the developers of the CEQ offers advice on using CEQ data (in the context of reporting on quality assurance) (Ramsden, 2003). Don't consider CEQ numbers in isolation from other sources of information, such as SET surveys, the open-ended CEQ responses, surveys of employers and graduates, and accreditation bodies. Other external sources of quality-related input include the program Academic Advisory Board, the Adjunct Professoriate and external academic colleagues. Benchmarking nationally is likely to be less useful than carefully selecting appropriate comparison institutions. CEQ time series data will contain random variations - it is only sensible to report trends that are significant, sustained and linked to some specific intervention. While CEQ numbers may be interesting, it is their use that is more important. A specific focus for improvement on one or two key areas related to the university's core mission is likely to be the most effective approach.

The quantitative responses to the CEQ may highlight potential areas of good performance or concern, but mere numbers don't say much about the underlying causes. This is where the open-ended Best Aspects (BA) and Needs Improvement (NI) student comments may assist. For small numbers of respondents/responses, it may be possible to manually analyse the open-ended student comments. For larger data sets the CEQuery software package may assist. CEQuery was developed as part of DEST Higher Education Innovation Program project for the purpose of analysing the large volumes of qualitative data produced at the national level by the CEQ. The software package is available for all higher education institutions to use, and comes with a dictionary of keywords classified into five principal domains (Outcomes, Staff, Course design, Assessment and Support) and 26 sub-domains that are used to automatically code/classify student comments to identify the frequency of student responses in all sub-domains (Scott, 2006). The total count of coded responses (known as 'hits') is typically larger than the number of student comments, as one comment may contain information about more than one sub-domain. The total count of BA and NI hits for a particular sub-domain is taken to be a measure of its importance to students. The ratio of BA hits to NI hits (known as the odds of a best aspect) is taken to be a measure of perceived student quality - the lower the odds, the lower the perception of quality for that sub-domain. Where a sub-domain has a low odds ratio (perceived quality) coupled with a high total number of hits (perceived importance), it is suggestive that further investigation is required, and the complete list of student comments relating to that sub-domain can then be examined in detail.

Where a specific CEQ item(s) is targeted for improvement, it must be recognised that a whole-of-program approach will be required. The CEQ measures the students' entire course experience, so fixing a problem in one study unit and not in others is unlikely to lead to a significant change in overall course perceptions. Likewise, currently enrolled students who complete the CEQ upon graduation may not be significantly influenced by immediate remedial actions when it comes to considering their entire course experience and completing the CEQ. Adding in the inherent time lag in the collection and processing of CEQ data, it may take a significant period (and a commensurate measure of faith) for current action to translate into statistically measurable CEQ results. For an example of a strategic, university-wide, long-term and evidence-based approach to course-level teaching and learning improvement with defensible statistical results see (Barrie, Ginns & Prosser, 2005).

Another possible use of the CEQ instrument itself is as a thought experiment or checklist template for new or revised courses; consider how what is proposed might impact on the student perceptions for the CEQ items - for example, if a course is to have intensive assessment, how might this impact on the overall workload, time available for feedback on assignments, etc?

Activity

Have you ever previously received/used any CEQ data for the program(s) that you contribute to?  If yes, how did you use it?  If no, how could you use program CEQ data?

Forward to: In general