The main purpose of the evaluation 'stage' of the design cycle is to determine the extent to which the intended learning outcomes have been achieved, and how elements of the unit may be improved towards this end.
However, a comprehensive evaluation should address every element of the unit, which implies that you should also review the ILOs themselves. Difficulties that arise with activities and assessments may in fact be traced back to ILOs that are poorly expressed or unachievable. It is best to consider decisions you make at every stage of the design cycle as 'subject to review', and adopt a process of gathering data on their effectiveness continually.
Other purposes for evaluating aspects of a unit are:
- To meet Deakin's quality assurance requirements - every unit must be evaluated using the 'Evaluation of teaching and units questionnaire' (SETU), and you are required to provide evidence of how you have responded to feedback in ways that improve teaching and learning quality.
- To document evidence of the effectiveness of your teaching - this may be used in PPRs, or when applying for promotion or new positions.
- To diagnose strengths and weaknesses in the unit overall or in specific items. With off-campus students, where it is not possible to observe body language, it can be difficult to gauge the effectiveness of some techniques if you don't actively seek feedback. Some form of usability testing should be used with any online techniques you use.
- To provide feedback to other stakeholders, eg managers and developers.
- To gather data for research purposes (you will need prior ethics clearance and separate consent from students for this).
Often, informal procedures such as chatting with students about things in the course of an eLive session can yield important insights about the effectiveness of aspects of your unit. However, a formal process will prompt you to:
- collect data while you have the opportunity
- document responses
- collect data on key questions
- collect data from a representative range of students
- use a range of methods to collect different types of data
- not overlook this evaluation stage amidst the pressure of everyday academic life.
- Who are the stakeholders in the unit: students, teaching team, others?
- Open your Unit design matrix. On the final page is a form for an evaluation plan with some suggested key questions and data sources. Edit these and add more if you wish. Put an x in the relevant cells to indicate which source of data you will use to answer each key question.
- Decide which evaluation strategies you will use. Here are some suggestions:
- Create a file for collecting data that might be useful for your unit evaluation. Add to it whenever you see an indication that something is/is not working well, a student or teacher makes a comment that might be helpful, you have an idea for an improvement, you read about an activity that might work in this unit, etc.
- Use the Surveys tool to gather feedback from students. You can use this more than once in a trimester to gather feedback on the unit in general and specific items that the students have just completed.
- Set up an online discussion topic for feedback. Again, this can be general or you could invite comments on particular activities or assessments. (See the discussions guide for instructions.)
- Hold a meeting with the teaching team around mid-trimester and at the end of trimester to gauge responses to the unit materials, assessments, aims and ILOs. Prepare a set of general and specific questions related to your key questions and make notes to capture their responses.
- Review SETU data, both quantitative statistics and qualitative comments. For each question that might not have received a response you are happy with, consult the ITL page Using student evaluation of teaching data to enhance teaching and learning to find strategies to address these responses.
- Analyse student performance in assessments. How well were students able to achieve the ILOs?
- Analyse the extent to which various online tools and resources were used. Which were used the most? Were there any that were not used very much? You might need to follow up why certain tools or resources were not used well by asking students or others in the teaching team, so this analysis is best done before the end of trimester while you can still access your students.
- Apply a generic online unit quality checklist for higher education. A widely-used, detailed and informative rubric to use is the Quality Matters program rubric.
Now back to the start
This is the last stage of the 'constructive alignment' sequence, but 'evaluation' should never be seen as an endpoint. Don't just file the valuable information you have gained from using these evaluation strategies, but use it to revisit and improve each of the other stages in the sequence - ILOs, assessments, activities, and site design and development.
back to top