Evaluations of technology-based teaching and learning
Since 1995, a range of technology-based teaching and learning projects has been undertaken at Deakin University. The projects included Deakin Interchange, the use of CAL in Accounting, the Psychology Electronic Warehouse, video link teaching in Chemistry, the V-Lab in Human Movement, a professional development course in computer conferencing for Business and Law staff, the Online Teaching and Learning Enhancement project, the Video-teaching Project, the implementation of the Learning Management System, TopClass, Online learning in first year psychology, and the implementation of Deakin Studies Online (DSO). Evaluations of these projects have resulted in a number of general findings about the following aspects that can inform future educational developments.
What has been learned about developing educational technology?
What has been learned about technology implementation and integration into teaching?
What has been learned about students' experiences
What has been learned about assessment
What has been learned about evaluation processes?
What has been learned about developing educational technology?
- The development of technology for educational use needs to be effectively project managed from the outset. Careful consideration needs to be given to selection of the project manager in light of the skills required for bringing together the educational, discipline and technical knowledge underpinning the development.
- Because of the costs involved in software development, there is no need to 'reinvent the wheel'. Any software developed for educational purposes should help to address a clearly identified educational problem. It should also be unique and offer features that previous software has not offered.
- Understanding the process of developing educational material for technology delivery is difficult for many academic staff. Conversely, ensuring that educational content is produced on time is a problem for developers.
- It is sometimes difficult to reach agreed understandings about educational uses of the technology. Academics are not always able to articulate what they want the technology to do because they are not sufficiently aware of its possibilities.
- Similarly technical experts generally do not have the educational or discipline expertise to suggest ways of using the technology for learning purposes.
- Sound educational design advice is needed on a continual basis to ensure that appropriate curriculum issues are fully explored during the development and implementation stages of technology-based learning environments.
- Gaining copyright clearance can be a costly and time-consuming exercise so the process needs to start as early in the development phase as possible.
- It is important to thoroughly test new software at beta and pilot stages before releasing it for mainstream use. However, a balance needs to be found between proper testing of versions and the need to release the software.
- It is not always possible for developers to anticipate all the incompatibilities that might arise between Deakin's software and configurations on users' computers that can be quite unique.
- It is important for software developers to build a prototype as quickly as possible to enable teaching staff to see the technical possibilities. They are then in a better position to determine whether it can meet their educational objectives.
- Developing quality software takes a long time and is a costly exercise. It inevitably takes longer than anticipated because of the unexpected problems that arise during development.
- Schedules for release often have to be extended, or the software released before it is properly tested.
- There is a need to set parameters for the product being developed. The number of features built into the software need to be limited since the possibilities are endless. There is a tendency to build new improved versions that probably add to the cost.
- It is more cost effective to write 'smart' code that can be used in several ways.
What has been learned about technology implementation and integration into teaching?
- Implementation of technology has to be seen as an experimental and exploratory process informed by continual evaluations.
- Systematic plans for supporting staff and students should be clearly articulated and mechanisms put in place prior to implementation.
- Teaching staff need to inform students what they expect from them in respect to using technology, and students need to know what they can reasonably expect from staff.
- With untested or inadequately tested software, staff and students are likely to experience many more technical difficulties and need much more support.
- Stakeholders need an environment where they can feel comfortable about making mistakes providing they learn from them and the knowledge is disseminated.
- It is crucial that thorough planning occur at the institutional level to ensure that adequate technical infrastructure is in place prior to implementation.
- It is essential to have contingency plans in place in the event that the technology or infrastructure fails.
- Where mainstreaming is done incrementally, the fall out from technical failures is minimised.
- Investment in staff training before mainstreaming will reduce the need for support during implementation.
- Irrespective of the level of good planning, staff and students need to be aware that things may not go according to plan in the first year of implementation—that is the nature of educational technology development and implementation.
- Implementation works best when relevant stakeholders, and in particular, Deans and Heads of Schools are committed to the use of the technology and recognise its potential for learning.
Campus labs
- Activities in campus laboratories need to be properly coordinated, particularly for first year students.
- Laboratories need to be fully equipped and manned by academic and technical support staff—troubleshooting tips and other instructions for users should be easily accessible to staff and students.
- It is sensible to allocate time for particular groups of students to use the labs rather than leave access to chance.
What has been learned about students' experiences
- Students expect online activities to add value and interest to their study. Most of them do not like large amounts of text online and prefer this to be delivered in print form.
- Students often have difficulty navigating resource-rich websites, and can become confused about the purpose and relevance of particular resources. They need clear study guidelines that integrate every aspect of their physical and virtual learning environments. Moreover, they need to know how, when, and why to use each recommended resource.
- It is important to provide an appropriate mix of resources and to ensure that resource overload is avoided.
- When designing learning environments, careful consideration needs to be given to the balance between print and online resources, print and face-to-face classes, and face-to-face and online discussion groups.
- There may be a need to rethink the nature of Deakin study guides to accommodate the move to online learning. They need a guide to learning that applies to and integrates the total learning environment.
- Many students lack experience with basic use of computers—during the implementation of technology, they are 'guinea pigs'.
- Students often experience anger, frustration, confusion and uncertainty when new technology is being implemented. They may experience a lack of control over their work, especially if they encounter technical problems that they cannot solve for themselves.
- When students are kept fully informed about what is happening, and when they sense that their needs are being considered, they are more likely to be positive about the experience of using new technology, and are more likely to cope with any technical mishaps.
- When educational technology is mainstreamed, the type and extent of support needed is more comprehensive and demanding than that required for early adoption.
- Particular consideration should be given to the learning environment established for first year students. They benefit from more direction, structure and personal support in the first semester of their course which is effectively a transition period during which they experience a steep learning curve.
- Implementation works best when there is a consultative and supportive approach taken by staff who are attune to students' reactions and have plans in place for responding to problems when they arise.
- Consideration needs to be given to anticipating difficulties that might occur and to ways of dealing with them.
- Students' perceptions may indicate that technology-based programs enhance their learning. However, perceptions are not a reliable indicator of the extent and nature of learning. This requires further investigation once the technology is stable.
What has been learned about assessment?
- Whether or not student assessment requirements should be embedded in the technology is an issue that needs careful consideration. If there are technical problems that delay students' study schedules or cause anxiety, it is clearly disadvantageous.
- In the initial stages of implementation, it is preferable that assessment built into the technology be optional or only a hurdle requirement, not graded.
- However, students who have to spend time using the technology want some credit for their efforts. Providing the technology is robust, linking assessment to it sends the message that it is an integral part of their course and is therefore the best way to integrate the technology into mainstream teaching and learning.
- Technology offers opportunities for continual self- assessment that can assist students' learning.
- Provision of timely, formative feedback online generally results in deeper learning and better outcomes, but can be time-consuming for staff.
You may find it useful to listen to what some academic staff have to say about online assessment:
http://www.deakin.edu.au/itl/teach-learn/cases/files/online_feat/assessment.htm
What has been learned about evaluation processes?
- Evaluations reveal multiple stories and different perceptions of the reality of an experience, sometimes giving rise to contradictory evidence.
- Evaluations highlight institutional politics because there are multiple stakeholders with vested interests in particular aspects of the development and delivery processes.
- When things go wrong, there is a tendency for groups to emphasise the failings of other groups and to rationalise their own actions.
- A number of tertiary institutions have difficulty reporting failure and problems in the context of implementing technology.
- Sometimes it may not be politic for an internal evaluator to be frank because it is not always in the best interests of fostering change if groups of stakeholders are humiliated or found wanting by evaluation findings.
- This leads to a sanitisation of findings that places the institution and groups of stakeholders in the best possible light.
- Power relationships between groups and even within groups sometimes cause conflict about how to respond to evaluation findings.
- Consideration needs to be given to ways of determining the effectiveness of evaluation processes, particularly the extent to which they enable improvements to be made to teaching and learning.
- If there is a lack of consensus about solutions, or if powerful groups choose to ignore recommendations, evaluation does not improve anything.
- The level of resources assigned to evaluation can indicate how much importance is placed on the process and the outcomes.
- The professional standing of the evaluators can affect the extent to which recommendations are implemented.
- Evaluation processes can be facilitated through the establishment of a framework within which evaluations take place.
- The roles and responsibilities of all involved in evaluation processes should be clearly articulated and agreed upon.
- With evaluation plans and procedures in place, there is less chance that evaluation will be compromised by the prevailing political climate.
