Nutrition Competency Framework (NCF) assessment tools evaluation

MCQS (Multiple Choice Questions)

A group of team members and external members were trained by an MCQ expert Mr Neville Chiavaroli, and worked together to develop a set of MCQs and Extended Matching Questions (EMQs) for years one and two of medical courses in Australia.  A pilot set of 35 questions was administered to 30 dietetic students from Deakin and University of Wollongong and these were subjected to item analysis to assess quality.

The quality components assessed were: the relative difficulties of the items, discrimination index (point biserial) and Rasch statistics which provide an indication of poorly written items and provide information on the extent that a question discriminates between high-ability and low ability-examinees.

After review of the item analysis the pilot set of questions were revised and additional questions drafted by members of the writing working group. Final testing of 84 MCQs was conducted on 171 students (140 dietetic students, from three universities and 31 second year medical students, from two universities). Item analysis was conducted and results reviewed by the expert team which resulted in the retention of 67 (56 for year levels 1 and 2; 11 for years 3 and 4) quality nutrition MCQs in the database.

OSCEs (Objective Structured Clinical Examinations)

Monash OSCE

Evaluation method: Evaluated by 5 external experts (MON, DU), and two team member experts (KR, JS).

Evaluation results: Reviewers agreed that the OSCE was probably most appropriate for assessment of students who had completed one year of clinical placement.  However, it could be also be readily modified to provide an OSCE focused upon eliciting a history for earlier years.  The OSCE provided an authentic realistic scenario that would be particularly useful for assessment of management skills. Positive feedback included the integration between clinical skills & health promotion, human development and the focus upon lifestyle aspects of history and management. Areas for improvement included some refinement of the stem and instructions to students.  The marking rubric would need to be revised and customised for the specific assessment requirements and format of individual universities.