Gauging how useful MCQs are as a learning technique

Lecturer:
Tony George  

Faculty:
Science

Project:
Assessment tracking of MCQ real-time interactive learning

Keywords:
MCQs, real-time interactive learning, lecture sessions

Associate Professor Tony George is assessing the value of MCQs (Multiple Choice Questions) beyond formal examinations, gauging students' learning in real-time.  

What the project is about 

Using trial quizzes in Canvas for students to assess their progress and learning of a subject’s content is a popular practice. Associate Professor George is adapting this model and bringing the trial quiz directly into the lecture. He limits the questions to the material just covered in that session to construct a real-time interactive learning experience. 

Throughout this project, Associate Professor George is intending to:

• Demystify and deconstruct MCQ questions for students through reviewing the five answer choices, and highlighting the reasoning behind the correct answer and incorrect alternative choices. 
• Engage students with a real-time interactive learning experience of the lecture material, translating some of the key ideas into MCQs for discussion and practice.
• Provide students with an early and direct indication of examination questions that will be encountered in the mid-test and final assessment.

Why the project is being introduced

MCQs are seen by some as a simplistic and superficial assessment method that brings too much guesswork and the element of chance into the examination process. However this project is enabling students to instantly reflect on what they’ve learned during the lecture, get feedback, and use the online MCQs as a revision tool at home.

How the project is being implemented

Associate Professor George is evaluating the effectiveness of MCQs through the following steps: 

1. Setting aside 15 minutes at the end of each two-hour lecture session for two trial MCQs relevant to that lecture’s content, taken from previous exams, or newly written.
2. Allowing 2-3 minutes for students to complete the MCQs either alone or in groups.
3. Giving solutions to each question.
4. Putting 2-3 MCQs into the online discussion board for students to attempt (in a peer to peer process) before the next lecture (at the end of which these MCQs will be solved and explained). Students are able to discuss MCQs directly with the lecturer in the discussion board.
5. Employing a casual academic as a ‘teaching and learning assessor’ (TLA) to monitor the scheme. They are:

a)   Attending two lecture sessions as an observer of the lecture content, delivery and student attentiveness.
b)   Attending four sessions as a surrogate student to follow the lecture content and attempt the MCQs to be able to give a personal appraisal in the final written report.
c)   Designing an instructive questionnaire to monitor and assess student opinion of the successes and/or shortcomings of the scheme.
d)   Presenting the questionnaire to the students in the final lecture session, then collect, assess, score and analyse the data.

Challenges and considerations

Associate Professor George used comments about MCQs taken from from the 2012 student questionnaire to inform this project.

Negative comments included:

• Although the questions were followed by an explanation often they, the explanations, did not necessarily contribute to our understanding.
• The wording of the questions was sometimes too cryptic and leads to decreased confidence. More focus on understanding basic concepts in a transparent fashion would be appreciated.
• The MCQs did not entirely reflect the difficulty shown in the mid-semester – MCQs were too easy.

Positive comments included:

• In my opinion they are quite helpful in gauging my own understanding of the lectures. I felt if there were more, or perhaps alternate measure such as online practice MCQ, it would be very beneficial.
• MCQs very helpful, but dealt with too quickly and would prefer hard copy to reflect on later.
• A fantastic way to engage students and reinforce material just learnt. Posting sample MCQs online would also be beneficial for revision purposes.

Exploring further
UTS. Multiple Choice Questions as a form of assessment
Read...

Delving even deeper
Tweed, M., Thompson-Fawcett, M., Schwartz, P., Wilkinson, T. (2013). Determining measures of insight and foresight from responses to multiple choice questions. University of Otago, New Zealand.
Read...
Case, S.M. & Swanson, D.B. (1996/2002). Constructing Written Test Questions For the Basic and Clinical Sciences. National Board of Medical Examiners. Philadelphia.
Read...