Suskie’s advice for writing better test questions (Chapter 11, 2009) is generally good: unless you are testing vocabulary, use the simplest possible language. Distractor answers based on common student misconceptions can help identify where a student has gone wrong. She also points out that creating a good test will take several cycles of testing, analysis of the results, editing and retesting. Following her advice, you can create multiple-choice tests that give you good data on student content learning. Her Exhibit 11.2 is an excellent example:

 

Which statement refers to measurement as opposed to evaluation?

A.    Emily got 90 percent correct on the math test.

B.    Lin’s test scores have increased satisfactorily this year.

C.    Justin’s score of 20 on this test indicates that his study habits are ineffective.

D.    Keesha got straight A’s in her history courses this year.

{A is correct}

 

The answers are plausible and of the same length, the question is phrased in the positive, and neither “all” nor “none of the above” is an option.  Most importantly, the answers cannot be found on Google! Such questions will help you assess basic understanding of content and can be very useful as a measure of student progress.

 

However, tests on content send an implicit message that knowing content is sufficient, that knowledge comes in discrete units and that the important (or at least measurable) things are black and white. If you want to emphasize that knowledge is a starting point and that argument, insight, and persuasion are where the added value is, write multiple-choice questions that emphasize the message that the important things in life are messy.

 

Each week I teach both new content and a new writing or critical thinking technique. As the semester progresses, so does the format of the pre-class multiple-choice questions. Early in the semester, I want students to learn about evidence and to practice identifying which facts are the most important, so for weeks all of the questions before class are in the same format:

 

Question 1

The following are all true statements about Jimmie Lunceford and Duke Ellington. Which of them are most relevant to why each (or both) band leaders are important to the history of jazz? (Check all that apply.  Partial credit is available.)

Answer (and percent correct for each answer)   Average score = 0.9 out of 2 points.

Y – Lunceford and Ellington both treated jazz as a serious art form. (69.7%)

N – Lunceford was famous for his slightly old-fashioned 2-beat swing feel (instead of the increasingly common 4-beat swing feel). (45.9%)

Y – The Lunceford band was extremely well rehearsed and could play together very precisely. (55.0%)

Y – Ellington was interested in the unique and individual timbres (sound and way of playing) of each member of his band and mixed these particular textures rather than just using the entire section as a similar sounding unit. (95.4%)

Y – Duke Ellington performed for floor shows for a white audience at the Cotton Club from 1927-1932. (53.2%)

N – Lunceford band performances often included hand motions and stage antics like throwing trumpets up in the air. (59.6%)

Y – Ellington’s most famous soloists included Johnny Hodges, Cootie Williams, Tricky Sam Nanton, and Ben Webster.  (42.2%)

 

Suskie suggests avoiding “select all that apply” questions as they can penalize slow readers. However, I want to reinforce the importance of careful and slow reading and thinking: that seems to me the entire point of a liberal arts education (and ironically, something that is hard to teach in class). The point of these questions is to teach students how to consider all of the evidence before they determine which is most important.

 

Since the point here is not speed, but judgment, none of these exams have much time pressure. I do not want to give students so much time that they try to beat the system with endless searching, and I try to ask questions for which such searching will be fruitless anyway–all of the answers are true. The emphasis here is on judging for themselves how the facts fit together. This question format makes every student a historian; they now need to assess the facts and decide which are relevant.

 

Most LMS also allow faculty to give feedback on individual questions, so students can learn from their mistakes. With most online testing tools, feedback and subsequent questions can be linked, so it is possible that the answer to one question can determine the next question. A student who is doing poorly can get the same type of question again (this is, of course, how computer games respond). The point is that feedback is a part of the learning cycle and that it also needs to be a part of the assessment strategy.

 

Allowing students to write their own feedback to questions can increase engagement and also help clarify both their thinking. A discussion board on this question might start with your feedback and invite argument, or simply ask the students why each answer is relevant.

 

A few weeks later in the semester, students start pulling together evidence into arguments and rebuttals, so the question format changes slightly.

 

Question 2

Which of the following statements about the music on the ECM label (all of which are true) would be the best evidence for supporting the argument that music on the ECM label is jazz?

Answer (and percent correct for each answer)  Average score = 1.6 out of 2 points.

Y – This music includes soloists over a rhythm section. (95.3%)

Y – This music features melodic improvisation over a modal groove. (91.6%)

Y – This music is a fusion of jazz process with the native musics of the players. (88.8%)

Y – The musicians are encouraged to find a unique personal sound. (92.6%)

N – This music does not relate to the cultural heritage of African-Americans. (96.2%)

N – There is a house sound. (86.0%)

N – There are American musicians in some of the bands. (78.6%)

N – This music does not swing. (96.3%)

N – The blues are not essential in this music. (91.6%)

 

Suskie suggests avoiding negative statements, but in more sophisticated argumentation, the absence of something can be significant. As the semester progresses, both the language and the concepts increase in complexity. By the end of the semester, students should be able to analyze the same data set (the same set of true statements listed in question 2) to gather evidence either for the positive argument or its rebuttal. This is not just a skill for trial lawyers.

 

Question 3 Which of the following statements about the music on the ECM label (all true) would be the best evidence for refuting the argument that music on the ECM label is jazz?

Answer (same data set as above) (Ave. = 1.3 out of 2):

 

The evidence that rebuttal questions are harder appears in the lower average class scores (1.3 points out of 2 instead of 1.6 for the positive argument). These questions send a constant message that what you do with the data (or what you find on the internet) is critical. Students learn that prosecutors and defense counsels, scientists and theologians, historians and anthropologists, have access to the same sets of data and that relevance depends on which story you want to tell.

 

Sample Multiple Answer Format Questions

Here are some sample multiple choice question formats. I prefer the multiple answer format where there are multiple right answers, and with partial credit allowed; i.e. if there are three correct answers and I get two of them I get 2/3^rds of the point, minus any wrong answers I selected.  These are harder, but this format makes most of the common student strategies less useful and requires more thought. Here is how to create multiple answer questions in Blackboard: http://ondemand.blackboard.com/r9/movies/Bb9_Tests_Surveys_Create_Multiple_Answer_Question.htm

 

These are organized by Bloom levels:

KNOWLEDGE (recall and recognition)

What are the X of Y?

Which of the following are important theories of X?

Which of the following best define or describe X?

Which of these are the same as X?

Identify which of the following are symptoms of X?

 

COMPREHENSION (understanding examples, meaning, and extrapolating)

Which of the following is an example of X?

The theory of Y means…?

Which of these are summaries of X?

Which of these might happen if…?

What would you expect to happen if?

What seems to be implied by X?

What is another way of stating X?

Why did X reject Y?

Match the Y with X.

 

APPLICATIONS (organize or solve with new situations or terms)

Which of the following are stated/formulated properly?

Which of the following Y might be most useful to X?

What would be the best way to improve X?

Which of the following are new examples of ?

Describe the process for determining if…?

 

ANALYSIS (breaking apart, compare and contrast, generalizing)

Which of the following are unstated assumptions in article Y?

What are the most important issues for author X in Y?

What is the point of view of X?

Which of the following statements from article X are fact?

Which of the following statements from article X are opinion?

Which of the following facts (all true) are most relevant for the argument X?

Which of the following facts (all true) are least relevant for the argument X?

What is the premise of ?

Which ideas/theories that we have studied are being applied in X?

Which of the following might be motives for X?

What distinguishes X from Y?

 

SYNTHESIS (combining elements into a new patterns)

Which of the following statements about X (all true) would be best evidence in SUPPORTING the argument Y?

Which of the following are restatements of the thesis in article X from a person who disagrees?

Which of the following tests would demonstrate X?

Which of the following develop the thesis of X further?

How might you label the authors X, Y and Z?

What changes might you make to solve X?

What could be done to maximize Y?

How could you change X to make it more Y?

What would happen if…?

Which of the following would be an alternative for X?

 

EVALUATION (presenting and defending judgments)

Which of the following statements about X (all true) would be best evidence in REFUTING the argument Y? (Same set of answers to chose from.)

Which of the following represents the strongest argument for why…?

What fallacies are inherent in argument X?

Which position is more appropriate/valid, logical for X?

Which critique of X is most compelling?

How would you prove X?

Which would be the best defense of X to Y?

What data was most important in conclusion X?

Why did X choose to Y?

What would you recommend…?

Prioritize the following in terms of X?

 

More Examples:

More sample question formats: http://tpri.wikispaces.com/file/view/05-2Bloom-16-17+Stems+for+Instruction.pdf

University of Texas Multiple Choice questions based on Bloom: http://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice-bloom.php

 

Using Multiple Choice Questions Written at Various Bloom’s Taxonomy Levels to Evaluate Student Performance across a Therapeutics Sequence: http://www.pharmacy.umn.edu/innovations/prod/groups/cop/@pub/@cop/@innov/documents/article/cop_article_347107.pdf

University of Oregon, Bloom levels with key words for questions: http://tep.uoregon.edu/resources/assessment/multiplechoicequestions/blooms.html

Dawn Zimmaro, Writing Good Multiple Choice Exams: http://ctl.utexas.edu/assets/Evaluation–Assessment/Writing-Good-Multiple-Choice-Exams-04-28-10.pdf

Article focused on higher level questions: http://www.learningsolutionsmag.com/articles/804/writing-multiple-choice-questions-for-higher-level-thinking

Article from Tagoras: http://www.tagoras.com/docs/Tagoras-Writing-Effective-Multiple-Choice-Questions.pdf

 

Great list of key words: http://www.cbv.ns.ca/sstudies/links/learn/1414.html