Mastery-Based Exams Are Self-Evidently Better than Traditional Exams

Self-Evidently Better

Welcome to the inaugural post on the Mastery-Based Testing Blog. We are all excited to share an assessment technique that many of us find to be superior (at least one of us self-evidently so) to traditional exams. (I should credit George McNulty at the University of South Carolina for introducing me to the idea by way of subjecting me to the exams as a graduate student.) What follows is an essay form of a short talk I’ve given at a few national conferences. My aim here is to give a brief overview of the method (others will speak in more detail about specific implementations) and share several aspects in which you and your students may find it preferable to traditional points-based exams.

A brief word about the title: It was intentionally chosen to be half facetious and half sincere. On the one hand, I lack the evidence to make a statistically significant case that this method promotes student gains, so my defense is to shame the reader for demanding such evidence; the claim stands on its own. (This is where you struggle to gratify me with brief chuckle.) More seriously, I think the aspects in which the method shines are varied and even somewhat nebulous, as I hope to demonstrate. To choose a single metric like improved grades or increased retention is, I claim, to miss the point somewhat. Large-scale studies on best practices are certainly worthwhile, but the end result is a claim about effectiveness that has been averaged across many different institutional missions, instructional strategies, student bodies, etc. As you read my thoughts on mastery-based exams, consider the impact you think its implementation would have at your institution, in your courses, and with your students. I hope that thinking about your particular details will help make it “self-evident” to you, at least, that the idea is worth exploring further.

What Is It?

As I mentioned in the introduction, my aim is to give only the barest sense of the method. My colleagues will provide more details on specific implementations. Among all the implementations, however, there appear to be three essential characteristics.

  • Clear content objectives
  • Credit only for eventual mastery
  • Multiple attempts with complete forgiveness

Clear Content Objectives

In my study guides, I partition course content into roughly sixteen big ideas, each consisting of a few related subtopics. For example, under the heading of “Limits” in multivariable calculus, my students would find:

  • Prove a limit does not exist by exhibiting two paths of approach with different limiting behavior.
  • Prove a limit exists by appealing to continuity.
  • Prove a limit exists using the Squeeze Theorem.

I feel that a student needs to demonstrate proficiency in all these situations before claiming to be a “master” of the concept of a multivariable limit. I accomplish this by including a single question on limits consisting of three parts, one for each item. There is no hint as to which part requires which technique, since selecting an appropriate tool for a new problem is itself a skill I want to develop.

Credit Only For Eventual Mastery

There is no partial credit in a mastery-based exam; a question is either mastered or it is not. What does it mean to master a question? One of the refreshing aspects of the method is that “mastery” is at the discretion of the instructor. My standard tends to center around a single question:

Will the student benefit from studying the objective again?

In the multivariable limit example, consider a squeeze theorem proof that goes off the rails early due to some absent-minded factoring error, but otherwise correctly applies the theorem. In my opinion, this problem is mastered; the student has demonstrated proficiency with the intended idea. Contrast this with a proof where the student divines the correct limit, but does not justify the bounds used in the proof. This student would most likely benefit from taking a second look at how inequalities allow the squeeze theorem to accomplish its goal.

Multiple Attempts With Complete Forgiveness

To allow students time to incorporate instructor feedback and progress toward mastery, it is important to allow multiple attempts on each big idea. Moreover, to emphasize our desire for eventual (rather than immediate) mastery, I believe it is crucial that previous failed attempts carry no penalty. There are many creative ways of accomplishing these goals, but my exam structure typically resembles the following:

  • Test 1: Objectives 1 – 4
  • Test 2: Objectives 1 – 8
  • Test 3: Objectives 1 – 12
  • Test 4: Objectives 1 – 16
  • Final Exam: Objectives 1 – 16

(For Tests 3 and 4, I often split the old questions from the new and hold the test over two days to give students more time to work.) So, for example, a question addressing Objective 1 appears on every exam. The versions appearing on each exam are different enough from one another that rote memorization is no help, but they are similar enough that they are clearly addressing the same objective. Under this exam structure, a student could fail to master Objective 1 four times without penalty, only to display true mastery on the final exam and earn full credit. At the other extreme, a student may display mastery on the first attempt and simply ignore the alternate versions appearing on subsequent exams. A student need only display mastery one time to earn credit for the objective. (I emphasize this point to stress that no student is attempting to solve sixteen calculus problems in an hour. Each student approaches an exam with a personal list of 3 – 5 objectives they hope to master on a given attempt.)

Armchair Pedagogy

It is an unfortunate reality that many of our students will not or cannot devote as much time as we might like to mathematical study. How does the structure of our assessment impact the way in which these precious hours are used? In the following sections, I hope to demonstrate some important ways in which I feel a mastery-based exam surpasses a traditional points-based exam.

Depth of Knowledge
Points: Understand everything superficially.
Mastery: Understand some concepts in great detail.

In most points-based systems, a blank exam question is a heavy blow to a student’s grade. On the other hand, a student who provides a couple relevant formulas and something resembling the beginning of a solution may receive half credit or more. In the presence of constrained study time, a good strategy is to learn some basics about every test item. Such a student may earn half credit on most items together with a few lucky shots on easier items, which amounts to a passing grade overall. Take a moment to consider whether this experience has adequately prepared the student to apply mathematical thinking to nontrivial problems in the future.

The “broad and superficial” strategy employed above earns no credit under a mastery-based system. Instead, a student who wishes to earn a passing exam grade must fully understand an appreciable subset of the main ideas of the course, and a student wishing to earn an A grade must fully understand most or all of the main ideas of the course. Even if students spend no time studying a particular item, I contend that the experience of pursuing deep understanding on the other items leaves them in a stronger position to engage deeply with the troublesome topic when it is needed in the future. Moreover, depth of understanding is critical to one’s ability to apply existing mathematical knowledge in novel domains.

Meaningful Office Hours
Points: “How can I easily pick up some points?”
Mastery: “What should I study to fully understand this concept?”

Making grades a function of understanding rather than accumulated trivia lends itself to more meaningful discussions during office hours. A student seeking only to gain more points asks superficial questions with easily-memorized answers (items that are readily available in the textbook). A student seeking mastery must be willing to reflect upon their partial understanding, ask pertinent questions that address their current misunderstanding, and synthesize the conversation into a holistic approach to the concept.

Perseverance
Points: Try a problem once (maybe twice) and hope for the best.
Mastery: Keep trying until you succeed (and I know you can).

One might make the case (and I frequently do) that even students who are far removed from science will benefit from mathematical study because it is exceptionally effective in training students to persevere in solving complex problems. Points-based assessment undermines this valuable experience; a student can often obtain a passing grade without working even a single problem to completion. Indeed, even those who might take a second look at a challenging exam problem may not have incentive to do so if it does not appear on the final. To contrast, a mastery-based exam requires that a student display satisfactory depth of understanding to receive any credit, and this may take several attempts. It falls on the instructor to engender a classroom atmosphere in which these multiple attempts are seen not as shortcomings, but rather as the very essence of deep learning.

Timely Review
Points: Review key concepts for the final (maybe).
Mastery: Review key concepts now.

A points-based system can provide a perverse incentive to ignore key early concepts. As an example, I ask students to provide the vector equations of lines and planes to help develop their intuition about the geometry of vector operations. Under a typical points based-system, a student does not directly benefit from revisiting these foundational concepts. Rather, they are encouraged to press on toward of superficial understanding of applications of that concept, which is both counterproductive and meaningless.

A mastery-based system gives students an immediate reason to revisit important concepts early in the semester, since they will have an opportunity to master it on the very next exam. Continuing with the same example, I did, in fact, recently work with a student who demonstrated poor understanding of vector geometry on the first test only to master the concept on the second. After receiving his graded test, he remarked that studying for that introductory question helped him better understand the new content. It seems understanding vectors is essential to understanding vector calculus.

Growth-Mindedness
Points: Failure is undesirable and incurs penalties.
Mastery: Failure is an opportunity to improve understanding.

The effect of testing on growth-mindedness is, in my estimation, one of the most important facets of this discussion. A points-based system sets arbitrary deadlines by which time perfection must be attained or else penalties are incurred. Each helpful remark from the instructor is coupled with the sting of a progressively lower grade; the more helpful the remark, the greater the deduction. A mastery-based grader can include plenty of penalty-free insight to help the student improve their understanding. Such feedback is actively desired by the student and is sure to be studied since subsequent exams will provide opportunities to earn the missing credit. The message of the mastery-based exam is well-aligned with the development of a growth mindset: “You don’t understand this concept yet, so here’s some advice on how to improve. Come back next time and show me what you’ve learned.”

Reduced Test Anxiety
Points: Every test has the potential to “ruin” GPA.
Mastery: No one test can harm grade.

Among all the items on this list, this one seems to vary most wildly across departments. My students frequently tell me that being able to try questions multiple times with no penalty considerably reduces their test anxiety. Some of my colleagues, on the other hand, report student frustration when they receive feedback on several near misses (which receive no credit) on their exam. There is no panacea for student discontent, but my advice is to hear their complaints while frequently (and gently) insisting that you think this obscure testing scheme is better for them in the long run. I also suggest you keep an eye on your gradebook. If a student is making unsatisfactory progress, consider a short office meeting where you and the student map out a plan of study for the rest of the semester. While I still contend that the general student reception of mastery-based testing is positive, a student with only one problem mastered at midterm is likely to be feeling quite hopeless. A short pep talk and a realistic plan to climb out of the pit will go a long way.

Formative Assessment
Points: How many points is this error worth?
Mastery: Will the student benefit from studying the concept again?

Points-based grading is inherently punitive; one must examine a proposed solution looking for opportunities to deduct credit. Moreover, the resulting feedback may not be terribly meaningful. Imagine two students computing the volume of a solid of revolution. One arrives at the correct integral but makes several errors in its evaluation. The other begins with the wrong integral but evaluates it perfectly. A points-based grader must decide how many points to deduct for each of these very different sorts of errors. On a mastery-based exam, one rather asks whether the student will benefit from additional study. Here the answer is clear and meaningful. Computational errors are worth an admonishment to be more careful in the future, but they do not merit additional time spent studying solids of revolution. The fundamental conceptual error, however, may indeed suggest to the grader that the core ideas have not been understood, and the student can be directed to revisit the topic.

Faster Grading
Points: Good answers are carefully checked. Point-grubbing is incentivized.
Mastery: Mastery is spotted instantly. All attempts are genuine.

That the ever-growing mastery-based test is faster to grade than the static points-based test seems implausible, but experience bears it out. In a points-based system, a proposed solution has to be carefully scrutinized to determine whether it is worth 6/10 or 7/10 points. Even worse, one has to wade through the mire of 2/10 solutions as students desperately fish for points. Most questions on a mastery-based exam, by contrast, are either left blank or are graced with complete, correct solutions. Both sorts of questions can be graded instantaneously.

The Plural of Anecdote

Several of my colleagues (many of whom are helping to develop this blog) issued a student experience survey at the end of their mastery-based course. The following data reflects some highlights generated by 140 students (both majors and non-majors) across six institutions. We hope to pursue a larger, formal study, but these blips of data give us hope for the time being.

  • The exams deepened my understanding of the ideas in this course. (80% agreement)
  • The results of my exams accurately reflect my knowledge. (77% agreement)
  • I feel prepared to approach a wide range of problems in this course. (75% agreement)
  • I was anxious before the exams in this course. (36% disagreement)

Give It a Try

Those of us developing this blog feel that the mastery-based examination is a superior form of assessment with much to offer teachers and learners of mathematics. I encourage you to read the experiences others are sharing. We are also compiling a collection of resources to help you implement the technique as painlessly as possible. Finally, if you agree with, disagree with, or wish to supplement anything I have said in this article, we would all love to hear your thoughts in the comments. This community looks forward to your continued interest.

Advertisements