A couple weeks ago, I announced the new Oklahoma Test and Item Specification release for mathematics grades 3-8 and 10. Then I told you not to use them.
The response was mixed.
On one hand, there were responses from folks like Moriah Widener, one of Oklahoma’s most recent Presidential Award winners, responding with support:
Thank you for the reminder that when we teach to a test, we only touch a fraction of student capabilities…and we are bored/boring…and we perpetuate the negative math culture that has kept many a bright mind from the love of math that we wish for for our students. I always enjoy your ideas on math instruction. Thank you so much for sharing your thoughts!
On the other hand, a mix of responses ranging from frustration about the realities of accountability to clarification about the need for content limits. One of my favorites is from Kelly Jackson Brown clarifying why she values the Item Specs:
I’ve been wanting guidance & perspective of how deep to go, guidelines for projects that would at minimum be to the real world ideals that are referenced in the standards. I prefer to bridge them to the next level when I can. I enjoy letting them see how it ties to other areas, cross curricular, or how they might use it as an adult someday soon.
A Few Bonus Thoughts
First, my core belief is that standards should drive instruction and we must intentionally develop instructional assessments (formative, interim, etc.) that increase the visibility of student thinking and learning. Second, I believe that state assessments play a role in accountability, but should never drive standards nor instruction.
And yet, the pressures of our failed accountability systems and the implicit high-stakes of the state assessments (even with the removal of the EOI graduation requirements) still take lead in this tango.
If we are to create learning experiences that promote rich learning of mathematics, we must continually reorient ourselves to the standards. And unfortunately, even standards leave us wanting for more. We’ve drastically improved the clarity of our standards in many regards, but we still sometimes see them as complex riddles that we just can’t crack.
So where do we turn?
As Kelly stated, we look for that guidance and perspective wherever we can. We are all trying to find out how to design instruction that is responsive to the learning trajectory our students are on. And, in the age of high-stakes assessments, the most reliable source for that information has been the Test and Item Specifications. So I get it; we do need guidance and perspective. We need all the help we can get to really dig into the standards to make sense of them and help translate them into instruction.
But I Still Don’t Like Them
And I think it’s because I have a little insight into the work we’ve been doing to help teachers with exactly what Kelly and others are looking for… I think. So, let me introduce to you the OKMath Framework Project!
The OKMath Framework represents a remarkably comprehensive standards analysis that is designed to 1) Increase Teacher Understanding of the Standards and 2) Help Teachers Implement the Standards to their Full Intent.
If I’m being honest, it has taken FOR. EVER. We have over 500 webpages full of guidance on the new standards and that took some serious time to create and upload to the site! Let’s look at a few of the components of the OKMath Framework Project that we believe provide powerful guidance to teachers. There are four main sections of the Framework other than the front page and the grade-level introductions:
- Grade-Level Mathematical Actions and Processes
- Unit Progressions
- Worthwhile Task Indicators/Examples
- Objective Analyses
If you want to skip the reading and watch a 6 minute video about the OKMath Framework instead, have at it! Stick with me here and I’ll walk you through our thinking as we designed each of these components.
1. Introduction and Grade-Level Maps
Central to the new Oklahoma Academic Standards for Mathematics are the Mathematical Actions and Processes. Here’s the introductory statement to them found in the standards document:
The Mathematical Actions and Processes simultaneously reflect the holistic nature of mathematics as a discipline in which patterns and relationships among quantities, numbers, and space are studied (National Academies of Sciences, 2014) and as a form of literacy such that all students are supported in accessing and understanding mathematics for life, for the workplace, for the scientific and technical community, and as a part of cultural heritage (NCTM, 2000). The seven Mathematical Actions and Processes leverage both the NCTM Process Standards and the Five Mathematical Proficiencies (NRC, 2001) to capture the mathematical experience of Oklahoma students as they pursue mathematical literacy.
But what are they? What do they look like in each grade level? The vision is decidedly left open to interpretation. So, the Framework writers took a stab at fixing that. For each grade level, there is now a simple introduction summarizing the significant ideas of that grade pretty much in one breath. (Easier said than done, by the way.) Linked directly from the landing page of each grade are the Grade-Level Mathematical Actions and Processes. They help give a better sense of what a student pursuing mathematically literacy might be doing in your class. For example, let’s look at the 5th grade MAPs:
I love these. Sure, we can probably imagine more and more ways to describe what a 5th grader is doing when she is developing “mathematical reasoning,” but this gives me a really nice launching point to think about how I can make sure this type of experience is central to the work in my classroom. If students are working to “find solutions to problems where no clear-cut strategy is evident,” then I probably need to make sure I’m not just giving my students a bunch of hints and examples followed by cookie-cutter problems that don’t require any reasoning at all. And, if that’s what my textbook suggests, I need to ask the pertinent question: Is my textbook actually aligned to the standards?
I think this alone represents a fundamental distinction between the Item Specifications and the Framework. In assessment design, controlling for variables reigns supreme. The test will be designed on “testable” features of the standards. In contrast, instructional design allows for the variables of the child, the curriculum, and the mathematics. It should be founded upon the intent of the standards and structured flexibly around the experiences we hold to be essential in the full development of mathematical literacy.
2. Unit Progressions
Another stark contrast between the Item Specifications and the Framework is found in the implications for planning, sequencing, and benchmarking. Too often, the blueprints are used to determine the pacing guide of a given curriculum. A content strand that is only 12% of the state assessment gets only 12% of the instructional time during a year (and sometimes less). While this is well-intentioned, it is not necessarily consistent with the belief that mathematics as a discipline is greater than the sum of its parts. This apparent mathematical paradox is provoking, no? The mathematics that results from boiling down the content into disjointed components is narrow and altogether unrealistic when compared to the mathematics that exists in the mathematical applications in the world around us.
So, the curriculum writers worked to rethink the traditional progression by focusing on two priorities: 1) Ensure the grade-level storyline is clear and intact. In other words, the progression should tell a story about what the mathematics allows you to do instead of being a hodgepodge of concepts strewn together. 2) Highlight connections across content strands through bundling. By bundling, we essentially break down some of the artificial barriers placed between Algebra and Algebraic Reasoning (A) and Date and Probability (D), for example. This helps us weave together a more cohesive learning experience that we believe will strengthen students abilities to store and retrieve.
One more point I’m proud of with the progressions… again I’m sure we’ll continue learning together about what sequences, bundles, and timelines are best, most efficient, etc. But we are also trying to strike the balance between too flexible and too rigid. To help give teachers a sense of how much time might be spent on each unit, we usually provide a 2-3 range. Often following the low end of the projected timeline will put a teacher on target for completing all of the grade level content approximately 4-6 weeks before the usual testing window. The midrange and high-end range timelines are designed to put a teacher on target for completing all of the grade level content with only a week or two to spare (approximately 33-35 weeks total). We hope folks who utilize the unit progressions will provide us with continued suggestions for improvement.
3. Worthwhile Task Indicators/Examples
One of the challenges with this type of work is thinking about completeness. What does it mean to really create a complete solution that will meet everyone’s needs? This might cause you, like me, to chuckle out loud… it just ain’t happening! We know that educators need example tasks to really create a more complete understanding of the learning experience students should have with the grade-level mathematics. And until lately, the best place to get an idea about sample tasks related to Oklahoma’s standards has been the Item Specifications. The thing is, we can’t develop instructional tasks and performance assessment items for every grade all at once. So, we tried to take an approach that could provide us and districts in future work.
Before I lay this out, let me say… I think we have a long ways to go on this. We really need folks to spend some time thinking more about how to create a strong vision for the learning experiences students should have so that we, as educators, are better able to filter the good from the bad as we make curricular decisions.
We began this work by first thinking about the work Margaret Smith and Mary Kay Stein have done with cognitive demand (read here, here, and here). Once we have a way of thinking about the types of tasks that students should be interacting with on a regular occasion, we were better able to think about the decisions a teacher might make when they are selecting these tasks. Here’s a little excerpt from some of the research shared at the MathematicalTasks.weebly.com link provided above:
“High cognitive demand tasks involve making connections, analyzing information, and drawing conclusions.” (Smith & Stein, 1998) High-level tasks require students to think abstractly and make connections to mathematical concepts. These tasks can use procedures, but in a way that builds connections to mathematical meanings and understandings. (Stein & Smith, 1998, p.270) “When completing higher demanding tasks students are engaged in a productive struggle, that challenges them to make connections to concepts and to other relevant knowledge.” (Van De Walle, Karp, & Bay-Williams, 2012, p.37)
Deciding what kinds of attributes to look for isn’t easy. If you want students actually making connections, analyzing information, and drawing conclusions, you really have to think about what that looks like for each bundle and then (for the nearly impossible part) find a matching task. For us, the “Worthwhile Task Indicators” are really meant to serve as our first attempt at putting into words what we want kids to do with the mathematics we are focusing on. Essentially, for a task to be worth our time and our students’ time, it has to meet these indicators. Remarkably, this really served as a nice filter allowing us to identify a number of OER tasks that aren’t just aligned, but are connected to the learning experiences we decided we care a lot about. Again I’ll add, we hope as you begin using the OKMath Framework, you will help us refine these indicators so they become a really nice compass as we look for worthwhile tasks.
4. Objective Analyses
Okay… this is really my favorite part!
The Objective Analysis has it all and I want to show it off to you! First, let’s go back to essential argument for why teachers said they like the Item Specifications: guidance and perspective. In my experience, the most likely buckets that most of questions I get about the standards and objectives fall into are 1) What does the standard really mean? and 2) What will the test items look like? Not surprisingly, I get many fewer questions in bucket #2 regarding Geometry and Algebra 2 nowadays. But, the question about what the standard or objective means… this is the essential question that, until now, has only really been addressed by the content limits and clarifications found in the Item Specifications.
Enter, the Objective Analyses.
The last key distinction from the Test and Item Specs is all about how it can be used to inform instruction. The emphasis, stimulus attributes, format, content limits and distractor domain can all be helpful… absolutely. In fact, they really can help give us a sense of what to prioritize as we develop our lessons. The reality though is that they aren’t designed with the intention of helping teachers plan your lesson. That is only a byproduct. Instructionally, we have the license to do whatever it takes to engage our students in the objective but we need guidance that is oriented toward the learning experiences students have, not just their performance experiences on a largely multiple choice test.
Let’s start with looking at one objective from 7th grade, 7.A.4.2. Here’s the Item Specification excerpt for 7.A.4:
Let’s look at one from 7th Grade for comparison and then break it down a bit.
- The Objective – Each Objective Analysis begins by simply restating the objective verbatim from the Oklahoma Academic Standards for Mathematics.
- In a Nutshell – Admittedly, even our best attempts at un-riddling the standards fell short. Here, we try to put the language in terms that are a little more accessible through elaboration, jargon reduction, and occasional clarifications or examples. Even as new questions come in from folks trying to better understand each objective, we will continue to improve each of these.
- Student Actions – Based upon the Mathematical Actions and Processes (see the Grade-Level MAPs section above), we tried to marry learning goals and the actions and processes highlighted in the new standards. One of the most critical shifts found in the new standards is the focus on teaching the Mathematics Actions and Processes through the content rather than an add-on.
- Teacher Actions – Based upon the Effective Teaching Practices (NCTM, 2014), these statements are directed to the teacher to support them in thinking about the kinds of activities they should incorporate in their instruction. In this example, the guidance is simple: Engage students in solving and discussing tasks that promote application and understanding of order of operations. This might seem obvious at first glance, but when you tie it together with the Student Actions and the ultimate goal of ensuring students have had ample opportunities to investigate and learn about how the order of operations work, you can see that it is essential that teachers create opportunities to both solve and discuss tasks.
- Key Understandings – In many ways, the Key Understandings portion of the Objective Analysis serves as the smallest grain-sized view of the objective. This component is designed to enumerate pretty much each tiny component of the objective. This list can be very helpful as teachers think about lesson goals for instruction and assessment, but we do want to caution the use of the Key Understandings as a checklist that might turn mathematics learning into a laundry list of disconnected ideas.
- Misconceptions – My personal favorite, the list of Misconceptions is designed to help teachers think about the points of confusion that often arise and persist related to each objective. Instructionally, these Misconceptions are not meant to be explicitly presented to students, but serve to provide a frame of reference for designing instructional assessments. So often, we can easily fall into the rut of creating items that are designed to assess what students know, but we forget to find out how they are thinking about the things they don’t know.
- A quick example: A standard Order of Operations task might involve asking students to simplify a problem like 1 + 2 – 3 x 4 ÷ 5 x 6 – 7 + 8.
- A strategic version of this problem that better hits the objective might add: “Add grouping symbols to this expression to ensure the calculator will follow the Order of Operations correctly.”
- A challenge version of this problem that helps to elicit evidence of student thinking around the misconceptions while also connecting to a number of other grade-level concepts might look like the following: Add one set of grouping symbols to make the simplified version of this expression as large as possible and as small as possible. (Allow ~5-7 minutes.) Compare your placement with your neighbor and determine the two winning grouping symbol placements. (Allow ~10 minutes.) As the teacher, select 3-4 groups to share increasingly accurate and sophisticated explanations for the placement of the grouping symbols for both versions of the competition.
- Resources – This section is pretty much blank everywhere, but I’m telling you about it anyways. I’m a big-time lover of research and pretty much any resource that is going to help me think more deeply about the mathematics content I teach. If you have a research article or resource that you think might help teachers better understand the objective, please share!
Conclusion
Here’s a reminder of what I said at the beginning:
First, my core belief is that standards should drive instruction and we must intentionally develop instructional assessments (formative, interim, etc.) that increase the visibility of student thinking and learning. Second, I believe that state assessments play a role in accountability, but should never drive standards nor instruction.
I want to affirm everyone who is out there working their tails off trying to bring the best mathematics education experience to their classroom day in and day out. God knows we aren’t doing much in Oklahoma to show our thanks so you should at least hear it from me through my blog. I hope you all continue to look for guidance and perspective, and I hope we at the Oklahoma State Department of Education and as a community of Oklahoma mathematics educators continue to strive to provide the best darn resources we can. I hope you enjoy the OKMath Framework and I hope you have a little better sense of why I’m such a downer when it comes to the Item Specifications.
As always, please let me and Robbyn know if we’re hitting or missing the mark. You can share your thoughts anytime by checking out our contact information at http://sde.ok.gov/sde/mathematics or tagging us in the #OKMath Facebook group.
Hope to hear from you soon!
