Smarter Balanced Assessments 30

Cut scores and proficiency labels.

SBAC announced cuts scores for each of the four achievement levels that will be used for reporting student performance on the Smarter Balanced Assessments earlier this week.

Education Week has coverage of the cut score announcement, the process used for setting the cut scores, and concerns raised about whether cut scores should have been set using only field-test data (PARCC will set cut scores after administering the operational test).

“It’s really bizarre to set cut scores based on field-test data,” said one state education department psychometrician. “You can’t possibly project” accurately what proportions of students will score at the four levels of the test. He and other assessment experts said that field-test data are not good predictors of performance on the operational test because students are unfamiliar with the test, and often, teachers have had less experience teaching the material that’s being tested.

And students might lack motivation to do their best on a field test, experts said.

Education Week also has coverage about the debate over use and reporting of test scores, particularly reporting test scores in performance (achievement-level) categories as opposed to reporting scale scores. Vermont abstained from voting to set SBAC cut scores and outlined concerns about the use of performance categories, and the lack of empirical evidence for the cut scores, in a memo to SBAC governing states. SBAC covered similar ground in a document titled Interpretation and Use of Scores and Achievement Levels.

Even though the predictions about student performance on the operational assessments may be flawed (based on field-test data only, Iowa students may outperform–or underperform–the multi-state averages), I thought it would be interesting to compare predicted performance on the Smarter Balanced Assessments to reported proficiency data from the Iowa Assessments.

I used SBACs performance predictions for Levels 3 and 4 (proficient) and Level 4 (college content ready at 11th grade) and Iowa Assessments intermediate and high performance levels (proficient) and the high performance level from the 2011-13 biennium (the most recent data reported by the state of Iowa). I chose grades four, eight, and eleven because those are the levels reported in The Annual Condition of Education Report (see pages 176-181).

Here’s how reading proficiency rates could change (predicted SBAC performance versus 2011-2013 Iowa Assessment performance):

Reading 2

Here’s how math proficiency rates could change (predicted SBAC performance versus 2011-2013 Iowa Assessment performance):

Math 2

Have Iowa proficiency standards been set too low? Are the Smarter Balanced Assessments proficiency standards set too high? Are the predictions grossly inaccurate? Who knows, but if the Iowa Legislature chooses the Smarter Balanced Assessments we had better be prepared for much lower reported proficiency rates, at least in the early years.

Just for fun: draft assessment tasks for the Next Generation Science Standards from Achieve (HT: Education Week). Iowa is expected to adopt the Next Generation Science Standards. If the Iowa Legislature chooses the Smarter Balanced Assessments, Iowa will need to choose a separate science assessment.

Just for fun 2: the State Board of Education will be submitting a recommendation about assessments to the Iowa Legislature. In a demonstration of minimal transparency, the State Board agenda for the November 19th meeting lists “assessment” as an agenda item. Tab M adds the following helpful background description:

This is a continuation of the conversation on the State Board’s priority on assessment. Possible ideas for Board positions and recommendations on assessment will be provided. Opportunities for interaction around these topics will be provided throughout.

I guess we’ll just have to wait for the minutes to be posted following the January 22, 2015 meeting to learn more.

Smarter Balanced Assessments 29

The Assessment Task Force issued recommendations last week, including a recommendation that the state adopt the Smarter Balanced Assessments as the statewide assessment of student progress in mathematics and reading.

The Gazette, Radio Iowa, and Ed Week have covered the recommendations.

Today, Matt Townsley asks:

Before I take a stab at answering Townsley’s question, some background clarifications might be in order.

First, contrary to Ed Week’s coverage, the Assessment Task Force reviewed the Next Generation Iowa Assessments, in addition to the Smarter Balanced Assessments, not the current Iowa Assessments.

Second, Iowa schools are required to assess science as well as mathematics and reading. The Smarter Balanced Assessments do not include a science component. The Next Generation Iowa Assessments, like the Iowa Assessments, will offer a science and social studies component.

Third, the Smarter Balanced Assessments are expected to take more than twice as long as the Next Generation Iowa Assessments (excluding science and social studies):

-                    Smarter Balanced     NGIA (ELA/Math only)     NGIA (plus science/social studies)

grades 3-5           7 hours                      3 hours                                  5 hours

grades 6-8           7.5 hours                   3 hours                                  5 hours

grades 9-11         8.5 hours                   3 hours                                  5 hours

Which brings me to my first suggestion for possible reasons other than cost to reject the Smarter Balanced Assessments:

Time to Administer

The Washington Post published a guest column earlier this month (HT: McLeod) comparing the length of required assessments for New Jersey 4th graders to the length of the New Jersey bar exam. Spoiler alert: the New Jersey bar exam, at only eleven hours and fifteen minutes, is actually fifteen minutes shorter than the 4th grade exams.

I suppose Iowa can afford to more than double the amount of time spent testing mathematics and reading because our bar exam clocks in at twelve hours or so. And perhaps third grade students should get to work on developing the assessment stamina to be on track to acquire the career readiness to sit for the bar exam, but questions remain. Is data from Smarter Balanced Assessments so much better than NGIA to justify requiring students, some as young as eight years old, to sit for such lengthy assessments? (This question could take years to answer, assuming anyone even bothers to look for the answer.) Are the Smarter Balanced Assessments a better use of instructional time than other instructional programming?

In any case, if seven hours to eight-and-a-half hours doesn’t seem like too much testing, remember that it doesn’t include the required science assessment and, if your school decides to administer the practice exams, you can double those times to fourteen to seventeen hours.

Technology Readiness

Some of the news coverage hints at this issue, but the evidence suggests that Iowa is not ready for statewide online assessments. The Iowa field tests went well but, with only an eight percent participation rate and many schools only testing one or two grade levels, some in only one subject area, they could hardly be said to have put Iowa’s school technology infrastructure to the test.

In 2013, it was estimated that “a need exists for greater bandwidth in about one-third of Iowa school districts.” In addition, some 1:1 districts were already exceeding ICN bandwidth capacities. Remember that the Legislature did not pass the broadband bill and it is unclear whether any progress has been made.

It might also be worth considering Michigan’s experience with Smarter Balanced Assessments, recently reported in an Ed Week article on waivers of state requirements for online testing:

In Michigan, a report released earlier this year found that while nearly 80 percent of schools did meet the “minimum” technology-readiness standards put forward by Smarter Balanced—one of the two main consortia creating online assessments aligned to the common core—far fewer school systems met the consortium’s “recommended” specification.

And even the recommended standards represented a lower tech threshold than what state officials believed would be necessary, director of the state’s office of standards and assessment Vince Dean told Cavanagh in an interview following the report’s release.

Over the past two years, Michigan has spent more than $100 million to support district technology improvements and professional development efforts, including those surrounding the transition to online testing.

So, we don’t know with any certainty how close to or far from being ready for statewide online assessments we are and we are running short of time to prepare. If the Legislature takes action in the upcoming session (spring 2015), we would essentially have just over one year (to fall 2016) to prepare all Iowa schools for online assessments–assuming that we want all schools to have an equal opportunity to use the Smarter Balanced Assessments interim assessments throughout the 2016-2017 school year in preparation for online administration of the summative assessments in spring 2017.

Equity

If we can’t resolve the technology readiness issues, students across the state could have very different testing experiences, perhaps to the point that results can’t fairly be compared. Consider the following possibilities:

  • Some students must take a longer, non-adaptive, paper-and-pencil format of the assessment in the spring of 2017 while other students take the computer adaptive version.
  • Some students are bused to the community college or other locations and must sit for the assessments in one or two sessions while other students take the assessments in their own classroom or school computer lab in shorter sessions spread out over two weeks.
  • Some students experience slow loading times or interruptions of the test due to insufficient bandwidth while others students take the assessments without interruption.

How about some other possible inequities?

  • Some students experience cuts in art, music, world languages, and other instructional programming and/or larger class sizes to pay for the assessments and the technology required to support them while others students see no changes in instructional programming or class sizes.
  • Some students experience reductions in art, music, or recess to make time for technology instruction to prepare them for typing in constructed responses and otherwise navigating the assessment software while other students, with better access to technology outside of school, see no changes in instructional programming.

Priorities

Those last few possible inequities are in some sense really about priorities. There are some eager to see statewide online assessment force districts into 1:1 computing environments or at least offering much more technology in the classroom. However, not everyone agrees that this should be a priority or that more technology is the answer to the question “what is a good (or great) education?” See any number of articles about Steve Jobs not giving his own children iPads. Or see StepfordTO’s thoughtful blogpost The Debate About Digital Literacy: Moral Panics, Contradictions, and Assumptions. Here’s a taste (but please go read the whole thing):

My own anxiety as a parent has to do with what the anxious rhetoric surrounding digital literacy can lead us to do. And by us, I mean parents and schools and governments. One thing that it has led us to do is to spend a lot of money on technology for schools, even though the research to date has failed to show a significant impact (good or bad) on learning. I’m not opposed to technology in schools, but when the provincial government announces that it will be spending 150 million dollars to put iPads in classrooms, while it’s making cuts elsewhere in education, it gives me pause.

If seven and eight year olds are going to be pushed to learn how to type, not because it has any value to the seven or eight year olds, but simply to prepare them for the Smarter Balanced Assessments, count me out.

Ask me to choose between higher percentage of Supplemental State Aid or money to pay for the Smarter Balanced Assessments, I’ll pick the Supplemental State Aid to support the art, music, and other great instructional programming that makes for a great education.

Ask me to choose between long-needed air conditioning upgrades and building projects to finally retire portable classrooms in use for decades and technology for assessments, I’ll pick the air conditioning and the building projects.

Which I suppose brings us right back to cost as a reason for rejecting the Smarter Balanced Assessments, but in the end there’s no avoiding the choices the costs will force us to make.

New Iowa Core Website

The Iowa Department of Education has launched a new Iowa Core website.

The website includes a series of parent’s guides to the Iowa Core for kindergarten through grade eight, and high school.

The educator resources page includes links to Iowa Core mathematics support materials, which include documents explaining the Iowa Core mathematics content shifts from prior practice for high school and grades six through eight, and content and practice shifts for kindergarten through grade five. At first glance, these documents look more accessible than reading the standards themselves, at least for understanding how things might be changing under the Iowa Core, as well as for understanding how the Iowa DE is interpreting the the common core standards.

The educator resources also include a link to the IowaLearns digital library of teaching and learning resources, thousands of which are available to the general public through a guest login.

With the inclusion of an Iowa Core Spotlight containing statements from Iowa Core supporters, the new website is clearly intended to promote a positive view of the Iowa Core, but it does seem to be an improvement over prior websites in terms of pulling Iowa Core resources together, in a way that they may be more easily found.

 

 

State Education Official Elections

This ad makes me somewhat sorry that we don’t elect the Director of the Iowa Department of Education.

Aside from school board, there are no elections in Iowa that focus entirely on education issues. With so many decisions with regard to public schools in Iowa being made at the state level, it might be interesting to see which issues emerged as campaign issues and to see how Iowans weigh in on them specifically.

On the other hand, maybe this ad just seemed interesting as a break from all the Bruce Braley and Joni Ernst ads. . . .

If This Is Close Reading, Count Us Out

Dan Willingham recently addressed close reading in a column at Real Clear Education, in which he noted that ““Close reading” has become strongly associated with the Common Core State Standards, as it’s touted as the reading technique that will allow students to get out of texts what they are meant to (and hence, score well on Common Core-aligned assessments).”

Willingham’s commentary focuses on close reading treating texts as if they are self-contained, ultimately concluding:

Still, it seems a valid question to ask whether this artificial type of reading is likely to be useful in students’ ongoing work in and out of the classroom. Except in very restricted academic settings — that is, among people who like close reading — it’s not obvious to me how this sort of reading will serve students well.

Careful study of language, focus on the author’s words, assumption that rereading pays off: yes. Excluding knowledge outside of the text: no.

So I was interested to find a video and other materials on close reading in an online collection of professional development materials meant to assist teachers in implementing the Common Core. Unfortunately, user agreements for that collection prohibit sharing of those materials on any other websites. Fortunately, the owners of the video have also made it available on YouTube, making it possible to share it with you here.

This video shows portions of a lesson devoted to a close reading of the story of the inventors of Magic Rocks. Comments during the lesson, and in the companion video showing the class during an earlier close reading lesson (embedded below), indicate that the students have been closely reading a series of stories on the inventors of toys including Silly Putty, Twister, Lego, Mr. Potato Head, and Slinky. Students are apparently (according to the companion video) supposed to be determining “what character traits are essential to being an inventor” in addition to using their close reading skills.

I don’t disagree with Willingham’s conclusion above that there are problems with treating texts as self-contained and, as I was reviewing the videos, I was particularly struck by this image from the first video:

close reading unknown words

Although the teacher acknowledges the use of background knowledge during several points in these videos, these students are not being directed to use dictionaries (or glossaries, for that matter) to determine the meaning of unknown words (perhaps because they exist outside of the text?). This strikes me, as a person who still keeps dictionaries conveniently located near the places I tend to sit and read, as an enormous disservice to these students. This disservice is evident in the second video (1:34 to 4:33) in which the teacher pretends not to understand the word stabilizing but guesses at its meaning through the root “sta-” meaning to stand or to stay and the reading of several more sentences of the text indicating that ships pitch, plunge, and rock every which way. It is also evident in the nonsensical discussion she has with a student in the first video (beginning at 1:10) in which she leads the child to explain that he has determined that the word composition (used in the phrase “chemical composition”) means formula because the other words nearby include “study” and “microscope.” Umm, obviously.

In any case, skilled use of a dictionary could resolve the matter of unknown words in short order while the students could be reminded to use context to choose which definition of a word the author meant (see, for example, the use of the word “pitch” above).

But these videos raise other issues for me as well.

First, if reading logs didn’t already make your child think reading is a tedious chore, close reading just might convince them. My eight year old couldn’t look away from the first video but also commented throughout, “I could not go to that school. It is like a meeting, a boring meeting.”

Second, the fact that the students in these videos appear to be absolutely engaged by close reading seems to me to be a testament to the capacity of many children to be compliant. But these children are engaged with the close reading of what strikes me as absolutely trivial material. Multiple class periods devoted to the study of toy inventors is a waste of instructional time that could be devoted to much less trivial content. This is a real problem when skills are elevated over content. It may also be a sign of disrespect to child readers, who may have been assumed to be incapable of interest in more substantial topics in history, including the biographies of inventors whose inventions have substantially affected human history and/or civilization.

Smarter Balanced Assessments 28

College Content Readiness

While preparing to draft yet another long overdue and perhaps forthcoming soon Smarter Balanced Assessments post, I stumbled upon Comments on Revised Achievement Level Descriptors dated February 21, 2013. This document contains comments from faculty at the University of Iowa, the University of Northern Iowa, Iowa State University, and Hawkeye Community College regarding the SBAC achievement level descriptors.

I have no idea if any of these comments have been, or will be, adequately addressed by SBAC as they work on setting cut scores for achievement level descriptors this fall, but they raise important issues about the value of the SBAC “college content readiness” designation.

The comments indicate that there is a significant mismatch between the SBAC definition of college content readiness in math (prepared for College Algebra, designated as Math:1005 (22M:008) at the University of Iowa) and Iowa regents institutions, at which many majors have higher level mathematics courses as expected entry-level, first year coursework (see a listing of here for the University of Iowa). The college content readiness mismatch is so great that UNI suggested that the term “high school fluent” be used instead (Comments, p. 9); ISU suggested that students be designated “proficient for 12th grade work” rather than college ready, at least until Smarter Balanced Assessments have been in use long enough to determine whether the assessments are in fact accurately predicting readiness for entry-level, credit bearing coursework (Comments, p. 18).

Another issue highlighted in the comments is that the Smarter Balanced Assessments will not be assessing the “Plus Standards”, which are the additional Common Core standards intended to prepare students for coursework in calculus, advanced statistics, or discrete mathematics (SBAC Math ALDs, p. x). Even if the Smarter Balanced Assessments level 3 and 4 achievement level descriptor cut scores accurately predict readiness for college algebra (which remains to be seen), they will provide no guidance to parents and students as to whether the student is adequately prepared for the higher level mathematics courses STEM/business majors are expected to be prepared to take their first semester in order to graduate in four years.

Your Robocall Needs Work

Scott McLeod is celebrating the 8th anniversary of his blog, Dangerously Irrelevant, with Leadership Day 2014. Last year I contributed a post on school and district websites, Your Website Needs Work.

I know that there are a lot of interesting discussions to be had about technology in the classroom, but as a parent, I would urge school administrators–as technology leaders in their buildings and districts–to allocate some time to thinking about how technology can be used to facilitate (or hamper!) communication with parents and the broader community.

Dear School Administrator,

Last night, for the first time since my child has been enrolled in your school, I hung up on a robocall from you. I hated to do it, but it was the third time in a week that you had called to remind me about the ice cream social and experienced technical difficulties with recording or transmitting your message. Specifically, part way through your message, the message began again at the greeting. Earlier in the week, I listened to your message restart twice(!) so that I could hear it all the way to the end. But last night, I’d had enough and suspected that there was no new information I would miss out on if I hung up.

I am genuinely sorry that it has come to this, that your number popping up on my caller-id screen is starting to leave me with the same feeling of dread with which I face calls from telemarketers and well-meaning campaign volunteers reminding me to get out and vote.

So I want you to know that my child has had a great experience at your school and we are looking forward to another great school year, which is another way of saying that we have positive feelings about you and your school and we want to keep it that way! Improving your robocall (and other technology-facilitated communication) skills should be on your to-do list this year, and here’s why: because my child is doing well at your school (and because I use Twitter but not Facebook and you use Facebook but not Twitter!), robocalls are the only communication I have ever had from you.

Even before the recent technical difficulties, your robocalls were overly long and overly frequent (though I admit, my perspective on this may have been colored by the sheer number of weather-related delays, early outs, and cancellations we had this past school year).

The length, quality, and frequency of robocalls seems like an ideal topic to take up with your PLN (you do have one, don’t you?). Perhaps they would be willing to listen to a few of your recordings and give you constructive feedback on how they might be improved. From my perspective, I will offer that while I appreciate your efforts to sound warm, friendly, and welcoming, that robocalls should probably be approached like all other voice mail messages–because this is essentially what they are–so please, be short and to the point.

Perhaps they would also be willing to share how often and for what purposes they find it effective to use robocalls to communicate with parents.

Perhaps, they could offer other suggestions for using technology to communicate with parents, like effective use of e-mail and Twitter, neither of which you use to communicate with me. (Hint: you can use Twitter to post links to the school’s Facebook page and e-mail is good for longer, more detailed messages).

Don’t overlook teacher-librarians, either, if you don’t have any in your PLN. The ones in our district are tech-savvy and probably could offer some suggestions for improving your technology-facilitated communication with parents–I know, because I follow several of them on Twitter!

Finally, without delay, please talk to someone to resolve your technical difficulties with robocalling before you record your next message. Then I can look forward to answering my phone to find out from you what’s going on at school.

Sincerely, Karen W

Follow Leadership Day 2014 on Twitter at #leadershipday14