Cut scores and proficiency labels.
Education Week has coverage of the cut score announcement, the process used for setting the cut scores, and concerns raised about whether cut scores should have been set using only field-test data (PARCC will set cut scores after administering the operational test).
“It’s really bizarre to set cut scores based on field-test data,” said one state education department psychometrician. “You can’t possibly project” accurately what proportions of students will score at the four levels of the test. He and other assessment experts said that field-test data are not good predictors of performance on the operational test because students are unfamiliar with the test, and often, teachers have had less experience teaching the material that’s being tested.
And students might lack motivation to do their best on a field test, experts said.
Education Week also has coverage about the debate over use and reporting of test scores, particularly reporting test scores in performance (achievement-level) categories as opposed to reporting scale scores. Vermont abstained from voting to set SBAC cut scores and outlined concerns about the use of performance categories, and the lack of empirical evidence for the cut scores, in a memo to SBAC governing states. SBAC covered similar ground in a document titled Interpretation and Use of Scores and Achievement Levels.
Even though the predictions about student performance on the operational assessments may be flawed (based on field-test data only, Iowa students may outperform–or underperform–the multi-state averages), I thought it would be interesting to compare predicted performance on the Smarter Balanced Assessments to reported proficiency data from the Iowa Assessments.
I used SBACs performance predictions for Levels 3 and 4 (proficient) and Level 4 (college content ready at 11th grade) and Iowa Assessments intermediate and high performance levels (proficient) and the high performance level from the 2011-13 biennium (the most recent data reported by the state of Iowa). I chose grades four, eight, and eleven because those are the levels reported in The Annual Condition of Education Report (see pages 176-181).
Here’s how reading proficiency rates could change (predicted SBAC performance versus 2011-2013 Iowa Assessment performance):
Here’s how math proficiency rates could change (predicted SBAC performance versus 2011-2013 Iowa Assessment performance):
Have Iowa proficiency standards been set too low? Are the Smarter Balanced Assessments proficiency standards set too high? Are the predictions grossly inaccurate? Who knows, but if the Iowa Legislature chooses the Smarter Balanced Assessments we had better be prepared for much lower reported proficiency rates, at least in the early years.
Just for fun: draft assessment tasks for the Next Generation Science Standards from Achieve (HT: Education Week). Iowa is expected to adopt the Next Generation Science Standards. If the Iowa Legislature chooses the Smarter Balanced Assessments, Iowa will need to choose a separate science assessment.
Just for fun 2: the State Board of Education will be submitting a recommendation about assessments to the Iowa Legislature. In a demonstration of minimal transparency, the State Board agenda for the November 19th meeting lists “assessment” as an agenda item. Tab M adds the following helpful background description:
This is a continuation of the conversation on the State Board’s priority on assessment. Possible ideas for Board positions and recommendations on assessment will be provided. Opportunities for interaction around these topics will be provided throughout.
I guess we’ll just have to wait for the minutes to be posted following the January 22, 2015 meeting to learn more.