Smarter Balanced Assessments 35

EdWeek reported yesterday that after about four weeks of problems, “Montana Lets Schools Cancel Smarter Balanced Testing After Technical Woes“.

In a statement issued April 15, Measured Progress [,Montana’s vendor for the Smarter Balanced assessments,] said it was working with Smarter Balanced to fix the problems in its three client states[.]

“Measured Progress increased server capacity well beyond the specifications provided by Smarter Balanced and its platform vendor. However, even with the increased number of servers, the platform does not support the number of students currently accessing the system.” the organization said in the statement. (The “platform vendor” is a reference to AIR.) (Emphasis added.)

Honestly, it is hard not to suspect that statewide online assessments are more of a technical challenge than its supporters really want to acknowledge. Note that the article indicates that Montana has previous experience with both the vendor, Measured Progress, and the Smarter Balanced assessments.

Montana got permission from the U.S. Department of Education to use Smarter Balanced field tests as a replacement for their typical state exams last year. Given that, I asked why that experience hadn’t better prepared the state for the tests, and she responded, “We were prepared. You would think that would be the way it goes.”

Bandwidth Survey

The review of district-reported bandwidth referenced in the answers submitted by the Assessment Task Force to the Legislature appears to be this Preliminary Report on Bandwidth in Iowa Schools.

The full report paints a more complicated picture of technology readiness, especially if we want schools to be able to use technology for other purposes during the testing window.

From page 3:

Many of our schools are not currently able to deliver the necessary level of technology for even moderate levels of technology use by students. Preliminary reports suggest that as much as two out of five schools in the state lack the bandwidth necessary to adequately support current and upcoming needs.

More from page 3, on technology capacity:

There are several key indicators [of technology capacity]. One is the amount of available bandwidth, which will be the focus of this document. Another involves the number of available computers or other devices that meet basic technical requirements. Additional concerns include ensuring that wireless and wired networks are configured to support all users, making sure schools can support infrastructure maintenance and upgrades, and determining whether the district has the staff and skills to support the technology. With the exception of the bandwidth indicator, the remainder of these issues cannot be addressed in this document because of the scarcity of complete information for all Iowa schools.

From page 6:

The same BEDS data collection that resulted in the bandwidth results summarized above also included a simple count of computer devices available. This report does not provide enough information to create a meaningful summary of hardware readiness. We do not know what sort of devices were counted, nor what technical standards they met. The hardware/software requirements for the Smarter Balanced Assessments are detailed enough that it will not be feasible to capture via a simple data collection or survey.

More from page 6, the summary of currently available information:

The same BEDS data collection that resulted in the bandwidth results summarized above also included a simple count of computer devices available. This report does not provide enough information to create a meaningful summary of hardware readiness. We do not know what sort of devices were counted, nor what technical standards they met. The hardware/software requirements for the Smarter Balanced Assessments are detailed enough that it will not be feasible to capture via a simple data collection or survey.

And more from page 6 and 7, on the recommendations for further study:

In order to have a clearer picture of the technology needs in Iowa schools, it will be important to collect more detailed information about the current state of building networks, available bandwidth, computer hardware and software. There also needs to be some evaluation of other aspects of network and technology capacity, including how the network is constructed, how data are aggregated as they flow into the Internet, and other issues too complicated to handle via a survey. It would be a good idea to evaluate technology readiness and capacity for all current and future education initiatives. It is recommended that the Iowa Department of Education staff work with the technology directors of the Area Education Agencies and school district technology staff to identify a group with the expertise to design and carry out a more thorough collection of information about technology readiness in Iowa schools. It would be best for all Iowa children to include the accredited nonpublic schools in this data collection effort because they will also need to be able to deliver the Smarter Balanced assessments, use the Iowa TIER system, and participate in other technology efforts as they educate the students in their charge. Resources from SBAC and ESH (among others) will be helpful in supporting this effort.

Once we understand the current capabilities of our schools, we will be in a better place to plan for improvement and sustainability of ongoing improvements to technology in Iowa schools. A thorough examination of meeting today’s and tomorrow’s needs will ensure the state’s education efforts will be maximized, giving Iowa’s students the best 21st century education possible.


The Long and Winding Road to the Smarter Balanced Assessments

The second funnel deadline for the Iowa Legislature is today and neither of the assessment bills (HF 446 or SF 429) will survive it. So despite the zealous advocacy of the Assessment Task Force, the State Board of Education, and the Education Coalition (Iowa Association of School Boards, School Administrators of Iowa, Iowa Area Education Agencies, Iowa State Education Association, Urban Education Network of Iowa, and Rural School Advocates of Iowa) and who knows who else, it appears that the Iowa Legislature will take no action regarding statewide assessments this session.

How did we get here: the executive branch loves SBAC, the legislative branch loves it not (apparently).

Governor Branstad, then State Board of Education President Rosie Hussey, and then DE Director Jason Glass signed off on making Iowa an SBAC governing state in June 2011. The letter requesting the change in status updated the SBAC MOU originally signed by Governor Chet Culver, interim DE Director Kevin Fangman, and Rosie Hussey in June 2010. The MOU contains the following language with regard to the Smarter Balanced Assessments: “The purpose of [the SBAC MOU] is to . . . (h) Bind each State in the Consortium to every statement and assurance made in the application . . . ” and “Each State that is a member of the Consortium in 2014-2015 also agrees to the following: . . . Fully implement statewide the Consortium summative assessment in grades 3-8 and high school for both mathematics and English language arts no later than the 2014-2015 school year, . . . .”

Thus, it would appear that the executive branch had committed Iowa to implementing the Smarter Balanced assessments during the 2014-2015 school year.

However, 2012 brought SF 2284, Division II of which fixed the Iowa Assessments as the statewide assessments for Iowa.  The state board was permitted to submit recommendations for modifying the assessment, but legislative action would be required to adopt the Smarter Balanced Assessments.  [SF 2284 is found in Chapter 1119 of the 2012 Acts and Joint Resolutions, which begins on page 434.  Division II of SF 2284 is on page 435.]

2013 brought further changes with HF 215, Division V of which allowed for a successor assessment administered by the same assessment provider (Iowa Testing Programs) and modified the assessment requirements as follows:  Beginning with the 2016-17 school year, districts will be required to administer assessments to all students enrolled in grades three through eleven.  The assessments shall be administered during the last quarter of the school year, must be aligned with the Iowa common core standards, must accurately describe student achievement and growth for accountability purposes, and must measure student progress toward college or career readiness. [HF 215 is found in Chapter 121 of the 2013 Acts and Joint Resolutions. Division V begins on page 13.]

HF 215 also directed the director of the DE to establish an assessment task force to review and make recommendations for a statewide assessment of student progress. The task force began working in October 2013.

Meanwhile, some Iowa schools participated in SBAC pilot tests in spring 2013 and SBAC field tests in spring 2014.

In July 2014, Governor Branstad and DE Director Brad Buck sent a joint letter to SBAC, stating in part:

How to best measure the academic performance of Iowa students is an important conversation under way in Iowa. The Iowa Assessment Taskforce established by the 2013 Iowa Legislature has been studying the state’s academic assessment needs, including past, present and future options for accountability. Taskforce recommendations are expected by Jan. 1, 2015.

To honor the work of the taskforce, Iowa will not sign a new Memorandum of Understanding with the Smarter Balanced Assessment Consortium as requested.

The Assessment Task Force submitted a report and recommendations, including a recommendation that the Smarter Balanced assessments be adopted as the statewide assessment of student progress in mathematics and reading, on December 31, 2014.

In February 2015, the State Board of Education endorsed the Assessment Task Force recommendation to adopt the Smarter Balanced assessments.

Senate and House assessment bills (HF 446 and SF 429) were passed out of respective education committees earlier this session so that the assessment issue would survive the first funnel deadline. Meetings about the statewide assessment issue were held by the Senate Education Committee and the House Education Committee on March 18th and 25th.

And then, somewhat unexpectedly, though questions about the effect of a delay had been raised, legislators have taken no further action and the assessment bills, as noted above, are effectively dead for this session. One might assume that legislative support was insufficient to assure passage of a bill adopting the Smarter Balanced assessments at this time.

What happens now: your guess is as good as mine.

It is too soon to despair or rejoice (depending upon your preferred assessment outcome) as there may yet be a path to Smarter Balanced assessments in Iowa. (District IT staff may want to go ahead and despair the uncertainty and lack of additional funding for school technology infrastructure). Here are a few scenarios to consider:

One: After time to think, twist arms, or otherwise make sausage, legislators return in January 2016 and vote to adopt the Smarter Balanced assessments–either for the 2016-2017 school year or with a delay to give districts more than a few months notice to get prepared for computer-based assessments.

Or two: The State Board of Education adopts the Smarter Balanced assessments through the administrative rule-making process–either in 2015, risking a legislative backlash that undoes the adoption of the Smarter Balanced assessments in January 2016, or after the 2016 session.

The (unsubstantiated) word is that there is a legal theory being floated that the State Board of Education or the DE has the authority to choose a new assessment if the Legislature fails to act.

I haven’t yet heard the details of the theory, so I won’t comment on the quality of it, but I do think there is an argument to be made that the Legislature has acted on this issue. The Legislature directed the State Board of Education to adopt rules to make the Iowa Assessments or a successor assessment administered by the same assessment provider (ITP) the statewide assessment of student progress on the core academic indicators of mathematics, reading, and science. The successor assessment that will be administered by the same assessment provider (the Next Generation Iowa Assessments) meets the minimum legal requirements that take effect for the 2016-2017 school year. No conflict, no further action by the Legislature needed.

Does No One Else Read YA Dystopian Fiction?

Broadband Matters, an awareness initiative of the Iowa Communications Network, tweeted a link to Connecting the Classroom with the Internet of Things earlier today.

While the tweet mentions a fairly innocuous sounding method of taking attendance (though see RFID chip tracking of students), the smart band described in the article “uses ECG patterns to authenticate identity.” There is also a suggestion that teachers could use EEG technology to measure student brain activity to decide which students need more attention. Or teachers could use technology to “map[] the record of behavioral incidents against a student’s heart rate.” Because apparently adults cannot be expected to effectively teach children without the benefit of monitoring and analyzing their heart rates and brain activity, at least not in the 21st century.

ATF: State Board Member Miller Weighs In

State Board of Education member Mary Ellen Miller, who participated in the task force presentations to the Senate and House Education Committees earlier this month, has a guest column advocating the adoption of the Smarter Balanced assessments in The Gazette. The guest column is a response to the Gazette staff editorial SMARTER BALANCED: Recommended assessment for Iowa’s K-12 students carries a hefty price tag, and for what?

Miller describes the staff editorial as second guessing the task force recommendations, as if that is a bad or impermissible thing for the Gazette staff to do. The task force undertook our work at the request of the Iowa Legislature for the express purpose of making policy recommendations about statewide assessments. Anyone interested in our work, including staff at The Gazette (not that they need my permission or approval), can and should poke, prod, examine, question, and even second-guess our recommendations and process.

Miller also describes the staff editorial as misinformed. That seems like a pretty strong word for what appeared to me to be a pretty well-informed take on the assessment issue. Might I suggest “written from a different perspective than mine” or “written with different priorities in mind than mine”?

Miller describes the task force as having “spent more than a year studying options for an assessment system.” I think it would be more accurate to describe the task force as having studied summative (end of year) assessment options, one of which also included interim assessments and a digital library.

Miller describes the Smarter Balanced assessments more than an annual test but “a system of quick, informal tests–some lasting only a few minutes”, an “approach to assessment [that] doesn’t take time away from instruction, as The Gazette suggests.” I can only guess that Miller is referring to the formative assessments that are part of the Smarter Balanced digital library or the interim assessment blocks. However, summative Smarter Balanced assessments and the full-length interim assessments do take more time to administer than either the current Iowa Assessments or the proposed Next Generation Iowa Assessments and do not include required science assessments. It’s hard to see how these lengthier assessments (plus the additional, required science assessments) don’t take more time away from instruction unless 1) the Legislature adds time to the school year or 2) the kids take them during lunch or recess time.

Miller than states “[s]upporters of the status quo will use misleading cost estimates or technology concerns to argue against the Smarter Balanced assessments.”

I have addressed the difficulty of crafting a fair, apples to apples comparison of costs for Smarter Balanced assessments and the Next Generation Iowa Assessments in the previous blog post. I think it is more helpful to be upfront about assumptions used to create the cost estimates than to accuse others of being misleading. In any case, the task force was charged with recommending a summative assessment,* so I don’t think it is inherently misleading to focus on the summative assessment cost estimates only and to exclude the other assessment spending by districts. It is up to legislators to decide whether to choose an assessment or an assessment system.

As for the technology concerns, the most recent task force documents show a bandwidth survey of schools has been done but not a computer hardware survey. Readers may be interested in the UEN technology directors memo to the task force on statewide assessment technology costs and support.

*See the text of 256.7(21)(b)(2) and (3).

ATF: Apples to Apples

On Wednesday, members of the Assessment Task Force appeared in front of the House Education Committee for a question and answer session follow up to the task force presentation the week before. Unfortunately, I was unable to attend.

However, in preparation for this appearance, the task force created an additional document comparing assessment options and I do have a copy of this to share. Find the first page (overview) here and the remaining pages (comparison table) here.

I want to focus my comments here on the portion of the table comparing the costs of the first year of an assessment system based on either the NGIA or the Smarter Balanced assessments as the summative (end of year/accountability) assessment.

This table illustrates that the Smarter Balanced assessments systems of assessments are less expensive than NGIA-based assessment system, even though the Smarter Balanced assessments have a higher per student cost than the NGIA.

I think the following unstated assumptions are baked into those numbers:

  1. Iowa’s vendor will match Connecticut’s vendor’s pricing for vendor services.
  2. Technology costs are zero (either no district or state entity will require technology upgrades to administer statewide online assessments or it is a district decision to upgrade or purchase additional technology not properly attributable to the decision to choose an assessment requiring online administration).
  3. Costs of science assessments (summative/accountability plus multiple measure) are zero.
  4. Districts will either stop administering all of those other multiple measures assessments that they have selected for district use or the costs of those assessments shouldn’t be charged to the “assessment system” if they don’t.

This table I think hints at the difficulty we have had creating a completely fair, apples to apples cost comparison of the two assessments. One requires technology, one doesn’t. One has an additional cost for a paper and pencil option (but perhaps not many schools will use it?), one doesn’t. One has a science assessment, one doesn’t. One has a suite of assessment options, one doesn’t. The predecessor to one has additional reporting costs paid by the state which are not included in the per student cost, one doesn’t.

Should we be comparing costs of summative assessments only or assessment systems and why? Which costs should be counted and why? Should technology costs have been included or is it just something districts should be doing anyway?