A Grass Roots Approach to Rethinking Accountability

By Elliott Asp and Rebecca Holmes | February 28, 2018

Accountability is one of education’s most vexing problems. It demands new approaches and a clarified purpose. At CEI, we are eager to explore approaches that drive ownership and improvement at the local level. One promising approach is the Student-Centered Accountability Program (S-CAP), developed by a group of small rural districts to make the accountability process a more useful and powerful tool for district and school improvement. As part of CEI’s EdPaper series on accountability, this paper presents an overview of S-CAP, shares our learnings as a partner, and offers suggestions to take action.

The Student-Centered Accountability Program (S-CAP) is a joint effort by a group of small rural Colorado school districts to develop an approach to accountability that incorporates a wide range of indicators of district and school quality, considers the whole child, is focused on the needs and interests of local communities, and is supported by a network of peers.
Three key features of the S-CAP accountability process are:

• Expanding the focus of accountability from student performance on state tests to include a variety of other indicators of school quality that are of importance to local stakeholders.
• Providing data that is useful, timely, and meaningful for continuous improvement.
• Incorporating a System Support Review (SSR) process in which colleagues from similar districts participate in on-site reviews and provide feedback to the host district.

Currently, S-CAP is made up of 5 core districts:  Buena Vista, Buffalo, Kit Carson, La Veta, and Monte Vista. Each core district hosted an SSR during the 2016-17 school year and that process has continued into the 2017-18 school year. In addition, participating districts have received support from CEI, Generation Schools Network, and Battelle for Kids, and have developed a research partnership with the school of education at the University of Colorado, Denver. Several other districts have expressed interest in joining S-CAP and have participated in the SSR process as observers; Holyoke School District is scheduled to join the group this spring. The S-CAP superintendents have presented their work to the State Board of Education on several occasions. The board unanimously passed a resolution of support and directed CDE to partner with S-CAP in this work.

The S-CAP districts developed this process because they believed the existing state accountability system did not provide data that was relevant to the unique needs and interests of rural school districts and relies too heavily on the results from state testing as a judge of the quality of districts and schools. District leaders wanted a more complete system that reflected what is most important to local communities and to provide data that would be more useful for improvement.

System Support Review (SSR) Provides a More Robust Evaluation

The most significant aspect of S-CAP is the SSR. Similar on-site review processes are used across the U.S. by states, districts, charter and private schools, and in the U.K. in a variety of public schools.

These reviews are conducted by teams of raters (teachers and administrators, including the superintendent) from S-CAP districts. After an overview and training, the raters use various data collection tools and protocols for classroom observations, conduct focus groups with stakeholders, and examine data from student and teacher surveys. The teams summarize their findings using specific tools and protocols for various aspects of district or school quality (i.e., curriculum and instruction, leadership, professional development, and climate/culture). The summaries from the teams are given to the visiting superintendents who jointly determine a final rating for the host district on each component of the SSR. The findings from the review are displayed on the district’s website and are incorporated into the district or school improvement plan.

This final rating represents a holistic picture of the host district along with comments and specific feedback from the visiting teams on a wide variety of indicators. The host superintendent signs off at the end of the process that the findings are an accurate representation of the district or school. The superintendents’ final ratings for each component and the ratings and comments from each team are available immediately to the host district and the host superintendent can also follow-up with any of the raters. It is left up to each host district to decide how, when, and with whom the findings are shared.

Peer-To-Peer and Community Input are Key

As S-CAP is finishing its second year of conducting SSRs in the core districts, some learnings have emerged regarding the benefits of the process for both the host district as well as outside raters. Notably, participants don’t complain about “having” to engage in an SSR and they are not afraid of what the data will reveal. They look forward to hosting a visiting review team and/or participating as a reviewer even though it takes extra time and energy to plan and orchestrate the event. Other key learnings include:

• Host district staff are excited to receive critical feedback on their work. Superintendents of host districts report that the data were very helpful to them as they worked with their staff and board to understand the root causes for their challenges and develop strategies to address them.
• Superintendents and staff from host districts felt the visiting team’s feedback generally validated their own thoughts about their district (both strengths and challenges), although the findings from the visit often pushed their thinking and typically uncovered some issues that they were not aware of or were uncomfortable raising on their own.
• Participants from visiting districts report that they gained many instructional ideas from observing and talking with teachers from the host district and from discussions with educators from other districts who were on their teams.
• The SSR enabled board members, parents, and community members to be involved in the accountability process in an authentic manner.

Right Tools and Training are Essential

While all strong education leaders make the best use of the current state accountability data, the reality is that it requires little time and investment from them. A system that gives more robust data demands more time and effort from school leaders and teachers who already have busy jobs.

CEI’s role has been to provide technical feedback on the process, along with other partners, so that the data for SSRs can be trusted as a basis for district/school improvement efforts. Here are some of our most important learnings:

• It is critical that the protocols and tools used to collect and summarize data and develop the final summary report be carefully vetted if the results from the SSR are to be an accurate reflection of the district/school.
• Experienced reviewers and trained facilitators are more comfortable with the process and provide more accurate and useful feedback. The team leader’s experience with the process and understanding of teaching and learning is also a significant factor in the quality of the data produced from the SSR.
• The facilitator plays a critical role in ensuring the SSR process is implemented in a similar way in every host district.
• Dedicated and focused training for reviewers on how to use the tools and follow the protocols is key to ensuring the accuracy of the findings.

To be useful in accountability the SSR process must be relatively standardized and implemented in the same way in each host district.

Next Steps for S-CAP

Based on our experience observing the SSR process, we have identified several next steps for improving the process and documenting its impact:

• Implement a dedicated training for a cadre of reviewers and facilitators.
• Continue to revise the SSR tools and protocols and standardize the process.
• Examine ways to provide support for this work through BOCES or community colleges to focus on the needs of rural districts in different regions across the state.
• Document (reports, videos, etc.) how the SSR works and how the findings are being used for continuous improvement.
• Identify S-CAP as a formal state pilot and allocate resources to support the logistical and operational requirements of the process.
• Determine readiness criteria for participating districts and begin to examine how SSR findings might be used in combination with state testing data for accountability purposes.

An Ongoing Learning Agenda on Accountability

S-CAP may not yet be perfect, but it is a powerful example of what might emerge if we designed accountability around driving local ownership and a process for continuous school improvement.

Inevitably, some policy makers and advocacy groups will approach S-CAP with skepticism. We’ve been trained to believe that it is important – and possible – to summarize the performance of a school into a single score, rating, or letter grade. Many of us also hold a mindset that suggests current leaders of school systems lack the skills to develop and push their peers. And some, particularly those of us in urban areas, believe that rural communities will shy away from addressing persistent evidence of opportunity gaps for low-income kids and kids of color.

The time CEI has spent in this process has made us eager to see more robust evaluation of a school incorporated in accountability ratings, has renewed our already strong faith in the leaders of our schools to engage in rigorous evaluation, and has reminded us that a stand for equity can be taken in any community. Moreover, an approach to accountability based on a system that drives ownership and improvement at the local level is something we all should be eager to explore. As a state, we should actively support this effort and encourage the development of other points of light around accountability so that all districts and schools can be authentically engaged in continuous improvement. We invite you to continue with us over the next several months as we progress through our examination of school accountability in Colorado. Share your thoughts at accountability@coloradoedinitiative.org.

We invite you to join an ongoing dialogue about Colorado’s school accountability system. There is an exciting opportunity at hand to revisit our priorities and examine the limitations of our current system with an innovator’s mindset. Each month we will publish an EdPaper that takes a deeper look at the subject of school accountability, as outlined below. We invite you to be part of the conversation and share your thoughts with us via email at accountability@coloradoedinitiative.org.
JANUARY EdPaper: The Nature of Accountability is Ready for Change
FEBRUARY EdPaper: S-CAP (Student-Centered Accountability Project) and its Implications for Local Measures
MARCH EdPaper: High School Accountability
APRIL EdPaper: Accountability in a Competency-Based System
JUNE EdPaper: To Whom Are Schools Most Accountable?

We invite you to be part of the dialogue. Email your thoughts to join the conversation!