The importance of measuring impact in district storytelling
When we began sharing the positive stories from within the buildings and classrooms or our district partners, we saw positive impact immediately. There were dozens of anecdotal data points coming in from residents, board members, parents, and other community members. Our superintendents were receiving emails, text messages, phone calls, and in-person comments as they encountered members of the district public. But we didn't have a good way of capturing hard data that could be used to explain the impact of these programs.
We explored various ideas for measuring impact and finally settled on a hybrid approach, enlisting a tool used by businesses the world over called the 'Net Promotor Score.' This tool utilizes data gathered through a single question survey. The typical question goes like this: "How likely are you to recommend our company to a friend?" Two problems with this. Number one, schools aren't businesses and, two, their 'customer' base is somewhat captive, bound by its district boundaries. So we decided we needed to measure something else and frame the question a bit differently. After consulting with the statistics and public policy experts at Washington University in St. Louis, we decided to seize upon the public's awareness of school district strengths, phrasing our single survey question around that idea.
Our single question on our survey therefore became: "How aware would you say you are of your local school district's strengths." The thought being, if a respondent is aware of a school's strengths, they are more likely to be supportive of that school. If they are less aware, either because they aren't paying attention or because they don't believe that district has any notable strengths, they are less likely to be supportive or engaged, two things we all want more of.
The way this score is calculated through the survey, which is designed to be extremely easy to use and accessible to every district taxpayer, is that we pose our question with a range of possible responses from 0-10. As with the Net Promotor Score, respondents answering with either a 9 or a 10 are considered statistically relevant as 'Promotors,' or, in our case, 'Strength Aware.' Respondent answering with a 0-6 are considered statistically as 'Strength Unaware.' And, like the Net Promotor Score, responses of 7 and 8 are considered statistically irrelevant, and are not counted.
The percentage of ALL answers, including 7s and 8s, which are Strength Aware (9s and 10s) is calculated and used as our minuend. We then calculate the Unaware responses in the same way and place that percentage number as our subtrahend. then we subtract our subtrahend from our minuend to get our Net Strengths Awareness score.
We make the reasonable assumption that with the landing of the first magazine, our district audience is as close to being unaffected by the Journey12 program as they will ever be to get a good baseline NAS. The survey is conducted again at the third issue on a quarterly program, to establish the statistical delta. This gives administrators an important tool when considering the value of this investment.
In the Spring of 2024, we measured 10 samples, which had followed this protocol of 1st and 3rd magazine surveys, and what we found was that the statistical data echoed our anecdotal responses. Our mean increase in the Net Awareness Score was 270.3%, a nearly 3-fold improvement from baseline taxpayer awareness of district strengths. Of note, also, was the delta between partner samples with the highest and lowest beginning Strength Awareness Scores. This ran from 18 on the low end to 37 on the high end. Interestingly, the sample which began at a Net Awareness Score of 18 moved the second furthest, by a percentage of 338.9% to a 61 after the third magazine issue dropped. the partner sampling, which began at a Net Awareness Score of 37 moved to the second highest overall score, an 82, but by a percentage of just 221.6%. Analysis on these samples seems to suggest that in the more connected communities, growth, while still quite strong, is overall lower than experienced by less connected communities.
