Impact Report 2025/26: Independent Evaluation

2023 was a big year for impact at Apps for Good. Our Theory of Change had been refined and was informing our work, content decisions, feedback surveys and participant interviews. We had built on our longstanding partnership with Skills Builder to map our courses explicitly against the Universal Framework, and had piloted our latest course, App for Social Action. However, we were also facing challenges. While our surveys produced insights on short-term outcomes, they were self-developed, and response rates were low. We also wanted to develop our approach to measuring long term outcomes. We needed expert external advice on making data collection more robust, creative and engaging.

A new partnership brought an exciting opportunity. The Hg Foundation supports education and employment programmes that widen the pipeline of talent into the technology sector, and works closely with its partners to learn and develop their evaluation approaches. Their funding from 2023 to 2026 has enabled Apps for Good to roll out the App for Social Action course to schools and students. It has also built our evaluation capacity; we recruited our first in-house Impact Manager and appointed the Institute for Employment Studies (IES) to conduct an independent evaluation of App for Social Action.

The evaluation aimed to:

• Review and refine Apps for Good’s Theory of Change and approaches to monitoring and data collection.
• Evaluate delivery, reach and outcomes for students and teachers taking part in App for Social Action.
• Explore the feasibility of measuring long-term outcomes, in particular impact on GCSE subject choice.

IES used a mixed methods approach, collecting evidence through student surveys, teacher interviews, Apps for Good management information and delivery data, and GCSE Computer Science option-choice data collected from schools. As we wrap up the three-year project, we’ve been reflecting on the key learnings.

In terms of reach, the evaluation showed that App for Social Action reached a range of secondary schools in England, Wales and Scotland, including many serving less advantaged communities (based on Apps for Good’s ‘challenging circumstances’ criteria: average proportions of students eligible for Free School Meals, or with higher levels of SEND or English as an Additional Language).

However, the learnings around what delivery looked like within these schools and outcomes for students were limited. One of the most challenging aspects of the evaluation was collecting a high enough volume of data. Despite a range of methods and incentives to encourage schools to take part, only a small minority of the schools delivering the course participated in the evaluation activities. Delivery and impact findings are therefore from a limited subset of all schools. This, along with a relatively low number of student surveys completed, mean that the findings should be interpreted with caution and should not be generalised to all schools that have delivered the course.

In schools that engaged with the evaluation:

• The course was most commonly delivered through timetabled KS3 computing lessons, and this curriculum delivery was generally the more inclusive model.
• Schools generally delivered the course in line with its intended model; students worked on real-world social issues, developed app ideas through a structured, project-based process, and used AppLab to build a prototype. Barriers were most often linked to teacher time and capacity, technical confidence, school systems and infrastructure, and the practical demands of components such as the Showcase and Industry Engagement sessions.
• Interviewed teachers felt that App for Social Action differed from business-as-usual computing provision in that delivery was not limited to Computer Science specialists, it had greater adaptability/flexibility than other materials, and it emphasised applied, project-based learning and external engagement from industry volunteers.
• Interviews and surveys suggested that the course was associated with a range of positive short-term outcomes, particularly the development of essential skills, increased confidence in computing, and greater understanding about how technology can be used for good.
• Strong engagement from girls came through from the survey, with higher proportions of girls than boys enjoying the course, coming up with ideas for and building a prototype of their app, and learning about social action and helping people in their communities.
• Data collected from schools about course participants’ GCSE choices showed that participants were on average four times more likely to choose GCSE Computer Science than peers at the same schools who did not take part. However, the evidence was based on a small and self-selecting group of schools, without a formal comparison group design and with likely selection effects. Alternative explanations, including pre-existing pupil interest, school context and delivery model differences, cannot be ruled out.
• Surveys showed little evidence of impacts on intentions to work in digital technology, but qualitative work with teachers suggested that the course may play a useful role in broadening horizons and making future digital pathways feel more relevant and accessible, even though the current evidence is not strong enough to demonstrate consistent changes in study intentions or career plans.

We’ve used the evaluation findings - and our reflections on the whole project and process - to improve our App for Social Action course, update our Theory of Change, improve our data and monitoring processes and systems, and develop our brand new course. Here are just some of the ways we’ve acted on our learning during the project, and how we intend to keep doing so as we move forward.

1. Improving our monitoring, feedback and impact measurement approaches:

• We developed and launched the teacher profile tool on our educator platform, which is embedded more seamlessly into teachers’ user experience and helps them to provide and update their delivery plans. This has resulted in an increased amount of delivery information collected from schools, and Apps for Good can now provide better targeted support to schools based on the information they provide.
• Our new course, AI for Good 2.0, sits on a new platform where educators access and download lesson resources in stages, and complete quickfire feedback questions in order to progress. This new system aims to provide even more reliable tracking data.
• We will continue to expand our suite of incentives for schools to provide feedback (e.g personalised impact reports, certificates and vouchers).
• We’ve streamlined our feedback survey to focus on post-course feedback (removing the pre-course survey), and used the IES-developed survey to help us refine the questions as we ask. This means we can still gather meaningful feedback and perceptions of change in a proportionate way, while rolling out our new course and new teacher facing platform (with feedback tools embedded). Both of these will lay important foundations for implementing baseline impact measures again in the future.

2. Developing our course content and delivery support in response to teacher feedback:

• Our new course contains even more guidance for teachers (through bitesize written and video guides), new learner worksheets, and more scaffolded editable resources.
• The new course goes much further on technical learning, offering JavaScript coding or AI-assisted (vibe) coding routes to app development. The course also responds to early insights from the curriculum and assessment review to help support teachers in future-proofing for upcoming changes.
• We’ve implemented automated communications to make Industry Engagement and Showcase entry clearer and more manageable for schools, so that even more of them can deliver the full course and maximise its impact opportunities. AI for Good 2.0 materials contain step-by-step guides for booking Industry Engagement early on in the course, and the course also dedicates an entire lesson to Showcase preparation and entry.

3. Thinking about future evaluation projects to strengthen the evidence base and track long-term outcomes robustly:

• We’ve increased our knowledge of potential ways of measuring hard outcomes, for example, what data is available from national datasets (and the timescales that are required), and we now have experience of collecting this data directly from schools. We’ve learned that sharing subject data is time consuming, requires sign-off from multiple people, and adds to an already excessive list of administrative burdens for school staff.
• We’ve learned that computing and technology pathways are not always linear or straightforward. Even if learners are interested in a tech pathway they may not be able to choose computing at GCSE due to the lack of options and flexibility provided by schools alongside other subjects. GCSEs in computing or related subjects are not a prerequisite for future study in these areas, and not all students who plan to pursue tech pathways will select these subjects at Key Stage 4.
• The Apps for Good delivery model presents challenges for robust evaluation, and future efforts are likely to benefit from a smaller, more focused study where a select group of schools are specifically recruited for an evaluation project.

The amount of learning that we’re taking away from this project cannot be overstated, and we want to say a big thank you to IES and The Hg Foundation for all their support. Please do reach out to Lou, our Impact Manager, by emailing louise.jones@appsforgood.org if you’d like to read the full report or discuss the project with us!