Oct 18, 2021
By: Jess Di Simone, Research Officer, EngineeringUK
Embedding evaluation in STEM outreach can be a powerful way to improve how future outreach programmes are delivered and ensure they have the intended impact. It can provide the evidence to drive improvements and contribute towards our understanding of what works in different contexts and for young people of different genders, abilities, ethnicities, or socio-economic backgrounds.
What is the measures bank?
The measures bank is a new resource that we have been developing for the Tomorrow’s Engineers website. Through these good practice guides, our aim is to help funders and providers of STEM outreach improve monitoring and evaluation activities.
The measures bank is an interactive resource to help users consider the STEM outreach outcomes they aim to evaluate and find relevant questions for their surveys.
The measures for this tool come from EngineeringUK’s research and evaluation surveys. It includes questions we have used in our Engineering Brand Monitor. This is our annual survey asking young people, their parents and STEM secondary teachers about their knowledge, perceptions and understanding of STEM and engineering. We have also included questions from our teacher and student evaluations. We use these surveys to assess the effectiveness of our events and activities in inspiring young people to pursue STEM education and careers.
How can the measures bank work for you?
There are over 250 questions included in the measures bank so let me walk you through how you might want to use it.
It’s helpful at the planning stage to know what aspects of your STEM outreach you’d like to evaluate. For example, you may want to collect feedback on the delivery of your programme. In that case you may choose to develop a survey to send out to the young people who take part in your activities. The measures bank can offer example questions and response options that can be used directly or adapted to suit the context. In this example, you would select the ‘Young people’ tab and filter the type of measure by ‘process’. Based on this selection, you’ll then find a host of questions that focus on different aspects of delivery, including questions related to content or participation. As part of your evaluation, you may also want to find out how young people’s experiences of your STEM outreach differ by their gender, ethnicity, or other sociodemographic characteristics. In this case, select ‘context’ and you’ll be shown questions related to demographic information.
User-friendly resource
When developing this resource, we reviewed our survey questions and considered how to present them in a way that could be most useful for you. The questions are organised based on the evaluation target group: young people, parents or teachers. They are grouped in three types of measures: impact, process, and context. Each measure is also tagged by topic, for example hard skills, careers or demographic information.
We encourage those funding or delivering STEM outreach activities to reflect on their evaluation approach:
- What are the outcomes of your STEM outreach? What outcomes are you trying to evaluate?
- What aspects of your STEM outreach do you want to evaluate?
- Who will need to participate in the evaluation to find this information out?
The answers to these questions will vary depending on a number of circumstances, including, the way in which your STEM outreach is delivered, the resources at your disposal, and a host of ethical, legal and practical considerations. There is not one evaluation approach that will work for us all.
Bearing this in mind, our intention is not to be prescriptive in the support we provide, instead we offer a range of options and examples that can be applied in your evaluation. For example, you can draw inspiration from these measures and adapt them to collect relevant feedback on your activities. You can also evaluate the success of an activity or programme and assess the improvements to be made, or whether you’d want to run the initiative again. If you’re a funder, you can share this repository of measures as part of a resource pack, encouraging others to make use of this tool.
Why does it matter?
Over the last year the Research team at EngineeringUK has been working on embedding the Impact Framework into our research and evaluation activities. This theoretical framework aims to support the engineering community consider how to measure the collective and long-term impact of STEM outreach on young people’s educational and career choices.
As part of this, we have been developing and testing survey questions that map onto the outcomes of the Impact Framework. Through the measures bank we wanted to make our work publicly available to also guide others in using the framework in practice in their own evaluations.
The measures bank is a starting point. We want this tool to be a living resource – it will be periodically updated as we learn from our own work as well as from feedback and contributions from the STEM outreach community. We aim to build on this tool and collectively improve the way we evaluate the impact of our STEM outreach efforts.
We know that diversity is an issue in STEM. I think it’s important that we work together on expanding our evidence base for effective STEM outreach initiatives so we can enable young people to learn, feel inspired and able to pursue subjects and eventually careers in STEM. All young people, regardless of their background, should be given the opportunity and support to make informed decisions about their future – let’s make sure STEM outreach is working for them!
If you have questions or comments on the measures bank, then get in touch here.
< Back to Blog
