Circle is a charity working with families to provide tailored support designed to help children deal with social injustice, poverty and health inequalities.
It provides one-to-one support sessions to families affected by drugs and alcohol, imprisonment and other social issues, offering advice and helping them to make positive lifestyle choices and sustain these choices as they mature, with an end goal of improving the life and prospects of the children within each family.
The Data Lab facilitated data collection and analysis for Circle by providing access to a data scientist who analysed data collected from families over the course of their engagement with the charity. This allowed for the identification of a benchmarking figure that would help estimate the improvement across individual families in a definitive way.
The challenge
Circle, which was established in 2006, has worked tirelessly towards improving family lives, but was unable to tangibly show the impact of this work. While undeniably having a positive impact on children and families, Circle had not been able to consistently and uniformly measure the impact of its support sessions, due to extremely high levels of variables affecting one family to the next.
In teaming up with The Data Lab, Circle was, for the first time, able to demonstrate its overwhelmingly positive impact by using data science to quantify the rate of improvement among each family it worked with.
The method
This project was set in place with two main objectives in mind: to verify whether disadvantaged families assisted by Circle improved their situation over time, and whether this process was less successful depending on how deprived a family’s neighborhood is.
According to the project application drawn up and submitted by Circle, these aims or expectations were formalised as the following hypotheses:
- Deprivation-related
- Higher area deprivation is linked to lower baseline scores for families. (H1)
- Higher area deprivation is linked to lower end scores for families. (H2)
As H1 and H2 were closely related, they were considered jointly (as a single hypothesis) – namely that families in more deprived areas do less well on Circle assessments.
- Length of contact
- Families working with the charity improved over time on the criteria they were assessed on. (H3)
These hypotheses were non-specific in terms of the types of criteria assessed – area deprivation was generally considered as a detrimental factor for family progress and similarly, length of contact, measured as the number of sessions undertaken with Circle, was also considered a general beneficial factor, regardless of the criterion/outcome in question.
The Data Lab’s Dr Caterina Constantinescu worked closely with the team at Circle to identify no less than forty key indicators that would clearly show whether the work the charity was doing was indeed effective, and to what scale.
A broad range of indicators were measured on a scale of 1-10, including (but not limited to) Circle’s efforts to:
- support the reduction of alcohol or substance use
- educate families on better budgeting to help manage household finances
- help families access appropriate housing
- limit activities which could lead to children’s school exclusion
- improve parents’ education and employment prospects
- increase parents’ responsiveness to their children’s emotional needs, and many more.
Analysing data collected over time on these indicators then allowed the charity to identify a benchmarking figure from which to estimate improvement across individual families.
Circle also wanted to understand whether deprivation of an area was linked to a family’s progress. In order to find this out, Circle data was merged with the Scottish Index of Multiple Deprivation (SIMD) (2016 version).
SIMD data includes overall ranks for small areas (called data zones), which are rated from most deprived (ranked 1) to least deprived (ranked 6,976). These ranks were further cut into vigintiles, deciles, and quintiles, useful to indicate areas situated among the top 5%, 10% and 20% most deprived areas in Scotland.
In addition, deprivation can be deconstructed into sub-domains, such as: Income, Employment, Health, Education, Housing, Access and Crime. Together, these sub-domains were used to describe the general level of deprivation in an area.
Family postcodes were first used to place each family within a certain datazone, with its associated SIMD16 rank, vigintile, decile and quantile. After this initial data merge, the second step was to add SIMD16 sub-domain ranks (such as income, employment, health, education, housing, access and crime) as well.
The results
Data provided by Circle allowed The Data Lab to build a series of line plot graphs, illustrating how families progressed over time across all criteria they were assessed on. The results suggested that most families did improve over time, regardless of the specific criteria in question, indicating that Circle’s work was effective 100% of the time, within the data set.
However, no trend emerged based on the potential impact of area deprivation.
The Circle data suggested that most of the families they supported were located in deprived areas, which meant it was difficult to compare with families from other areas. Family scores were set against SIMD16 deciles (both for overall deprivation levels, and for sub-domains) and, interestingly, no systematic relationship between deprivation and family scores emerged.
Discussing her work on the project, Dr Caterina Constantinescu said:
We knew Circle was doing fantastic work with families across Edinburgh but due to the nature of the sessions, and highly tailored approach, it was historically difficult for the charity to quantify how much of an impact it was having through data science – instead relying more heavily on anecdotal evidence.
“We worked with the team to determine how effective the sessions were for individual families. Moving forward, the charity is now able to analyse sessions in much greater detail, and leverage data science techniques to inform policies and decisions in the future, providing Circle with key stats to use in future funding applications. It’s a great example of using data as a force for good.
Data impact for Circle and beyond
As a result of the counselling sessions, the data showed that, on average, each family improved by 0.77 points after every support session. This figure clearly demonstrates the value of the work the charity is undertaking and helped it strengthen its case to secure additional Government, Trust and Foundation funding – vital in helping Circle to support more families throughout Scotland.
Not only that, but the initial figure can now be used to determine how well individual families continue to respond to the support sessions, by comparing improvement rates against this benchmark – allowing for appropriate measures to be taken to offer additional support if the benchmark is not met.
The Data Lab has been instrumental in helping us unlock the potential of data science as a means of measuring our impact. We’ve since used the results from the partnership to start planning more activities, for example, advocating with more confidence to policy makers that the duration of support to families be increased, as the data analysis clearly demonstrates that this has a beneficial impact on children’s outcomes. This in turn supports Circle to seek future funding and continue to work with and have a positive impact on families across Scotland.
Alex Collop, Project Manager from Circle
It’s now hoped that this methodology can be applied across more third sector organisations facing similar challenges in quantifying impact due to the nature of their work.