LSF EVALUATION REPORT

PRE TO POST ODT ANALYSIS

OCTOBER 2017

Authors: Jack Egan and Nick Ockenden

Analysis Team: Jack Egan, Nick Ockenden and Joy Dobbs

1
Contents
1. Introduction and key findings ................................................................................................... 3
1.1. Grant holder profiles ...................................................................................................... 4
1.2 Potential routes to sustainability .................................................................................... 5
2. Change in ODT score ................................................................................................................. 8
2.1 Overall ODT score ............................................................................................................ 8
2.2 ODT sub-scores .............................................................................................................. 10
3. Thematic analysis .................................................................................................................... 13
3.1 Stage of sustainability.................................................................................................... 13
3.2 Size of organisation ....................................................................................................... 15
3.3 Change in volunteer numbers ...................................................................................... 15
3.4 Generating greater income ........................................................................................... 17
3.5 Improving systems ......................................................................................................... 17
3.6 Age of organisation........................................................................................................ 17
4. Sub-score analysis ................................................................................................................... 17
4.1 Market & opportunities ................................................................................................ 17
4.2 Track record & capabilities............................................................................................ 19
4.3 Quality & impact ............................................................................................................ 19
4.4 Sustainability .................................................................................................................. 20
4.5 Strategy & planning ....................................................................................................... 21
5. Conclusions .............................................................................................................................. 21

2
1. Introduction and key findings
Key findings

Across all grant holders the change in the overall ODT score ranged from a 33% increase
to a 7% decrease.
The average (mean) change in score across all grant holders was a 11% increase (average
score in the 'pre' project ODT score was 66% and 76% in the 'post' project score).
The vast majority (93%) saw an increase in their score over the course of their LSF project
and only 7% saw a decrease.
Of the five different sub-scores within the ODT, Track Record & Capability saw the highest
average increase of any, at 12%.
Organisations classified by the evaluation as 'Growers' (i.e. experiencing increasing
income prior to LSF) generally experienced lower levels of changes in ODT scores than
organisations classified as 'Squeezed middle' (little or no growth in years prior to LSF) or
'Survivors'(declining income in years prior to LSF).
Larger organisations (i.e. those with an annual income of over 500,000) were more likely
to have seen bigger positive changes in their sustainability score.
There is some evidence of increases in the number of volunteers over the course of the
LSF project and higher levels of overall ODT score growth, but only for larger
organisations with income over 250,000.

Our analysis

The purpose of this report is to analyse the Online Diagnostic Tool (ODT) scores for grant
holders, with specific focus on how scores have changed over the course of Local
Sustainability Fund (LSF) projects and what characteristics are associated with changing
scores. To do this we have compared the score that grant holders got when they completed
the ODT during the application stage in 2015, before their projects began (the pre score)
with the score they got after they had completed their projects (the post score). This
represented a period of at least 12 months, but up to 15 months in some cases. As well as an
overall score representing the holistic level of sustainability an organisation has, the ODT
also produces five sub-scores which focus on some distinct organisational areas, these
being: Sustainability, Market & Opportunities, Strategy & Planning, Track record & Capability
and Quality & Impact.

3
The analysis in this bulletin will focus on comparisons between pre- and post- scores in the
ODT for grant holders.1 Where relevant it will refer to other data from the rest of the
evaluation of LSF to help contextualise or explain findings from the ODT data.

The ODT was developed by Creative Co-op, on behalf of the Big Lottery Fund (The Fund). The
scores have been made available to NCVO to analyse. The analysis in this bulletin is based on
scores completed by 227 grant holders. Further detail on the ODT and our analysis of the
data is available in Appendix A. The appendix also describes a small bug within the tool itself
which inflated some of the scores slightly; we agreed with The Fund to continue our analysis
with this data as the inflation applied to both the pre- and post-scores so did not affect our
findings.

Before presenting the detailed results of the analysis, it is useful to highlight two key
observations we were able to make about grant holders after examining the ODT data.
These observations relate to the organisational profiles of grant holders at different levels of
sustainability and the potential routes grant holders can take towards building sustainability.

1.1. Grant holder profiles

Our analysis of the ODT data enables us to describe some of the important characteristics of
those organisations that have high, moderate and negative/low growth. While these
characteristics, presented below, provide a useful summary of some of the correlating
factors, it should be remembered that they do not seek to describe a typical organisation;
there is considerable diversity throughout all grant holders and within these three
categories.

Growth profile Characteristics

High Growth N= 51 High growth grant holders were most likely to:

(16%+ overall ODT - Be a Squeezed middle organisation

score increase) - Have an income of over 250,000
- Have a Low or Lower Mid level of sustainability from
the pre ODT
- Have increased its number of volunteers over its LSF
project
- Be aiming to generate greater income through their
project

1
Throughout this report any statistically significant findings will be flagged, all other reported relationships or
changes should be seen as directional or indicative.

4
 Moderate Growth Moderate growth grant holders were most likely to:
N= 126
(6% - 15% overall ODT
score increase)
- Be a Squeezed middle organisation
- Have an income of over 250,000 (grant holders with an
income of 250,001 - 500,000 most likely to be seeing
6% - 10% growth, whereas grant holders with an income
of 500,001+ more likely to be seeing higher growth, of
11% - 15%)
- Have a Lower Mid or Upper Mid level of sustainability
from the pre ODT (grant holders seeing 6% - 10%
growth more likely to be Upper Mid, those seeing 11%
- 15% growth more likely to be Lower Mid)
- Have maintained the same number of volunteers over
its LSF project
- Be aiming to generate greater income through their
project, particularly through developing new funding
streams and diversifying income
- Those grant holders seeing the upper half of moderate
growth, 11% - 15%, are also more likely to have included
Impact assessment in their LSF project than not (52.4%)

Negative/low Growth Negative/low growth grant holders were most likely to:
N= 50
- Be a Squeezed middle organisation, although around a
(5% and below overall quarter of Negative/low growth grant holders are
ODT score change) Growers (24%)
- Have an income of less that 250,000
- Have an Upper Mid or High level of sustainability
from the pre ODT
- Have maintained or reduced its number of volunteers
over the LSF project
- Not be aiming to improve systems via the LSF project

1.2 Potential routes to sustainability

The ODT data also indicates some potential routes or journeys that organisations appear to
be taking towards increased sustainability. We have highlighted two potential routes below,
with the aim of providing some insight into how different organisations are working towards
sustainability.

5
Closed to open
Whether an organisation is Closed or Open in their outlook impacts upon sub-scores
differently and may indicated that grant holders go through a process of moving from a
Closed organisation which is linked to stronger improvement in the track record &
capabilities sub-score to becoming more Open which in turn is linked to improvements in
Market & opportunities score. This typology of how open or closed organisation are, in
terms of user involvement as well as their pursuit of partnerships or relationships, was
developed as a part of the Further analysis of ODT and application data bulletin and is
described further in Section 4.1

It could be inferred that a grant holder may begin their journey as a Closed organisation
and utilise this more inward-looking stage to focus on its own developing its internal
capabilities and the ability to demonstrate the effectiveness of their work. Having sufficiently
improved in these areas, focus then shifts on to more external issues such as partnerships
and surveying the market for opportunities, thus in becoming a more open organisation the
grant holder will begin to improve another element of sustainability.

The flow chart visually below demonstrates how this journey may work for some grant
holders.

6
Improving systems
Grant holders which aimed to improve systems through their LSF project tended to be more
likely to see higher levels of growth on two of the five sub-scores. The fact that improving
systems appears to help grow both Quality & impact and Track Record & Capabilities scores
implies that it could be the first step in the process of building sustainability for some grant
holders. Central to this is building the capacity of organisations in order for them to be able
to take new projects or develop new ways of working which ultimately help grow the
organisations long-term sustainability.

The flow chart below visually demonstrates how the relationship may exist and that if
properly implemented, improvements to key systems could make a real difference to the
sustainability of grant holders.

7
2. Change in ODT score
2.1 Overall ODT score
The overall ODT score serves as a holistic indicator of sustainability for each grant holder and
is derived from a range of questions asked as a part of the ODT. Looking at results for all
grant holders that completed the ODT at the start and end of their LSF project, the change in
overall score ranged from a -7% decrease to a +33% increase. Whilst bearing in mind this
variation, it is also useful to examine the average level of change in score in order to get an
indication of the overall levels of change being seen by organisations across the network.
The average change in score across all grant holders was a +11% increase, with average
score increasing from 66% in the first run of the tool to 76% in the post run.

The overwhelming majority of grant holders have seen some increase in their overall ODT
score over the course of their LSF project (94%) and only a small proportion have seen either
zero or negative growth (6%).

While it is not possible to directly attribute the increase in score for organisations to their
involvement in LSF, the scores do suggest that the majority of organisations have undergone
a positive journey in this period. Any changes in organisational sustainability are likely to be
relatively slow and will take time to be seen, so such changes observed here are
encouraging. Other data from the evaluation, including results from the snapshot surveys
and analysis of the end-of-project monitoring reports also suggest many organisations have
started to see notable and meaningful changes in their organisational sustainability. It is
worth noting, however, that while these changes in scores suggest a positive experience for

8
most grant holders, the scores themselves are subjective (based on self-reporting) and
somewhat approximate, which is a necessity of a tool of this nature. Grant holders we spoke
to in this evaluation tended to recognise that sustainability is a journey rather than an end
point, therefore it is not necessarily possible or desirable to achieve 100% sustainability
as these scores could suggest. Nonetheless, an increase in the percentage score can, for the
purposes of this bulletin, be equated with progress along a journey towards greater
organisational sustainability.

The table and chart below show grant holders grouped into five categories according to the
size of change in overall ODT score.

Change in Overall ODT Number of (%)

organisations

-7% 0% Growth 13 5.7

1% - 5% Growth 37 16.3

Total with negative/low growth 50 22.0

6% - 10% Growth 63 27.8

11% - 15% Growth 63 27.8

16%+ Growth 51 22.5

Total with medium/high growth 177 78.1

All organisations 227 100*

* Sums to more than 100 due to rounding

9
 Change in overall ODT score (%)

5.7 16.3 27.8 27.8 22.5

-7% 0% Growth 1% - 5% Growth 6% - 10% Growth 11% - 15% Growth 16%+ Growth

For much of the analysis in this report, we will use the two combined sub-groups shown in
italics: organisations with Negative/low growth (a change in score of between -7% - +5%)
and Moderate/high growth (change in score of 6% and above). That provides sufficient
sample sizes in the two groups to be able to analyse differences between them, with the aim
of identifying key characteristics of those grant holders that have increased their score the
most. Just over three-quarters (78%) of grant holders fall into the Moderate/high group
with just under a quarter of grant holders seeing Negative/low growth (22%).

It is interesting to note that some (albeit a small minority) of grant holders experienced a
worsening situation with regard to their organisational sustainability during this period. As
with any positive growth, it is not possible to determine whether this decrease was the
result of LSF or was associated with wider environmental factors, although the latter is a
more likely explanation. Many organisations have, in other parts of the evaluation, noted
that the wider environment in which they operate remains challenging and has, in some
cases, acted to limit their progress towards achieving greater organisational sustainability. As
part of this analysis we also looked at the end-of-project forms from those organisations that
reported negative changes in score, although there was nothing notable to help explain this
change.

2.2 ODT sub-scores

The ODT also produced five separate sub-scores on components of sustainability, these
being: Sustainability, Market & Opportunities, Strategy & Planning, Track Record & Capability
and Quality & Impact, outlined briefly below as described in the ODT report produced for
each grant holder. As mentioned in the introduction to this bulletin, these five sub-scores

10
fed in to the overall sustainability score (section 2.1), which was automatically calculated by
the ODT.

Sustainability Organisations who are sustainable can demonstrate effective leadership

and financial management and the ability to change, adapt and reach to market
opportunities.
Market & Opportunities It is really important that the market potential is clearly
understood and the opportunity genuinely achievable. An organisation may have
identified a clear market with real opportunity but have some challenges to overcome in
order to turn this into real success.
Strategy & Planning Having up to date, realistic and relevant strategic plans are vital.
Such plans include overall direction and objectives for the organisation and how these will
be met.
Track record & Capability This is about the performance of the organisation up to the
present and its capability and readiness for the challenges ahead including its
effectiveness in delivering services and in managing within its financial constraints.
Quality & Impact This covers both the quality of services provided and the impact of
the organisation in its community and/or its customers.

The five sub-scores will now be reported on individually, to examine changes between the
pre and post ODT runs. The same groupings of Negative/low growth (change in score of
+5% and below) and Moderate/high growth (change in score of 6% and above) will be used
to help analyse each sub-score.

As the chart below shows that on average all sub-scores saw an increase and the average
level of change ranged from an +8 increase for Market & Opportunities to an +12% increase
for Track record & Capabilities.

11
 Change in average ODT score from pre to post(%)
90
79 80
80 76 76
68 67 65 68 68 66
70
60 59
60
50
40
30
20
10
0
Market & Quality & Strategy & Sustainability Track record Overall ODT
opportunities impact planning & capability score

Pre Post

Sustainability
Change in sustainability score ranged from -10% decrease to +37% increase and the average
change in score across grant holders was +12%.

Over nine in ten grant holders (93%) saw some level of increase in sustainability score, with
7% seeing a decrease or zero growth. Just over three-quarters of grant holders (77%) saw
Moderate/high levels of growth (6% and above) with 23% seeing Negative/ low levels of
growth (5% and below).

Market & opportunities

Change in market & opportunities score ranged from -10% decrease to +25% increase. The
average increase was 8%, the lowest average increase of the five ODT sub-scores.

This sub-score has the highest proportion of grant holders that have seen zero or negative
change in score at 14.5%, consequently around a third (34%) saw Negative/low growth.
Two-thirds (66%) of grant holders saw Moderate/high levels of growth.

Strategy & planning

Change in strategy & planning score ranged from -10% decrease to +36% increase. The
average change was +11%.

A large proportion of grant holders (92%) saw an increase in this score, with less than one in
ten (8%) seeing zero or negative growth. Furthermore, 78% saw Moderate/high growth and
23% saw Negative/low growth.

12
Track record & capability
Track record & capability score saw the widest rage in score, ranging from -9% decrease to
+38% increase. The average increase was 12%, the highest average increase versus other
ODT sub-scores.

This sub-score also had the smallest proportion of grant holders that saw Negative/zero
growth (6%) or Negative/low growth (22%). In addition, 94% saw some growth in sub-score
and 79% saw Moderate/high growth.

Quality & impact

Change in quality & impact score ranged from -9% decrease to 35% increase and the
average increase was 9%.

This sub-score saw the largest proportion of Negative/zero growth at 11%, with a third of
grant holders (33%) seeing low/negative growth. In terms of growth, 89% saw some increase
in sub-score and 67% saw a Moderate/high increase.

3. Thematic analysis
In this section we will be analysing both overall ODT score and sub-scores according to a
range of themes that we have explored and developed in previous bulletins as part of this
evaluation. These range from the stage of financial sustainability grant holders were at the
start of the LSF programme to specific characteristics of their LSF project.

3.1 Stage of sustainability

As described in the Needs Analysis and Thematic Mapping and Further analysis of ODT and
application data reports produced as part of the evaluation, grant holders can be grouped
in to three categories according to their financial health when they initially started the LSF
programme:

1. Growers those organisations who have seen consistent growth in income over the
period 2013 to 2015 (26 organisations, 13% of grant holders);
2. Squeezed middle those organisations who have seen little or no growth in income
over recent years (168 organisations, 74% of grant holders);
3. Fighting for survival or Survivors those organisations who have seen sharp
declines in income over recent years (29 organisations, 13% of grant holders).

Using this classification in combination with level of ODT score change allows us to identify
relationships between score growth and stage of sustainability at the start of the LSF
programme. As with earlier findings in this bulletin, this does not, however, allow us to

13
determine causation; we are not able to say the extent to which any observed changes in
score were the direct result of involvement in LSF.

When comparing the levels of growth between stages of sustainability groupings it becomes
evident that those grant holders who were Growers when first taking the ODT are less likely
to be seeing Moderate/high levels of growth than the Squeezed middle or Survivors. As
shown in the chart below, around four in five Squeezed middle grant holders (82 %) and
three-quarters of Survivors (77%) were seeing Moderate/high level growth versus only
59% of Growers. Instead Growers are more likely to have seen Negative/low growth than
other groups (41%). This could possibly be because Growers might be further along on their
journey towards sustainability and therefore have less far to travel in the future; this will be
examined in more detail in the final evaluation report.

Level of overall ODT score growth by Stage of sustainability (%)

100

80
58.6
60 76.9 82.1

40

20 41.4
23.1 17.9
0
Survivors Squeezed middle Growers
Negative/low growth Moderate/high growth

This pattern is also observed across all the ODT sub-scores, with Growers being least likely
to see Moderate/high increases in score compared with Survivors and the Squeezed
middle. It is particularly evident for the Strategy & planning score, where 55% of Growers
are seeing Moderate/high growth versus 77% of Survivors and 82% of the Squeezed
middle. The differences are least pronounced when looking at sustainability score:
Growers perform strongest on this sub-score versus other sub-scores with 62.1% achieving
Moderate/high score growth however they remain behind Survivors (77%) and the
Squeezed middle (79%).

It should be noted that the fact that Growers are seeing lower score growth is not
necessarily a sign that that they are underperforming as a group and instead could be an
indication that they are maintaining their already high(er) ODT scores, whilst Survivors and
the Squeezed middle are making progress in pushing up lower scores from the pre ODT

14
run. If we look at the description of Growers from the Needs Analysis and Thematic
Mapping report we can see that their LSF project was seen by many as a way of adapting to
growing pains that are associated with the sustained growth in income that these grant
holders had seen. Thus relatively low levels of growth during this period could be linked to
mitigating the rapid rises they have seen in different aspects of sustainability.

This also raises an interesting dilemma about which organisations to support around growth
in organisational sustainability, or more precisely, at which stage of their sustainability
journey would targeted support yield the greatest benefits or returns. The data suggest that
organisations at an earlier (or less sustainable) part of their journey have the most to gain
and therefore an investment from a programme such as LSF may demonstrate a bigger
impact for such organisations, whilst an organisation that is more stable or more sustainable
(i.e. the growers) may have less to gain. However, there is also the question of whether
programmes such as LSF would see it as more desirable to support organisations that are
more stable (but with less to gain) or organisations that are more vulnerable and unstable
(but with potentially more to gain). These issues will be examined in more detail in the final
evaluation report.

3.2 Size of organisation

The current size of grant holders, based upon their estimated 2017 income, appears to be
related to the growth in their overall ODT score. Grant holders with an income of <250,000
were least likely to have seen Moderate/high score growth (59%) when comparing across
size bands. Grant holders with an income of 500,001 - 100,000 were the most likely to
have seen Moderate/high growth (88%) followed by 250,001 - 500,000 (81%) and
1,000,000+ (78%). This shows that even though over half of the smallest organisations saw
Moderate/high increases in score, they were still less likely to see marked score growth
than larger grant holders.

This pattern was also present to varying degrees across all ODT sub-scores other than for
Quality & impact. Interestingly for this sub-score grant holders with <250,000 were the
income band most likely to have seen Moderate/high score growth (72%) although the
differences in proportion were relatively small, with 250,001 - 500,000 being least likely to
see Moderate/high growth (63%). Whilst only being indicative, this finding may suggest
that smaller organisations are more concerned with improving the ways that they can
demonstrate the value and impact of their work than larger organisations.

3.3 Change in volunteer numbers

Looking at score growth by number of volunteers showed minimal differences between
bands of grant holders grouped by number of volunteers, aside from some indications that
grant holders with fewer volunteers seemed slightly less likely to have seen Moderate/high
score growth. For example, just under two-thirds of grant holders with 0-10 volunteers had
seen Moderate/high increases in overall ODT score versus 83.8% of those with 51-100

15
volunteers. Furthermore, the number of volunteers a grant holder has could potentially be
an indication of organisation size, which has some impact on level score change itself.

There is some evidence, however, of an association between increases in the number of

volunteers and higher levels of overall ODT score growth. Looking at those grant holders
where there has been an increase in the number of volunteers, only 7% of these have
experienced the lowest growth in ODT scores (-7% 0% growth) whereas 39% experienced a
score growth of 16% or more. Amongst grant holders with decreased volunteer numbers,
only 6% experienced growth of 16%+, compared with the aforementioned 39% of those with
increased numbers of volunteers. The full breakdown is shown in chart below.

Change in overall ODT score by change in volunteer numbers (%)

38.7
40 35.3
30.5
30 28.0
26.5
23.5 24.0 24.0
20.3
20 16.9

8.8
10 5.9 6.7 6.7
4.4

0
Decrease No change Increase

-7% - 0% Growth 1% - 5% Growth 6% - 10% Growth 11% - 15% Growth 16%+ Growth

However, it should be noted that it is organisations with an income of over 250,000 that
appear to be driving this relationship. Although overall there is a moderately strong and
statistically significant correlation (R=0.25, p <.05,) between change in number of volunteers
and growth in overall ODT score, there is no such correlation for the smallest income grant
holders (<250,000). For the 250,001 - 500,000 group, on the other hand, the correlation
is more positive than the overall level (R=0.3, p <.05,). This indicates that it is size of
organisation in combination with change in volunteer numbers explaining this relationship,
rather than solely change in volunteer numbers.

This positive relationship exists across all ODT sub-scores but is most pronounced in terms of
the Market & opportunities score, where another moderately strong positive correlation
(R=0.27p <.05,) was found.

16
3.4 Generating greater income
By coding grant holder application forms, we were able to classify which organisations were
aiming to generate greater income through their LSF project; 134 grant holders in total.
Comparing those grant holders who were planning to generate income with those who were
not showed little difference in level of score growth between the two groups. The coding of
application forms also allows us to classify how grant holders were planning to generate
income, either through developing existing funding streams or by developing new funding
streams and diversifying income. Again, in terms of overall ODT score there were limited
differences observed between groups. However, for the Market & opportunities sub-score
there were some interesting findings, explored in section 4.

3.5 Improving systems

Again through the coding of application forms we were able to classify which organisations
were aiming to improve systems via their LSF project. Grant holders who were improving
systems were somewhat more likely to have achieved Moderate/high overall ODT score
growth (84%) than those who were not (74%). In terms of ODT sub-scores, improving
systems had the most marked effect on the Track record & capability score, which will be
examined specifically in section 4.

3.6 Age of organisation

How long an organisation has existed appears to have had little relationship with either the
level of overall ODT score growth or sub-score growth. Organisations established in the
1990s were somewhat more likely to have seen Moderate/high levels of score growth than
other groups. However, low sample sizes for some of the other age groups means making
comparisons is not valid in this case.

4. Sub-score analysis
This section further explores the data around some of the ODT sub-scores outlined earlier in
the bulletin and builds upon findings highlighted in Section 3 that are related to these. The
Market & opportunities, Track record & capabilities and Quality & impact sub-scores all
exhibit some interesting relationships with various characteristics of LSF project that do not
necessarily exist for overall ODT score. These will be examined below.

4.1 Market & opportunities

Existing versus new funding streams
As mentioned above, the way in which grant holders were intending to generate greater
income through the LSF project appears to have had some impact upon the level of
improvement they achieved on the Market & opportunities score.

17
At an overall level, 69% of orgs that planned to generate greater income in any way
improved their market and opportunities score by a Moderate/high amount, compared to
61% of those that did not. This difference in Market & opportunities score was greater for
organisations that were ONLY diversifying and developing new funding streams, 73%
compared with 61% for those not including this in their project. Organisations that planned
to develop existing income were less likely to see Moderate/high growth in score (60%) than
those that did not include this activity in their application (68%). The chart below shows the
variation in Moderate/high score growth for Market & opportunities, with those
organisations who were developing existing funding streams having reduced scores (by 8%)
compared with increases of between 8% and 12% for those that did not.

Difference in Moderate/high levels of Market & opportunities score

growth by generating greater income activity (%)

15
11.8

10 8.3 8.6

5
Developing existing
funding streams
0
Generating greater Developing new funding ONLY developing new
income (total) streams & diversifying funding streams &
-5
income diversifying income

-10 -8.2

Open versus closed organisations

As a part of the Further analysis of ODT and application data bulletin we produced earlier in
the evaluation, a typology of grant holders was developed based on how open or closed
they were, in terms of user involvement as well as their pursuit of partnerships or
relationships. Open organisations actively involved users and were pursuing
partnerships/relationships. They felt their board was balanced with regard to the proportion
of board members representing users/the community. They were actively involved in some
important networks and partnerships and were investing significant resources in
being/becoming an established and respected organisation in their field. Closed
organisations on the other hand werent necessarily involving their users or pursuing
partnerships/relationships to the same extent. They acknowledged their board was relatively
unbalanced, with an under-representation of users/the community. Whilst they were

18
possibly involved in local networks, there were gaps in their links to others and they lacked
active involvement with partnerships or networks which would enhance the organisations
knowledge, skills or opportunities.

Interestingly a larger proportion of grant holders classified as closed saw Negative/zero

growth in Market & opportunities score (18%) than open organisations (7%). This may
suggest that being open as an organisation is connected at some level to some of the
factors that could help contribute to increasing organisational sustainability: for example,
accessing new sources of income or developing new partnerships and networks. However,
data from other parts of the evaluation suggests that there can nonetheless be important
motivations behind being more closed: many organisations had deliberately focused on
improving internal systems and processes before they could look outwards, and saw this as
an important part of their journey towards improving their sustainability.

4.2 Track record & capabilities

Improving systems
As highlighted in Section 3, whether a grant holder was aiming to improve systems via their
LSF project appears to have had a significant impact on the level of growth they achieved for
the Track record & capabilities score.

Around nine out of ten grant holders which were improving systems saw Moderate/high
score growth (88%) against 72% of those that were not. This finding indicates that improving
systems is a potential route for grant holders to build their capacity, something that
ultimately helps organisations to improve their sustainability and thus their overall ODT
score.

Open versus closed organisations

Unlike the pattern observed for the Market & opportunities score, where Open
organisations were more likely to see stronger score growth, for the Track record &
capabilities score it was Closed organisations which were most likely to see improvements
in this score. Although differences were not as marked as for some other relationships, 82%
of Closed organisations achieved increases in their Moderate/high score compared with
73% of Open organisations. This may suggest that these Closed organisations are more
inward-looking and focussed on developing their own reputation and track record, rather
than looking to exploit external market opportunities.

4.3 Quality & impact

Impact assessment
Somewhat unsurprisingly, those organisations that included impact assessment in their LSF
project application were more likely to see improvements in this sub-score than those who

19
did not. At an overall level, 81% of organisations undertaking impact assessment saw
Moderate/high levels of growth compared with 56% of those that were not, demonstrating
a clear difference between the two groups.

Furthermore, the purpose of the impact assessment can also be ascertained from our coding
of application forms, falling into two main groupings: impact assessment for funding, and
impact assessment for service development. The chart below shows the levels of growth
across the types of impact assessment used and shows that impact assessment for service
development saw a slightly higher proportion of Moderate/high score growth than impact
assessment for funding.

However, almost a third (32%) of those grant holders employing impact assessment for
funding were likely to see the highest levels of score growth (16%+), more than twice the
proportion of those that were not (14%). A similar relationship existed for among those who
were using impact assessment for service development, with 29% achieving 16%+ growth
compared with 16% of those that were not.

Level of Quality & impact score growth by Impact assessment (%)

100

80

60 81.3 81.5 87.8

40

20
18.7 18.5 12.2
0
Impact assessment (total) Impact assessment for Impact assessment for
funding service development
Negative/low growth Moderate/high growth

4.4 Sustainability
In terms of the Sustainability sub-score, the findings of note have been covered in Section 3
following the same patterns as overall ODT score.

20
4.5 Strategy & planning
In terms of the Strategy & planning sub-score, again the interesting findings have been
covered in Section 3 following the same patterns as overall ODT score.

4.6 User involvement

There are some indications that those who have targeted user involvement as a part of their
LSF project were more likely to see Moderate/high sub-score growth for Quality & impact.
Nearly three-quarters of grant holders that included user involvement in their application
saw Moderate/high score growth (73%) compared with 65% of those that did not. However,
with a relatively small difference between the groups of only 8%, we need to be cautious
about what this means and how much can be inferred from the finding.

4.7 Improving systems

As was seen with the Track record & capabilities sub-score, those improving systems are
more likely to see stronger growth in their Quality & impact score than those that were not,
73% seeing Moderate/high growth compared with 63% of those that were not.

5. Conclusions
This bulletin has analysed the different levels of change from pre to post in ODT score and
sub-scores, as well as exploring some of the factors that may lie behind the differing levels
observed. The potential linkages between sustainability and various key themes such as size
of organisation and the different aims of LSF projects were examined. The five sub-scores
were then also investigated to uncover any relationships that were specific to any specific
element of sustainability. The analysis undertaken allowed us to make some speculative
conclusions about both the potential routes grant holders have taken towards improving
sustainability and the broad profiles of grant holders which have achieved different levels of
score growth. These key findings were discussed in Section 1 of the bulletin.

21
Appendix: The Online Diagnostic Tool and our analysis

266 grant holders completed the ODT before their projects and 227 had completed it by the
end of July 2017, the point at which we analysed the data. No individual organisations were
named in this analysis but unique identifiers have been used to help us identify the change
between different categories of organisations.

The ODT contains a variety of different scores, including an overall measure of organisational
sustainability and multiple sub-scores which feed in to this over-arching score. This bulletin
reports on a variety of different measures when comparing the pre and post scores,
highlighting where there appears to be an interesting change and also where no change (or a
negative change) has been seen.

In July 2017 a bug was discovered in the program underlying the tool; in the score
calculations one question had been over-represented, resulting in the calculated scores
being higher than they should have been. The bug was fixed by Creative Co-op on 27th July
2017. The direct impact was on two indicators: Indicator 4, Financial Systems delivering,
where the score for that indicator was overstated by up to 18 percentage points and
Indicator 1, Cost base control, where the score for that indicator was overstated by up to four
percentage points. These indicators fed into three different scores, the Sustainability,
Strategy & Planning and Track Record & Capability scores, all of which were overstated by
three to four percentage points. The overall diagnostic score was overstated by up to two
percentage points. The error occurred consistently for the ODT scores for LSF grant holders
who used the tool before they received funding and after the end of their 12-month
projects, so the comparison between the before and after is still valid. Given this, and
given the fact that the overstatement on the overall diagnostic score was relatively small, we
agreed with the Fund to continue using the uncorrected scores for the analyses in this
bulletin, for the previous bulletin on the 'before' scores, and for the evaluation overall. To
ensure consistency a small number of organisations (five) were excluded from the
final analysis because their later scores had been calculated on the basis of the corrected
program after July 27th meaning the before and after scores would not have
been comparable.

22