OECD Framework For The Evaluation of SME and Entrepreneurship Policies and Programmes

P
O
L
IC
Y
E
V
A
L
U
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T

ENTREPRENEURSHIP PO
LICY EVALUATIO
N FIRST FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH FO
URTH THRID SECO
ND FIRST FIF
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
SM
E PO
LICY EVALUATIO
N ENTREPRENEURSHIP D FIRST FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH FO
URTH FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E

PO
LICY EVALUATIO
N ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N ENTREPRENEURSHIP S
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
I
SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATI
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O

PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N ENT
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N
ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
I
SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATI
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O

PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N ENT
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V

PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N ENTREPRENEUR
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
N
S
M
E
E
N
T
R
E
P
OECD Framework
for the Evaluation of SME
and Entrepreneurship
Policies and Programmes
-:HSTCQE=UYUU]X:
The full text of this book is available on line via this link:
www.sourceoecd.org/industrytrade/9789264040083
Those with access to all OECD books on line should use this link:
www.sourceoecd.org/9789264040083
SourceOECD is the OECDs online library of books, periodicals and statistical databases.
For more information about this award-winning service and free trials, ask your librarian, or write to
us at SourceOECD@oecd.org.
ISBN 978-92-64-04008-3
85 2007 04 1 P
OECD Framework for the Evaluation
of SME and Entrepreneurship Policies
and Programmes
This Framework provides policy makers with a concrete, explicit, practical and accessible
guide to best practice evaluation methods for SME and entrepreneurship policies
and programmes, drawing upon examples from a wide range of OECD countries.
It examines the benets of evaluation and how to address common issues that arise
when commissioning and undertaking SME and entrepreneurship evaluations. Key
evaluation principles are set out, including the Six Steps to Heaven approach, and
illustrated with examples of evaluations of national, regional and local programmes
that can be explored further by the reader. The publication focuses not only on the
evaluation of individual policies and programmes but also on bigger picture peer
review evaluations and assessment of the impact on SMEs and entrepreneurship
of mainstream programmes that do not have business development as their principal aim.

O
E
C
D

F
r
a
m
e
w
o
r
k

f
o
r

t
h
e

E
v
a
l
u
a
t
i
o
n

o
f

S
M
E

a
n
d

E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p

P
o
l
i
c
i
e
s

a
n
d

P
r
o
g
r
a
m
m
e
s
852007041cov.indd 1 11-Dec-2007 10:43:50 AM
OECD Framework
ORGANISATION FOR ECONOMIC CO-OPERATION
AND DEVELOPMENT
The OECD is a unique forum where the governments of 30 democracies work
together to address the economic, social and environmental challenges of globalisation.
The OECD is also at the forefront of efforts to understand and to help governments
respond to new developments and concerns, such as corporate governance, the
information economy and the challenges of an ageing population. The Organisation
provides a setting where governments can compare policy experiences, seek answers to
common problems, identify good practice and work to co-ordinate domestic and
international policies.
The OECD member countries are: Australia, Austria, Belgium, Canada, the
Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland,
Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand,
Norway, Poland, Portugal, the Slovak Republic, Spain, Sweden, Switzerland, Turkey,
the United Kingdom and the United States. The Commission of the European
Communities takes part in the work of the OECD.
OECD Publishing disseminates widely the results of the Organisations statistics
gathering and research on economic, social and environmental issues, as well as the
conventions, guidelines and standards agreed by its members.
Also available in French under the title:
Cadre de lOCDE sur lvaluation des politiques et des programmes lgard des PME
et de lentrepreneuriat
Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda.
OECD 2007
No reproduction, copy, transmission or translation of this publication may be made without written permission.
Applications should be sent to OECD Publishing rights@oecd.org or by fax 33 1 45 24 99 30. Permission to photocopy a
portion of this work should be addressed to the Centre franais dexploitation du droit de copie (CFC), 20, rue des
Grands-Augustins, 75006 Paris, France, fax 33 1 46 34 67 19, contact@cfcopies.com or (for US only) to Copyright Clearance
Center (CCC), 222 Rosewood Drive, Danvers, MA 01923, USA, fax 1 978 646 8600, info@copyright.com.
This work is published on the responsibility of the Secretary-General of
the OECD. The opinions expressed and arguments employed herein do not
necessarily reflect the official views of the Organisation or of the governments
of its member countries.
FOREWORD
OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007
3
Foreword
The OECD Working Party on Small and Medium-sized Enterprises and
Entrepreneurship (WPSMEE), in line with a recommendation of the 2004 Istanbul
Ministerial Declaration on Fostering the Growth of Innovation and Internationally
Competitive SMEs, has prepared this report aimed at strengthening the conceptual
framework for SME policy evaluation. This report seeks to be of direct practical
assistance to public administrators and politicians concerned with evidence on the
effectiveness of SME and entrepreneurship policies and programmes at a national and
local level.
The Framework was written by Dr. Jonathan Potter, Principal Administrator,
OECD Centre for Entrepreneurship, SMEs and Local Development, and Prof. David
Storey, Warwick Business School, UK, and prepared under the supervision of Mme
Marie-Florence Estim, Deputy Director of the OECD Centre on Entrepreneurship,
SMEs and Local Development (CFE).
A Steering Group, co-chaired by Dr. Roger Wigglesworth, New Zealand and
Mr. George Bramley, United Kingdom, guided the preparation of the Framework. The
Co-Chairs along with the members of the Steering Group offered many valuable
comments during the drafting, revisions and review of the Framework: Mrs. Sue
Weston and Ms. Vicki Brown, Australia; Mrs. Laura Morin, and Ms. Kaili Levesque,
Canada; Ms. Annukka Lehtonen and Mr. Pertti Valtonen, Finland; Mr. Serge Boscher
and Mr. Jean-Hugues Pierson, France; Mr. Tamas Lesko and Dr. gnes Jnszky,
Hungary; Mr. Young-Tae Kim and Dr. Sung Cheon Kang, Korea; and Ms. Ana Mara
Lagares Prez, Spain.
Sincere appreciation is extended to the Delegates of the OECD WPSMEE for their
numerous comments and inputs during the compilation of the Framework.
Thanks also go to Mr. Kevin Williams, Principal Administrator, OECD Council and
Executive Committee Secretariat, Mr. Hans Lundgren, Head of Section, Evaluation,
Development Co-operation Directorate, and Mrs. Mariarosa Lunati, Administrator, CFE/
SME and Entrepreneurship Division for their drafting suggestions and Ms. Brynn Deprey,
Mr. Jorge Glvez Mrdez, Mr. Damian Garnys, and Ms. Elsie Lotthe for their operational
support.
TABLE OF CONTENTS
5
Table of Contents
Summary and Route Map. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Section 1. Evaluation Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Defining evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Why do an evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Typical objections to evaluation and responses . . . . . . . . . . . . . . . . . . 19
Key evaluation debates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Doing evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Some key principles for evaluation practice . . . . . . . . . . . . . . . . . . . . . 32
Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Section 2. Evaluation of Individual National Programmes. . . . . . . . . . . . . . 37
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Evaluations of financial assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Enterprise culture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Advice and assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Section 3. Evaluation of Regional and Local Programmes . . . . . . . . . . . . . . 53
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Advice, consultancy and financial assistance . . . . . . . . . . . . . . . . . . . . 55
Clusters and local innovation systems . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Support to areas of geographical disadvantage. . . . . . . . . . . . . . . . . . . 59
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Section 4. The Role of Peer Review in Evaluation . . . . . . . . . . . . . . . . . . . . . 67
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
The peer review methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
OECD national SME reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
OECD regional and local entrepreneurship reviews . . . . . . . . . . . . . . . 71
OECD evaluation guidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Section 5. Reviewing the Aggregate Impact of Public Policies . . . . . . . . . . 75
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Impact of mainstream policies on SMEs. . . . . . . . . . . . . . . . . . . . . . . . . 77
Capturing the total policy package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
TABLE OF CONTENTS
6
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Appendix B. Six Steps to Heaven: Methods for Assessing
the Impact of SME Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Appendix A. The

OECD

Istanbul

Position
. . . . . . . . . . . . . . . . . . . . . . . .
103
Appendix C. Examples of Evaluation Guidance . . . . . . . . . . . . . . . . . . . 109
Appendix D. Assessing the Quality of an Evaluation. . . . . . . . . . . . . . . 111
Appendix E. Framework Condition Indicators: Entrepreneurship
Conditions in Denmark in 2005 . . . . . . . . . . . . . . . . . . . . . 113
Appendix F. Summary of the Evaluation of State Aid to SMEs
in the Member States, European Economic Area
and the Candidate Countries . . . . . . . . . . . . . . . . . . . . . . . 120
List of tables
1.1. Qualitative compared with quantitative evaluation . . . . . . . . . . . . 23
1.2. The choice of internal and external evaluators . . . . . . . . . . . . . . . . 25
2.1. SME and entrepreneurship policy areas covered . . . . . . . . . . . . . . . 38
2.2. Loan guarantee scheme, Japan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.3. Loan guarantee scheme, Canada . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4. Assistance to new enterprises started by young people, Italy . . . . 40
2.5. Grant assistance and small firm performance, Ireland. . . . . . . . . . 40
2.6. Public subsidies to business angels: EIS and VCT, UK . . . . . . . . . . . 41
2.7. Public subsidies to business angels: EIS, UK . . . . . . . . . . . . . . . . . . . 41
2.8. Assisting young disadvantaged people to start up businesses, UK . . 42
2.9. Graduates into business, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.10. Investment readiness, New Zealand . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.11. Impact of marketing advice, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.12. Impact of business advice, Belgium . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.13. Impact of advisory support, Bangladesh . . . . . . . . . . . . . . . . . . . . . . 44
2.14. Bank customers receiving business advice, UK . . . . . . . . . . . . . . . . 45
2.15. Assistance and advice for mature SMEs, UK. . . . . . . . . . . . . . . . . . . 45
2.16. Use and impact of business advice, UK . . . . . . . . . . . . . . . . . . . . . . . 46
2.17. Evaluating entrepreneurial assistance programs, US . . . . . . . . . . . 46
2.18. Encouraging partnerships amongst SMEs, Sweden . . . . . . . . . . . . . 47
2.19. Technology assistance to small firms, US . . . . . . . . . . . . . . . . . . . . . 48
2.20. The SBIR program, US. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.21. The UK SMART programme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.22. Impact of science parks, Greece . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.23. Impact of science parks, Sweden . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.24. University/SME links, New Zealand . . . . . . . . . . . . . . . . . . . . . . . . . . 50
TABLE OF CONTENTS
7
2.25. Impact of management training on SMEs, UK . . . . . . . . . . . . . . . . . 51
2.26. Small firms training loans, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.1. Regional/local policy areas covered . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2. Subsidised consulting, Belgium, Wallonia. . . . . . . . . . . . . . . . . . . . . 56
3.3. Business advisory services, UK, South West England . . . . . . . . . . . 56
3.4. Enhancing the capability of the SME owner through use
of consultants, UK, Scotland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.5. Export information and advice, Canada, Quebec . . . . . . . . . . . . . . . 57
3.6. Enterprise partnerships for exporting, Sweden, rebo . . . . . . . . . . 58
3.7. Small business grants, UK, North East England . . . . . . . . . . . . . . . . 58
3.8. Regional development agency grants, Ireland, Shannon . . . . . . . . 59
3.9. Local innovation system policy, EU regions . . . . . . . . . . . . . . . . . . . 60
3.10. Business networking, UK, North East England . . . . . . . . . . . . . . . . . 61
3.11. Enterprise Zone evaluation, US, Indiana . . . . . . . . . . . . . . . . . . . . . . 61
3.12. Enterprise Zone evaluation, US, Five States . . . . . . . . . . . . . . . . . . . 62
3.13. Enterprise Zone evaluation, UK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.14. Evaluation of enterprise support in disadvantaged areas, UK . . . . 63
3.15. Regional policy evaluation, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.16. Regional policy evaluation, Italy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.17. Rural policy evaluation, Canada, Quebec. . . . . . . . . . . . . . . . . . . . . . 64
3.18. Rural enterprise support, United Kingdom, Northumberland . . . . 65
5.1. The indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.2. Ease of Doing Business ranking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.3. Starting a business in 1999, 2004 and 2006 . . . . . . . . . . . . . . . . . . . . 85
5.4. Average conversion rates young businesses/nascent
entrepreneurs, 2000-2004. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.5. Selecting policy areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
B.1. Six Steps to Heaven: Methods for assessing the impact
of SME policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
D.1. Grid for a synthetic assessment of the quality
of evaluation work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Figure
1.1. New Zealand Trade and Enterprise (NZTE) Growth Range
Programme Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
ISBN 978-92-64-04008-3
OECD Framework for the Evaluation of SME and Entrepreneurship
OECD 2007
9
Summary and Route Map
This Framework document provides a forum for the international exchange
of knowledge on best practice evaluation of Small and Medium-sized
Enterprise (SME) and Entrepreneurship policy. Its target readership is public
administrators and policy-makers concerned with the formulation,
development and implementation of SME policy, together with professionals
concerned with evaluation of such policies. It seeks to be concrete, explicit,
practical and accessible, drawing upon examples from a wide range of OECD
countries. Almost all the evaluations documented are publicly available
online. It is also intended that the text will assist SME policy makers in non-
member countries.
In line with the OECD Istanbul Position, which underlines the need to
strengthen the culture of evaluation of SME and entrepreneurship policies
(Appendix A), this document has four objectives:
To increase the awareness of politicians and public officials of the benefits
from having an evaluation culture.
To disseminate examples of good micro evaluation practice at national and
sub national levels.
To highlight key evaluation debates: Who does evaluations? What
procedures and methods should be used? When to do the evaluations?
What about the dissemination of findings? Should all policies be
disseminated in the same way?
To make a clear distinction between policies that operate at the micro level,
i.e. SME and entrepreneurship specific policies, and those that operate at
the macro level, i.e. mainstream policies that nonetheless influence SMEs
and entrepreneurship.
To achieve this end, the Framework is divided in three main parts. The
first deals with evaluations of micro-entrepreneurship and SME policies
formulated and delivered at the national level. The second deals with
entrepreneurship and SME policies delivered at the local/regional level. The
third section is rather different. It reviews approaches to establishing the
aggregate impact of a range of public policies that strongly influence
entrepreneurship and SME performance, yet are rarely the responsibility of
SUMMARY AND ROUTE MAP
10
the main department of government responsible for SMEs. Prior to that, the
Framework reviews good practice in evaluation more generally.
It should be noted that the Framework does not seek to be a handbook or
manual that sets out the steps that need to be taken to complete an
evaluation. A substantial body of such handbooks and manuals exist and
selected examples are provided in Appendix C. Rather the focus of the
Framework is on discussing the difficult issues that arise in evaluating SME
and entrepreneurship policies and programmes, particularly with respect to
quantitative impact evaluation, and providing examples of evaluation
approaches that have been used to address these issues. The Framework
should therefore be read in conjunction with, rather than in place of, other
evaluation guidance in this field.
This summary provides a route-map for the reader, highlighting its key
conclusions. It then moves on to setting out the key conclusions from each of
the three parts. Finally it sets out a proposal for continuous improvement in
the evaluation of SME policy.
So, why do evaluation?
To establish the impact of policies and programmes.
To make informed decisions about the allocation of funds.
To show the taxpayer and business community whether the programme is
a cost-effective use of public funds.
To stimulate informed debate.
To achieve continued improvements in the design and administration of
programmes.
When and how should programme evaluation be done?
Evaluation has to be integral to the policy process. Hence there is merit in
undertaking prospective evaluations as policy options are being
formulated; formative evaluations as the policy is in operation; and
summative evaluations once a clear policy impact can be judged. The
summative evaluation findings have to feed back into current policy
making.
For summative evaluations we favour a dual approach. The first is to
establish the impact of established large scale programmes by using
quantitative, statistical methods using control groups that score highly on
the Six Steps to Heaven metric.
These can be valuably complemented with qualitative approaches such as
case studies and peer reviews for more detail on how policy works and how
11
it may be adjusted. Qualitative approaches are also useful for smaller scale
programmes for which the costs of quantitative evaluation may be too high.
And by whom?
Evaluation undertaken by specialists is essential for reliable impact evaluation.
Sometimes the necessary independence can only be delivered by outsiders
but independent evaluation units within government can also perform this
role.
The bedrock of good evaluation comprises:
The programme has to have clearly specified objectives from which it is
possible to determine whether or not it succeeded.
The evaluation has to be set in progress and data collection begun as, or
even before, the programme is implemented.
The evaluation has to be able to lead to policy change.
The evaluation of national programmes
This section of the Framework provides examples of evaluations that have
been undertaken on the following policy areas: Financial Assistance;
Enterprise Culture; Advice and Assistance; Technology; and Management
Training.
It concludes that, whilst there are examples of high quality evaluations, this
is not the norm.
Broadly, lower quality evaluations seem to produce more favourable
outcomes for the project because they attribute observed change to the policy
when this may not be justified.
The evaluation of local and regional programmes
At the regional and local level less costly and less sophisticated approaches
are often adopted because the programmes are often smaller and because
evaluation structures in terms of information bases, professional
evaluation capabilities and understanding of evaluation methods by users
may be weaker.
The work of the OECD Local Economic and Employment Development
(LEED) Programme with city and regional governments and development
agencies has shown that a critical issue for policy development is increasing
understanding of the real policy needs of the region or locality and
assessing the alternative options for intervention given the specific local
context.
12
Peer review: a tool for evaluation
Whilst evaluation of programme impacts is still required, broader peer
reviews are also useful in providing big picture assessments of the full
range of entrepreneurship and SME policies including in selected regions.
Reviewing the aggregate impact on SMEs and entrepreneurship
of public policies
Although explicit and targeted SME and entrepreneurship policies influence
the creation of new firms and the development of SMEs, so also do other
government policies which do not have such a focus. They are also rarely
the responsibility of the main SME department of government. These
policies include control of interest rate and tax policies, social policies such
as the setting of unemployment benefits, the cost and time of starting a
new business and the role of immigration and emigration.
These policies represent substantial expenditures in many countries and
our review shows they impact powerfully on entrepreneurship and SME
development. However, control, or influence, over that total expenditure is
rarely exercised by the department of government responsible for SME
policy. Instead, other departments or organisations of government often
have considerably larger budgets, but may have different priorities to that of
the main SME department.
The challenge for SME and entrepreneurship policy makers is to identify
these macro policies and their links to enterprise. It is then to seek to
ensure that they work in a way which is congruent with the objectives of
enterprise support.
Evaluation approaches need to be developed that permit policy makers with
SME and entrepreneurship responsibilities to be able to engage more fully in
cross-government discussions on priority setting.
Future work
The current document is not to be regarded as the definitive or final
statement on how SME policy and its constituent parts should be evaluated.
In our judgement there remain too few examples of top quality evaluations. We
also have too little knowledge about the impact which these evaluations have had
upon the formulation of policy and the impact which policy changes have upon
SMEs and the economy more widely for this to be the last word. We therefore
propose that this document should evolve over time to reflect the direct interests
13
of policy makers, SMEs and the taxpayer. A future text could benefit from the
following:
More examples of high quality evaluations that can be shared between
countries; and
More evidence, probably in a case study format, of the links between
evaluations undertaken and policy changes. An example here might be the
review by OECD of SME policy in Mexico and the changes that subsequently
occurred in that country.
In short, what we are able to provide in this current text are some generic
approaches to evaluation and some examples of evaluations undertaken,
some of which are better than others in terms of their technical merit.
ISBN 978-92-64-04008-3
OECD 2007
15
Section 1
Evaluation Issues
1. EVALUATION ISSUES
16
Defining evaluation
In their review of policy evaluation in innovation and technology,
Papaconstantinou and Polt (1997) provide a very helpful definition of
evaluation. They say: Evaluation refers to a process that seeks to determine
as systematically and objectively as possible the relevance, efficiency and
effectiveness of an activity in terms of its objectives, including the analysis of
the implementation and administrative management of such activity.
Several words or phrases in this definition merit strong emphasis. The
first key-word is process. This emphasises that evaluation is not a once-off
activity, undertaken once a particular programme has been completed.
Instead it is an integral element of a process of improved policy or service
delivery.
A second key phrase in the definition of evaluation is as systematically
and objectively as possible. Given that evaluation traditionally takes place at
the end of the line
1
there are likely to be strong entrenched interests in place
once a programme has been in existence for a number of years. These
entrenched interests include the direct beneficiaries of the programme, such
as the businesses receiving funds, but they will also include those who are
responsible for initiating and administering these programmes. All else held
equal, it is to be expected that all these groups will choose the programme to
continue or expand. The task of the evaluator, however, is to systematically
and objectively assess the merits of the programme. In this task, the evaluator
may well conflict with those committed to the programme. Only through the
use of objective techniques, discussed later in the paper, can the evaluator
demonstrate their independence to those delivering programmes.
The third key phrase in the definition is the relevance, efficiency and effect
of an activity in terms of its objective. The implicit assumption in this statement
is that the policy has clear objectives and that these are stated in sufficiently
clear terms for them to be used by the evaluator. In practice, this is by no
means always the case. As will be shown later, a key role for evaluators is often
to formalise for the first time the objectives of programmes, often after such
programmes have been in operation for many years.
This definition, and the OECD Istanbul paper,
2
emphasised that
evaluation has an integral role to play in the policy process. Evaluation cannot
be left at the end of the line. Instead, it has to be a key element of initial
17
policy formulation. Once the policy is operational, all organisations and
individuals responsible for delivery have to be aware that evaluation is to take
place. Once the evaluation has been undertaken, and sometimes as it is taking
place, it should be used as the basis for dialogue with policy makers, with the
objective of delivering better policy. The outcome of the evaluation can then
become an input into a debate on the appropriate ways for governments and
SMEs to interact.
Why do an evaluation?
Whilst some countries have a long established tradition of undertaking
evaluation, others do not. For those seeking to champion a culture of
evaluation, the following arguments summarise the case in favour. We then
also take the arguments that are often used against evaluation and address
them.
To establish the impact of policies and programmes against their
objectives
The principal reason for doing evaluation is to establish whether or not
policy has contributed to correcting or ameliorating the problem it set out to
resolve. This is often thought of in terms of tackling market failures that
reduce economic efficiency, such as inadequate availability of finance, skills,
advice and technologies, but may also encompass a desire to improve equity
among groups of people or places, for example by supporting entrepreneurship
among unemployed youth or entrepreneurship in poor localities. Evaluation
of these impacts is facilitated by a clear statement of measurable outcomes
right at the start of the policy/programme design and the collection of relevant
data throughout its life.
To make informed decisions about the allocation of funds
Governments manage a portfolio of policies and programmes each with
its own rationale and justification. Evaluation assists managers to assess the
relative effectiveness of these policies and programmes and to make
judgements about where to place their efforts in order to obtain the greatest
benefits for given costs. Evaluation evidence can help to identify where
government can make the biggest difference to its objectives and targets.
To show the tax payer and business community whether
the programme is a cost effective use of public funds
The scale of tax-payers funding for entrepreneurship and small business
policies clearly varies from one country to another. It also varies according to
precisely what is incorporated into the definition. Nevertheless the amounts
18
are usually substantial. For example, EIM (2004) reports that approximately
six billion Euros was spent annually by EU Member States on state aid to small
and medium sized enterprises.
3
However, even this may be a considerable
underestimate. One EU Member State the United Kingdom in a
comprehensive review of tax-payers funding directed towards SMEs, reported
that 2.5 billion GBP of public money was spent on direct support to SMEs in
England alone, PACEC [2005] quoted by [National Audit Office (2006)]. A third
example is a programme in the United States the Small Business Innovation
Research Program. Cooper (2003) reports this programme made annual awards
of USD 1.1 billion in the 1997-1999 calendar years.
4
These examples illustrate that, probably for most developed countries,
public funding of SMEs is substantial, even if it is extremely difficult to
quantify in aggregate and may still be relatively modest in terms of tax-payers
support to large enterprises. Given these substantial sums of public money, it
is reasonable for tax-payers to be reassured that their funding is being spent
in an appropriate manner. It is reasonable for tax-payers to demand evidence
that public programmes are spending funds in accordance with their stated
objectives. This role is normally played by public auditors. A second role, but
one not normally played by auditors, is to assess whether the public funds are
achieving the objectives set out by politicians. This is the function of
evaluators.
To stimulate democratic debate
In democracies, it is reasonable for the electorate to question the
decisions made by governments. In order to facilitate that debate, it is
appropriate for organisations to be able to have access to evidence on the
impact of policies. In this regard, SME and entrepreneurship policies are no
different from other areas of government expenditure. For this reason, the
results of evaluations enhance and inform public debate.
This debate only takes place when the results of evaluations enter the
public domain. This emphasises not only the importance of undertaking
evaluations, but also of their findings being disseminated.
To achieve continued improvement in the design and administration
of programmes
Politicians and public servants administering SME and entrepreneurship
programmes should be seeking continuous improvements and there is of
course a need to ensure adaptation to changing conditions. Evaluation is a key
tool for learning about how well policies and programmes are delivering, what
problems may be emerging, what elements work well and less well and what
could be done better in the future. For example, policy makers may seek to
19
deliver policies to different groups, for example by directing more resources
towards enterprises established by the socially disadvantaged or by those
likely to employ others, or those in high technology. They may seek to deliver
policies using different organisational forms, to stimulate the take-up of
policies or to deliver them in a more cost effective manner. All these changes
of focus can emerge from undertaking appropriate evaluations. Alternatively,
existing policies can be delivered more effectively as a result of accumulated
evaluation experience.
Typical objections to evaluation and responses
The discussion above focussed on the positive aspects of evaluation.
However, one of the barriers to spreading an evaluation practice is a resistance
to evaluation amongst a range of politicians, policy makers and practitioners.
Here we discuss some of the most common objections to evaluation and the
degree to which they stand up to critical assessment. Our judgement is that
although the objections have some weight, on balance, they do not amount to
a solid case for rejecting evaluation and hence sacrificing the benefits cited
above.
But evaluation is expensive and bureaucratic
Evaluation is not costless. Costs include the payment of consultants/
evaluators, the collection of data and the time taken from those delivering
programmes to inform the evaluation. The United Kingdom statistical office,
for example, requires the time of recipients of the programme in providing
their opinions and information about the programme to be costed (i.e. the cost
of the respondents time must be explicitly included in the cost of the
evaluation). Data may also have to be collected from both clients of the
programme, and a control group of non clients.
However, the resources committed to evaluation are normally very
modest in comparison with the total size of the programme. For example, the
review by Sheikh and Steiber [2002], Evaluating Actions and Measures
Promoting Female Entrepreneurship identified an appropriate budget of
between 2% and 5% for the purposes of evaluation. This may be appropriate
for small programmes but for programmes in larger countries a figure of
between 0.5% and 1% of annual expenditure would be more usual.
Given the opportunity which evaluation provides for using resources
more efficiently, and for the design of new programmes, these seem to be very
modest costs indeed.
20
But evaluation does not always lead to policy improvements
Evaluations of programmes can fail to lead to policy change for several
reasons. It may be because those responsible for programme management are
hostile to the concept of evaluation. It can also happen where evaluators fail
to engage programme managers, or where they fail to understand the details
of the programme. Evaluators themselves may fail to express their findings in
a language that is easily understandable to policy makers and those
responsible for policy delivery.
Although there are instances where evaluations have not led to
improvement, this is not a sufficient justification for being reluctant to
undertake any form of evaluation. To minimise the potential problems,
programme managers have to be persuaded that the quality of programme
delivery can be enhanced through evaluations and the consultants have to
reach out to programme managers to engage them wherever possible.
But ultimately evaluation takes place for the benefit of the tax-payer, and
not for the provider[s] of the programme. Those programmes that are shown
to be demonstrably ineffective have to be closed, and this has to be recognised
by programme managers.
In practice, if evaluation is to lead to change, a balance must be struck
between, on the one hand, ensuring the independence of the evaluator whilst,
on the other, engaging support of those involved with programme delivery.
And risks diverting attention away from programme delivery
It is the case that there are cultural differences between evaluators and
deliverers of programmes. The former are often analytical individuals, often
with an academic background, whereas the latter consider themselves
practical individuals focused upon delivering services to their clients. Because
they are so close to their clients they view themselves as the best judge of the
effectiveness of the programme. They have difficulty seeing what value a
detached consultant can provide in terms of programme improvement. For
this reason, programme deliverers often resent the time taken in completing
forms and collecting data which are, however, vital to the success of an
evaluation. Programme managers and deliverers understandably can also feel
threatened by an evaluation, especially when they know they do not fully
understand the techniques used by the evaluators, but fear the evaluators do
not fully understand the programme.
For an evaluation to be a success however, these cultural differences have
to be managed. The most effective way of achieving this, as identified above,
is to demonstrate that the interests of both the evaluators and the programme
managers/deliverers can be more closely aligned by both parties focussing on
areas for programme improvement. This can be most effectively achieved by
21
engaging those delivering the policy through ensuring the issues of concern
are addressed in the evaluation and by being adequate opportunity to
comment upon, and offer their interpretation of, provisional findings.
It is, of course, a simplification to imply that the programme managers
most hostile to evaluation are those fearing negative feedback. Nevertheless
senior policy-makers need to be aware that evaluation, whilst it is in the
taxpayers interests, may provoke considerable hostility from programme
deliverers. The latter have to be engaged but not the ultimate voice.
But evaluation is only for advanced countries
It is the case that programme evaluation is more frequently undertaken
in advanced, rather than in developing economies. In part this may be because
it is more difficult to find sufficient numbers of individuals with the type of
analytical skills necessary to conduct good quality evaluations in developing
economies. Major donor organisations, such as the World Bank, can therefore
play a role in both undertaking evaluations themselves and in training others
to perform these tasks.
Nevertheless it is not only the most developed countries that undertake
evaluation. In its review of state aid to SMEs, EIM [2004] surveyed EU Member
States, European Economic Area and candidate countries. A total of
29 countries were identified. Only Ireland, the Netherlands and Slovakia
performed state aid evaluations on all schemes, implying that evaluation is
not simply characteristic of the more wealthy countries. EIM specifically noted
that the State Aid Act obliges the Slovak Government to evaluate all state aid
using statistical analysis of aid recipients and control groups. They also noted
that the analyses are performed on macro and micro levels. Full details of this
important survey are provided in Appendix F.
Mexico has also recently committed to undertaking SME evaluation. It
believes this will improve support systems and identify areas of opportunity,
thus granting certainty to the population on the efficient use of resources.
These examples illustrate that it is not necessarily the most economically
developed countries which are committed to undertaking evaluation.
But there is no history of undertaking evaluation
In countries without a tradition of evaluation it can be difficult to make
this transition. Nevertheless, it is clear that the electorates in many countries
are becoming more sophisticated, in part because of access to the media and
the internet. Countries where evaluations do not take place are likely, in the
future, to be asked why it is that such policy assessments take place
elsewhere. The, perhaps unjustified, inference is that evaluations do not take
place because there is something to hide. It is not sufficient to imply that
22
policies are being delivered efficiently because there is no information to the
contrary.
Key evaluation debates
This section reviews four key evaluation debates. The first is the
appropriate technique for evaluating SME and entrepreneurship policies. The
second is the appropriate level of sophistication of the quantitative evaluation
approaches. The third is whether evaluation should be undertaken by
insiders or outsiders. The fourth is whether the same evaluation techniques
should be used for all programmes.
The choice of technique
There are two basic options in undertaking summative evaluations
5
the
quantitative and qualitative approaches. Quantitative evaluation involves
assessment of the impact of programmes through a comparison of outcomes
between the group in receipt of aid and some form of control group, for
example a similar group of enterprises that have not benefited from policy or
the same enterprises before and after receipt of policy support. Such data may
be collected either directly from the firms themselves or from official data.
Qualitative evaluation or approaches are much more likely to rely upon the
opinions of programme stakeholders including managers and beneficiaries
about the functioning and impact of the programme through techniques
including surveys, case studies and peer reviews. Both approaches will rely
upon a careful scrutiny of programme documentation.
Table 1.1 reviews the advantages and disadvantages of the quantitative
and qualitative approaches.
The principal advantage of qualitative evaluation is the additional
information that it can provide beyond that associated with quantitative
evaluations. Qualitative evaluation normally involves face-to-face discussions
with those in receipt of aid, those responsible for delivering programmes and
other stakeholders. These conversations help not only to obtain information
from stakeholders that can lead to a deeper understanding of the mechanisms
by which policy impact is achieved and how policy might be adjusted but also
to engage stakeholders in policy learning processes. The approach can also
pick up a wide range of other information of interest to policy makers, going
beyond impact to issues such as client satisfaction, policy appropriateness,
sustainability and conflict with other policies.
However, qualitative evaluation has the major disadvantage that it is not
good at providing reliable estimates of policy impact for a number of reasons.
First, surveys of a sample of stakeholders run the risk of being unrepresentative
of programme participants. Increasing the numbers however either adds
23
considerably to budgets or reduces the quality or depth of the interviews.
Second, despite the best efforts of interviewers, there remains a strong risk of
interviewer bias. Thirdly, the outcome of qualitative evaluation is more often
to describe a process rather than to evaluate an outcome. Fourthly, there is no
opportunity for independent verification. Finally, programme participants
may be asked questions that are virtually impossible to answer. The classic
example is What impact do you think this programme had on your
business? Implicitly the respondent is required to hold every other influence
on their business constant and estimate how a programme which probably
took place some years previously has influenced their business in the
intervening period. Even if some programme participants were able to
undertake such mental gymnastics others clearly are not and there is no way
of distinguishing between the answers of the two groups.
The principal disadvantages of the quantitative approach concern its
technical difficulties and the relatively narrow nature of the results it offers,
which focus primarily on issues of effectiveness and efficiency. In terms of the
technical issues, effective quantitative evaluation requires extensive data
collection on the performance of policy-targeted and control group firms.
More importantly, however, in SME and entrepreneurship policy evaluation
situations, there may sometimes be no natural, uncontaminated control
group. Whilst good quantitative analysis seeks to match as closely as
Table 1.1. Qualitative compared with quantitative evaluation
Qualitative evaluation methodologies Quantitative evaluation methodologies
Advantages Disadvantages Advantages Disadvantages
Engages participants in
policy learning
Respondents and
interviewers may be biased
or poorly informed
Clear answers on impact Cost of data collection and
technical demands
Can vary the scale and
hence cost
Rarely provides a clear
answer
If well done will get close to
true impact
Lacks information on
context and mechanisms
behind policy impacts
Deeper understanding of
processes leading to
impacts
Tends to describe rather
than evaluate
Can be independently
verified
Absence of pure control
groups
Should be easy to interpret Risks including un-
representative groups
Possible false impression
of precision
Can assess against a wide
range of evaluation criteria
No opportunity for
independent verification
Narrow focus on
effectiveness and efficiency
Picks up unintended
consequences
Hard to judge efficiency and
effectiveness
Difficult to use on indirect
interventions that seek to
influence the business
environment
Better understanding of
policy options and
alternatives
Hard to establish cause and
effect
24
possible policy-influenced and non-policy-influenced firms and seeks to
account for possible selection bias between the two groups, there are always
some differences between the treatment and control groups that cannot
be taken into account. To address this, evaluators in several OECD countries
have collaborated with their own statistical agencies to derive samples of
SMEs with prescribed characteristics so as to act as a control group. Even
so, some evaluators may be tempted to give a false impression of precision
in reporting their results. In terms of the nature of the results the main
drawback is the problem of the black box, i.e. that little information is
provided on the nature of the policy problem and how it is addressed by
policy and hence on how policy might be adjusted to increase impact. This
can be reflected in an unduly narrow focus of quantitative approaches on
two evaluation criteria, namely efficiency (impacts against expenditure) and
effectiveness (impacts against targets), that can leave other evaluation
questions unanswered.
On the other hand, the fundamental advantage of quantitative evaluation
is that it should provide clear answers. If it is well done it will get as close as
possible to a value-free assessment of impact. Of course, no evaluation is
wholly value-free.
Given the advantages and disadvantages of both approaches, this
Framework argues for the use of a plurality of approaches that are able to
gain from the complementarities in the information they can provide. The
role of the qualitative approach to evaluation is recognised and the role of
survey, case study and peer review approaches is outlined in this respect.
However, the Framework focuses in particular on setting out the issues
involved in undertaking good quantitative evaluations, reflecting the
original concern of the OECD Working Party on SMEs and Entrepreneurship
(WPSMEE) to share information on best practices in impact evaluation. This
reflects both the perception that quantitative impact evaluations are not
sufficiently used in SME and entrepreneurship policy evaluation and the
presence of some difficult issues that are not sufficiently well understood by
policy makers, particularly in accurately establishing the counterfactual.
Assessing quantitative evaluations: The Six Steps to Heaven
A useful guide in developing robust quantitative evaluations and
assessing the quality of such evaluation evidence is the so called Six Steps
to Heaven approach, (Storey, [2000]), reviewed and operationalised recently
by Lenihan et al. [2007], Bonner and McGuiness [2007], and Ramsey and Bond
[2007].
25
The Six Steps methodology is a categorisation in which Step 1 is the least,
and Step 6 the most, sophisticated approach. The six steps are:
Step 1. Take up of schemes.
Step 2. Recipients opinions.
Step 3. Recipients views of the difference made by the assistance.
The above three steps tend to be associated with qualitative approaches,
but the following three steps typify quantitative evaluations:
Step 4. Comparison of the performance of the assisted with typical firms.
Step 5. Comparison with match firms.
Step 6. Taking account of selection bias.
This is an approach that is mainly relevant to quantitative and ex post
evaluations rather than to qualitative and ex ante evaluation. It is nonetheless
a very helpful framework for assessing the former type of evaluations and is
referred to a number of times in this Framework, notably in relation to where
the evaluation examples provided later in the report stand in relation to the
different levels of sophistication in establishing impact.
Fuller details of the approach are to be found in Appendix B, which is
taken from OECD (2004a).
Evaluation by insiders or outsiders?
A second key evaluation debate is who should undertake the evaluation
should it be insiders or outsiders? The arguments for and against are set out
in Table 1.2 below.
The key argument in favour of using external evaluators is that they are
not only less likely to be influenced by the political regime, but they are also
more likely to be seen, by others, to be independent.
6
This independence is
likely to provide more objectivity to the evaluation. A second argument for the
Table 1.2. The choice of internal and external evaluators
External evaluator Internal evaluator
Advantages Disadvantages Advantages Disadvantages
Less likely to be influenced
by political regime
Less well informed of the
real situation
More insights through
understanding the realities
on the ground
Lack of independence
Seen by others to be
independent
Less able to drive through
change as a result of the
recommendations
More chance of buy-in
from those delivering
the programme
Less likely to be able to
think outside the box
Brings new ideas and fresh
approaches
More chance of really
changing policy
26
use of external evaluators is that they can bring new ideas and fresh
approaches not only to the evaluation but also to subsequent policy
development.
In contrast, the key advantage of using internal evaluators is that they
frequently have a much better knowledge both of the policy itself and of the
political context in which it is undertaken. Internal evaluators therefore have
to spend less time in acquainting themselves with the detailed workings of
policy and can focus much more upon producing targeted recommendations.
Internal evaluators also are more likely to engage the support of the managers
delivering the programmes because of their greater knowledge and because
they are perceived to be less threatening. Finally, internal evaluators are also
more likely to be careful about their policy recommendations, since they will
have to perhaps live with, and possibly implement, any changes they
recommend. Unlike external evaluators they are less likely to be able to walk
away from the issue.
The OECD (2004a) recognised that the choice of internal or external
evaluators was a close call. Much might depend upon whether, in commissioning
the evaluation, the purpose was to undertake a root and branch approach in
which case external evaluators might be preferred. In contrast, evaluation
designed to ensure programmes were on track might favour the use of
internal evaluators.
Ultimately, therefore, there is a broad choice between selecting
evaluators who are more independent but with perhaps less policy insight and
evaluators who may be less radical in their recommendations but who
perhaps are more likely to induce changes in programmes. The Istanbul
Ministerial Declaration, however, made it clear that it favoured independent
but informed evaluators.
It is also possible to develop alternative models that are neither fully
internal nor fully external. For example, some government departments and
agencies create independent evaluation units that are not directly attached or
responsible to the particular units responsible for the programmes that they
evaluate. Another option is to create teams of evaluators, with some coming
from inside and some from outside the organisation. This latter approach is
typical of the peer review method described in Section 4.
Should the same evaluation techniques be used for evaluating
all programmes?
A third debate is whether the same approaches should be used to review
all programmes. The central argument favouring a similar approach to
evaluation is that the ultimate purpose, if the tax-payer is to obtain value for
money from SME and entrepreneurship policies, is that all programmes
27
should have the same effectiveness at the margin. In simple terms it should
not be possible to transfer funds from one programme to another and increase
the benefits to SMEs and/or the wider economy. So, the economic impact of
policies to reduce taxes for SMEs should have the same marginal benefit as
policies to provide export advice or management training or access to finance.
There is clear evidence from work on SME evaluation that the methods
used for evaluation appear to influence the apparent effectiveness of
programmes and policies. Expressed baldly, the less sophisticated the
evaluation the more likely it is to apparently demonstrate benefits. This
reflects the more simple evaluations failing to hold constant the myriad of
influences on outcomes and, by implication, attributing them to the
programme. In contrast, the more sophisticated approaches strip out the
other influences, and so only attribute to the programme its real effects.
This finding has major implications because it means that it is not valid
to compare the findings from a study which uses a Step 2 or Step 3 approach
with that which uses a Step 5 or Step 6 approach. Indeed it may even be
invalid to compare findings between Steps 5 and 6. Hence it means that only
by using a uniform methodology can governments really ensure that
entrepreneurship and SME policy is efficiently delivered.
The opposing argument is the following: that programmes vary
considerably in scale and budget and that if a fixed proportion of programme
funds are to be allocated to evaluation then inevitably evaluation budgets will
also vary. More sophisticated evaluations are, of course, generally both more
expensive and with higher fixed costs than less sophisticated approaches.
This means that if the same approach were used across all programmes then
small programmes would have to devote a much higher proportion of their
funds to evaluation than is the case for larger programmes. This is unrealistic.
Both arguments, of course, have validity, but some form of compromise is
possible. If the desirability of uniform evaluation procedures is accepted, then it
still may be possible for individual smaller programmes to be evaluated less
frequently, or possibly as part of an evaluation of a package of small programmes.
What is clear is that programmes with small budgets should not either escape
from all evaluation or be assessed by radically different and by implication less
challenging procedures.
Doing evaluations
This section examines the practical issues of how to prepare, manage and
disseminate evaluations. Further useful information on preparing, managing and
disseminating evaluations is provided in the evaluation guidance documents
referred in Appendix C, including the web resources of the OECD Development
Assistance Committee (DAC) Evaluation Resource Centre.
28
Preparing an evaluation
A number of key issues have to be addressed when preparing an
evaluation: the first is to identify precisely what it is that will be evaluated.
This can be a problem when one item in a package of assistance is being
assessed. For example, some SME programmes combine both financial
assistance and business advice. It is therefore important, at the outset, to
decide whether the whole programme is to be evaluated or whether the
component parts are to be evaluated separately. The advantage of examining
the whole programme is that an overall assessment of the use of public funds
can be undertaken. But separately examining the component parts the
finance and the advice may show that it is only one or the other that is really
effective. For example, the evaluation might show that the impact on firm
performance is primarily due to access to finance. In that case, resources
might be moved away from the advice towards activities that improve access
to finance.
A second question is when the evaluation should be conducted. Here
again there is no simple answer because some forms of assistance take longer
to impact on firm performance than others. For example, a programme
designed to network firms with one another at a trade fair might be expected
to have an impact in terms of additional sales within 3-6 months. In contrast,
a programme to provide management training for SME owners might not be
expected to have significant impacts for at least 2-3 years. Finally,
entrepreneurship policies such as those designed to influence the attitudes
of school children to enterprise creation might not be expected to be
observable for at least 20 years. Given these varying likely outcomes the period
after the policy is implemented after the evaluation takes place is also likely to
vary. However, a broad rule of thumb is that SME policy initiatives such as the
impact of loans and grants should plan for the evaluation immediately the
policy is introduced and begin the formal evaluation within 2-3 years.
The objectives of the programme have to be clearly specified
Unless programmes have objectives which are in principle capable of
measurement then a quantitative evaluation cannot be undertaken. Very
often these objectives are set out in a logic model that provides policy makers
and evaluators with a clear understanding of possible programme outcomes
and how they are likely to be achieved. This is important for evaluation
because it provides a guide to what should be assessed and measured. Logic
models can take many forms, which will all be valid as long as they clearly
express what policy is seeking to achieve. An example from New Zealand is set
out in Figure 1.1. The University of Wisconsin online reference guide on
Enhancing Programme Performance with Logic Models provides further useful
29
information for the development of these tools.
7
The involvement of skilled
evaluators at the outset of programme formulation will help ensure that
objectives and programme logic are clearly stated. In practice, however, this is
not always the case. This means that a key function of the evaluator has to be
to infer perhaps even guess what the objectives of the policy maker were
when the programme was designed. Although this might seem curious, it is
often in practice a very valuable role of the evaluator, and even more so for
policy-makers and programme manager.
8
Specifying the content of the evaluation
Those responsible for preparing the evaluation have to be clear,
particularly when the evaluation is to be undertaken externally, about what it
is expected to achieve and what the role of the evaluation manager will be.
The latter may have to assist consultants in clarifying both the objectives of
the programme and the current requirements of politicians. However
evaluation managers should not normally specify in detail the methodology to
be used but merely identify the questions to be addressed such as
additionality, dead weight or displacement.
9
To specify the methodology in
detail would be to exclude the possibility of evaluators employing new or
novel approaches. The only clear exception would be where the purpose is to
directly compare policy impacts at two points in time. Here there would be
merit in using a similar methodology, providing the chosen method was
deemed to be satisfactory.
Ensuring good data are available
Whatever level of sophistication of evaluation is used, a minimum
requirement is that data are available. For example an evaluation of the
impact of business advice or of loans or grants requires, as a minimum, a
complete and up to date list of clients to be available. Until such information
exists no evaluation should even be contemplated.
Managing an evaluation
Six major issues arise in managing an evaluation.
Should the evaluator be internal or external?
This issue was discussed in-depth above and it was concluded that whilst
the external evaluator was more likely to be independent from the
organisation responsible for designing and delivering policy, and hence more
likely to be critical of it, the internal evaluator was likely to have greater
knowledge and political awareness. There would be, therefore, circumstances
30
in which either was appropriate but, all else equal, the independence of the
evaluator was critical.
The scale of the budget
As noted earlier the scale of budget strongly influences the methodology
which can be undertaken. It also influences the outcome of the evaluation
since it seems that inexpensive evaluations seem to produce more positive
findings. This can produce considerable pressure to undertake only the most
simple of evaluations.
Terms of reference
Terms of reference need to be clearly specified, but not in a way which
precludes innovative thinking.
Choosing a consultant
As noted earlier the first key choice is whether the consultant should be
external or internal. If the choice is to restrict the consultants to those from
outside the organisation, the key choice then is to select individuals or
organisations which have a track record in delivering timely and appropriate
evaluations using their chosen methodology. So, for example, if a statistical
study was required, it would be inappropriate to choose consultants whose
prime track record was in the use of qualitative methods.
Timetable
A timetable for the delivery of the evaluation has to be specified. Very
often this coincides with a wider appraisal of policies within the
commissioning department. To achieve this there has to be agreed milestones
in the form of interim and final reports to ensure that research is on track. To
achieve this it may be necessary to have a small steering committee.
In practice, however, the more sophisticated evaluations frequently tend
to overrun in terms of time. This is because of the difficulty of contacting
appropriate numbers of enterprises often because the lists given to the
consultants are incomplete. Hence, a crucial element of ensuring that
evaluations are on time is to ensure that the base data are of high quality
before the evaluators begin their work.
Quality assessment
Initially an assessment has to be made of the quality of the evaluation.
One clear purpose of this Framework is to enable an accurate assessment of
the quality of the evaluation to be made. Our overwhelming focus is upon the
technical quality of the evaluation defined as the extent to which polic
31
F
i
g
u
r
e

1
.
1
.
N
e
w

Z
e
a
l
a
n
d

T
r
a
d
e

a
n
d

E
n
t
e
r
p
r
i
s
e

(
N
Z
T
E
)

G
r
o
w
t
h

R
a
n
g
e

P
r
o
g
r
a
m
m
e

L
o
g
i
c

M
o
d
e
l
S
o
u
r
c
e
:
N
e
w

Z
e
a
l
a
n
d

T
r
a
d
e

a
n
d

E
n
t
e
r
p
r
i
s
e
.
i
t
e
r
a
t
i
v
e
i
t
e
r
a
t
i
v
e
G
O
V
E
R
N
M
E
N
T

B
U
S
I
N
E
S
S

A
S
S
I
S
T
A
N
C
E

P
R
O
G
R
A
M
M
E
S
P
O
L
I
C
Y

P
R
O
B
L
E
M
S
I n i t i a l a p p r a i s a l
A
l
l

N
Z
T
E

S
e
r
v
i
c
e
s
I
n
t
e
r
m
e
d
i
a
t
e
*
U
l
t
i
m
a
t
e
A
c
c
e
l
e
r
a
t
e
d
d
e
v
e
l
o
p
m
e
n
t

o
f

f
i
r
m
s
w
i
t
h

h
i
g
h

g
r
o
w
t
h
p
o
t
e
n
t
i
a
l
:

I
n
c
r
e
a
s
e
d

r
e
v
e
n
u
e

I
n
c
r
e
a
s
e
d

p
r
o
f
i
t
s

I
n
c
r
e
a
s
e
d

s
a
l
a
r
i
e
s
a
n
d

w
a
g
e
s

I
n
c
r
e
a
s
e
d
e
m
p
l
o
y
m
e
n
t

(
F
T
E
s
)

I
m
p
r
o
v
i
n
g

m
a
r
k
e
t
e
n
g
a
g
e
m
e
n
t

a
n
d

m
a
r
k
e
t
d
e
v
e
l
o
p
m
e
n
t

o
u
t
c
o
m
e
s

I
m
p
r
o
v
i
n
g

s
t
r
a
t
e
g
i
c
,
m
a
n
a
g
e
m
e
n
t

a
n
d

b
u
s
i
n
e
s
s
c
a
p
a
b
i
l
i
t
i
e
s
,

i
n
c
l
u
d
i
n
g

a
b
i
l
i
t
y
t
o

i
d
e
n
t
i
f
y

a
n
d

r
e
s
p
o
n
d

t
o
m
a
r
k
e
t

o
p
p
o
r
t
u
n
i
t
i
e
s

a
n
d

t
o
c
o
n
f
i
d
e
n
t
l
y

m
a
n
a
g
e

g
r
o
w
t
h

I
n
c
r
e
a
s
e
d

c
a
p
a
c
i
t
y

t
o
i
n
n
o
v
a
t
e

a
n
d

a
c
c
e
s
s

n
e
w
t
e
c
h
n
o
l
o
g
i
e
s
,

i
n
c
l
u
d
i
n
g

a
b
i
l
i
t
y
t
o

c
r
e
a
t
e
,

a
b
s
o
r
b

a
n
d
c
o
m
m
e
r
c
i
a
l
i
s
e

n
e
w

i
d
e
a
s

I
m
p
r
o
v
i
n
g

l
i
k
e
l
i
h
o
o
d

o
f
a
c
c
e
s
s
i
n
g

c
a
p
i
t
a
l

f
o
r

g
r
o
w
t
h
N
o
n
-
N
Z
T
E
g
o
v
e
r
n
m
e
n
t
b
u
s
i
n
e
s
s

a
s
s
i
s
t
a
n
c
e
E
x
t
e
r
n
a
l

f
a
c
t
o
r
s
b
e
y
o
n
d

t
h
e

c
o
n
t
r
o
l

o
f
N
Z
T
E
N
o
n
-
N
Z
T
E
g
o
v
e
r
n
m
e
n
t
b
u
s
i
n
e
s
s

a
s
s
i
s
t
a
n
c
e
p
r
o
g
r
a
m
m
e
s

A
l
l

c
l
i
e
n
t
s
C
l
i
e
n
t
m
a
n
a
g
e
m
e
n
t
G
r
o
w
t
h
s
e
r
v
i
c
e
s

f
u
n
d

H
i
g
h

g
r
o
w
t
h
p
o
t
e
n
t
i
a
l
M
a
r
k
e
t
d
e
v
e
l
o
p
m
e
n
t
s
e
r
v
i
c
e
s

M
e
d
i
u
m

o
r

h
i
g
h
g
r
o
w
t
h

p
o
t
e
n
t
i
a
l

D
e
v
e
l
o
p
m
e
n
t
p
l
a
n
s
*
N
o
t

a
l
l

o
f

t
h
e

i
n
t
e
r
m
e
d
i
a
t
e

o
u
t
c
o
m
e
s

w
i
l
l

b
e

a
p
p
l
i
c
a
b
l
e

t
o

a
l
l
c
l
i
e
n
t
s
.

T
h
e

r
e
d
e
v
a
n
c
e

o
f

t
h
e

o
u
t
c
o
m
e
s

w
i
l
l

d
e
p
e
n
d

o
n

c
l
i
e
n
t
n
e
e
d
s

a
n
d

s
e
r
v
i
c
e
s

r
e
c
e
i
v
e
d
.
D
e
n
o
t
e
s

s
c
o
p
e

o
f

t
h
i
s

e
v
a
l
u
a
t
i
o
n

p
r
o
j
e
c
t
A
s
s
e
s
s
m
e
n
t

(
I
t
e
r
a
t
i
v
e
p
r
o
c
e
s
s
.

D
e
p
t
h

o
f
a
s
s
e
s
s
m
e
n
t

d
e
p
e
n
d
s
o
n

n
e
e
d
s

a
n
d

g
r
o
w
t
h
p
o
t
e
n
t
i
a
l

o
f

f
i
r
m
)
F
i
n
a
n
c
i
a
l
N
e
e
d
s
C
a
p
a
b
i
l
i
t
y
M
a
r
k
e
t
S
e
g
m
e
n
t
a
t
i
o
n

b
y
g
r
o
w
t
h

p
o
t
e
n
t
i
a
l
(
f
l
u
i
d
)
H
i
g
h
C
l
i
e
n
t

m
a
n
a
g
e
r
a
s
s
i
g
n
e
d
M
e
d
i
u
m
C
l
i
e
n
t

m
a
n
a
g
e
r
a
s
s
i
g
n
e
d
E
m
e
r
g
i
n
g
H
o
t
l
i
n
e

c
l
i
e
n
t
m
a
n
a
g
e
r
a
s
s
i
g
n
e
d
L
o
w

F
i
r
m
s

l
a
c
k

m
a
n
a
g
e
m
e
n
t
e
x
p
e
r
t
i
s
e

r
e
q
u
i
r
e
d

t
o

g
r
o
w
t
h
e
i
r

b
u
s
i
n
e
s
s
s
i
g
n
i
f
i
c
a
n
t
l
y
.

D
a
i
l
y

m
a
n
a
g
e
m
e
n
t
p
r
e
s
s
u
r
e
s

a
n
d

r
e
s
o
u
r
c
e
c
o
n
s
t
r
a
i
n
t
s

c
a
n

c
r
o
w
d
-
o
u
t

a

f
o
c
u
s

o
n

l
o
n
g
-
t
e
r
m
s
t
r
a
t
e
g
i
c

i
s
s
u
e
s
.

O
w
n
e
r
s

r
e
l
u
c
t
a
n
c
e

t
o
c
e
d
e

c
o
n
t
r
o
l

o
f

t
h
e
i
r
b
u
s
i
n
e
s
s
e
s
,

o
r

l
a
c
k

o
f
c
o
n
f
i
d
e
n
c
e

t
o

t
a
k
e
n
e
c
e
s
s
a
r
y

r
i
s
k
s
,

t
o

g
r
o
w
t
h
e
i
r

b
u
s
i
n
e
s
s
e
s
.

F
i
r
m
s

m
a
y

n
o
t

r
e
a
l
i
s
e
b
e
n
e
f
i
t
s

o
f

e
x
t
e
r
n
a
l
b
u
s
i
n
e
s
s

a
d
v
i
c
e
.

S
o
m
e

o
f

t
h
e

b
e
n
e
f
i
t
s

o
f
e
x
t
e
r
n
a
l

b
u
s
i
n
e
s
s

a
d
v
i
c
e
a
r
e

e
x
t
e
r
n
a
l

t
o

t
h
e

f
i
r
m
.

C
o
s
t

a
n
d

t
i
m
e

r
e
q
u
i
r
e
d
t
o

e
s
t
a
b
l
i
s
h

a
n

o
f
f
s
h
o
r
e
n
e
t
w
o
r
k

m
a
y

b
e
p
r
o
h
i
b
i
t
i
v
e

f
o
r

s
m
a
l
l
e
r
f
i
r
m
s
,

o
r

m
a
y

n
o
t

b
e
j
u
s
t
i
f
i
e
d

f
o
r

s
h
o
r
t

t
e
r
m

o
r
l
i
m
i
t
e
d

e
n
g
a
g
e
m
e
n
t

w
i
t
h
p
a
r
t
i
c
u
l
a
r
.
32
makers can be confident that an identified programme impacts genuinely
reflect the contribution of the programme. However other considerations may
concern those managing evaluations and these are outlined in Appendix D.
Dissemination
A key decision is the extent to which the results of an evaluation enter
the public domain. Some evaluations normally those conducted by insiders
are specifically targeted towards the interests of, for example, departmental
committees. Other evaluations, however, are intended to contribute to a wider
public debate on SME or entrepreneurship policy. These are likely to be
undertaken by academics with a commitment to disseminating their work.
Therefore, a key decision is the nature of any dissemination which is to be
undertaken.
A second issue is the extent to which the data collected by the evaluators
is able to be accessed by other researchers in the area. Such researchers may
wish to verify the interpretations and conclusions that have been placed upon
the data, but much depends upon data protection legislation. The latter also
influences the opportunities for researchers to use the data for extending the
research over time by following up these businesses or individuals or deriving
samples from similar businesses or individuals in other industries or regions.
In principle therefore, subject to complying with data protection legislation, it
is desirable for the data derived from evaluations to be available to bona fide
researchers.
Ultimately, however, the purpose of evaluation is to stimulate and inform
public debate. It is desirable that evaluation reports are made available to the
media and other interested parties. Only in this way can evaluation truly lead
to policy change.
Some key principles for evaluation practice
Drawing on the discussion above, the following key principles for
evaluation can be set out.
1. Evaluation should lead to policy change
The prime purpose of undertaking evaluation is that it informs key
decisions. Such decisions may be to change policy. For example, it may lead to
a policy budget being increased, decreased or the policy itself being
abandoned. It may also lead to different objectives of the policy being
specified and, most likely, will lead to the policy being delivered in different
ways possibly to different target groups. Alternatively, the policy decision
may be that no change is required and that the programmes are on track.
33
2. Evaluation should be part of the policy debate
If it is to be at the heart of policy making, evaluation cannot be confined
to providing a historical review of previous policies. Instead, evaluation has to
lead to policy learning so that current policies may be amended in the light of
this knowledge and new policies developed from such learning.
3. Evaluators should be in at the start
Evaluation is, as noted above, vital in the formulation of new policies.
Those skilled in evaluation techniques can make major contributions to policy
development, most notably in helping policy makers to clearly formulate
policy objectives. Without the input of evaluators, policy objectives may not be
specified at all, or be expressed in such a way that they cannot fail to be
achieved, or be specified in such opaque language that it is impossible to
determine at a future point whether or not they were achieved.
The role of the evaluator is to ensure that policy makers specify clear and
tangible objectives as policy is being developed. A second merit in evaluation
being included at an early stage is that budgets for evaluation are specified
when programmes are being formulated. A third advantage of evaluators
being in at the start is the methods to be used to determine the success or
otherwise of the programme are clearly brought to the attention of the policy
makers. Finally, policy makers are made aware that a programme is to be
evaluated and the criteria of success that will be used to assess effectiveness
are agreed in advance by all parties.
4. Evaluation techniques should always use the most appropriate
methodology
Subject to the provisos outlined below, the most appropriate evaluation
techniques should be employed. The Six Steps to Heaven is a simple method
of assessing sophistication in this area, and for impact evaluation of medium
to large sized programmes we recommend that at least a Step 4 approach
should be used. By this we mean that, as a minimum, the beneficiaries of a
programme or policy should be compared with a control group of otherwise
similar firms or individuals, but who did not participate in the programme. By
comparing outcomes for the two groups a crude estimate can be made of
policy impact. Governments, with their substantial access to statistical data,
always have the opportunity to formulate such control groups, even if on some
occasions their own statistical agencies are unwilling to collaborate in such
work. The importance of evaluation however is such that the statistical
agencies should be mandated to develop this aspect of their evaluation
culture.
34
However, in order to understand more deeply the processes of how policy
works and to involve stakeholders in policy learning qualitative methods such
as peer reviews and case studies also have their place. These methods are not
suited for robust impact estimates but are necessary for many other aspects of
evaluation.
5. Evaluation should apply to all policies and programmes
It appears that some policies and programmes are evaluated many times,
and with some rigour, whereas others seem to either escape scrutiny
altogether, or are evaluated in a much less challenging manner than other
policies/programmes. This is extremely unfortunate since the overall purpose
of policy is to ensure that, at the margin, all policies are equally effective.
6. International comparisons should be made where necessary
Finally this Framework concludes that for some policy areas, evaluation
can only be undertaken on an international basis. For example, comparing the
impact of tax regimes is best undertaken across countries. As noted above,
international comparisons and peer reviews are appropriate for reviewing
policies at the regional or local level. In this respect, OECD can serve a valuable
function as a clearing house for information on policy effectiveness, for
producing harmonised data and in having access to experts in this field.
Notes
1. This term was coined in OECD (2004a). It refers to evaluation only being
considered after the objectives and targets have been set, and the programme has
been in operation for some time. When evaluation is at the end of the line it can
only serve as a historical auditing function. OECD (2004a) contrast this with the
more preferred Integral Evaluation. Here Evaluation is integrated into the policy
process. So, as the policy is being developed, consideration is given to how it will
be evaluated. This has four merits. The first is that it ensures that the objectives and
targets of the policy are clearly specified. Second, it ensures that the necessary data
collection can begin immediately and is built in to the programme. Thirdly, it
ensures that the evaluation budget is identified. Fourthly, it ensures that the
progress of the programme is monitored so that modifications and improvements
can be made in the light of evidence.
2. Evaluation of SME Policies and Programmes, background report for the 2nd OECD
Conference of Ministers Responsible for SMEs, Istanbul, June 2004.
3. Source: European Commission State Aid Scoreboard, Spring 2004 update.
4. The awards specified by Cooper were 1.107 in 1997, 1.067 in 1998 and 1.097 in 1999
all in billions of USDs.
5. For formative and prospective evaluations, the opportunities for undertaking
quantitative/statistical approaches are generally much less, unless they draw
upon previous summative evaluations.
35
6. Examples of external evaluators are generally those not employed as public
servants by the government. These will include private sector consultants and
research contractors, such as academics.
7. www.uwex.edu/ces/lmcourse/#.
8. For example, when the objectives of a programme are unclear the evaluator might
ask the question What outcome from the programme would you identify as
failure? Experience of asking this question is that it is initially met with hostile
silence. Frequently it is then followed by specifying outcomes that are [close to]
impossible. These might include Nobody is interested in the policy. After
consideration, however, more realistic objections normally emerge.
9. Additionality refers to the proportion of the supported activity that would not
have gone ahead without the support, deadweight activity that would have gone
ahead anyway, displacement refers to reductions of activity elsewhere (e.g. in
other firms) as a result of increased competition from the supported activity (e.g.
the firms receiving assistance).
ISBN 978-92-64-04008-3
OECD 2007
37
Section 2
Evaluation of Individual National Programmes
2. EVALUATION OF INDIVIDUAL NATIONAL PROGRAMMES
38
Introduction
This section takes examples of SME and entrepreneurship policies from a
number of countries and seeks to report, using a common format, the results
of evaluations. The vast majority of these evaluations are in the public
domain.
For ease of access the published evaluations are categorised by topic.
These are:
Financial assistance;
Enterprise culture;
Advice and assistance;
Technology; and
Management training.
Table 2.1 shows some specific policies that are included under these
headings.
The central purpose of this chapter is to provide a number of examples of
evaluation of public programmes to assist SMEs in each of the five key policy
areas. Some of these evaluations can be considered to be sophisticated
according to the Six Steps framework but others are less so. The inclusion of
Table 2.1. SME and entrepreneurship policy areas covered
Financial assistance Enterprise culture Advice and assistance Technology Management training
Loan Guarantee
Schemes.
Programmes to
encourage young
disadvantaged
individuals to start
businesses.
Provision of
Marketing Advice.
Subsidies to New
Technology Based
firms.
Subsidies to
stimulate the take up
of management
training in SMEs.
Subsidising the
creation of
businesses and
growth of SMEs
perhaps by
disadvantaged
groups.
Programmes to
encourage graduates
to start businesses.
Provision of general
business advice.
Creation of Science
Parks.
Tax relief to business
angels.
Enhancing
Investment readiness
of SME owners.
Encouraging SMEs to
export.
39
the less sophisticated approaches is justified partly on the grounds that there
is an absence of top quality evaluations from which to draw. It is also to
emphasise that there is an opportunity for improvement in evaluation
methods. Thirdly it shows that good practice is found in evaluations in almost
all SME policy areas but, in our view, is not yet the norm.
Evaluations of financial assistance
Table 2.1 shows there are primarily three types of financial assistance
programmes that have been evaluated. These are, firstly, loan guarantee
programmes, secondly subsidies to disadvantaged groups to encourage
them to start or grow a business, and thirdly, subsidies to develop the market
for business angels.
Loan guarantee programmes
Evaluations of loan guarantee programmes from Japan and Canada are
shown in Tables 2.2 and 2.3.
Subsidising the creation of businesses and growth of SMEs perhaps
by disadvantaged groups
Two examples of evaluations of these policies are presented in this
section although the programme in Table 4.8 could also be taken as another
example. The first is of Law 44 in southern Italy where the disadvantaged
group is young people (Table 2.4). The second is grant assistance to SMEs in
Northern Ireland and the Republic of Ireland (Table 2.5).
Table 2.2. Loan guarantee scheme, Japan
Available online http://isb.sagepub.com/cgi/content/abstract/23/1/48
Country Japan
Time period of study 1980-2002
Title of report Promoting Enterprise Development or Subsidising Tradition: The Japanese Credit
Supplementation Scheme.
Date of report 2005
Author/details M. Nitani and A. Riding, International Small Business Journal, Vol. 23[1], pp. 48-71.
Objective of policy To contribute to the smooth flow of funds by guaranteeing loans that are advanced to
SMEs by banks or other financial institutions.
Key findings The Japanese Credit Supplementation Scheme is much more targeted to averting the
failures of weak firms and much less focussed on new and growing firms.
Sophistication of evaluation Step 1:
Reviews publicly available data.
No survey undertaken.
Comments Helpful review placing Japan in context of other OECD countries operating an loan
guarantee scheme.
40
Table 2.3. Loan guarantee scheme, Canada
Available online www.ingentaconnect.com/content/els/08839026/2001/00000016/00000006/art00050
Country Canada
Title of report Loan Guarantee: Cost of Default and Benefit to Small Firms.
Date of report 2001
Author/details A.L. Riding and G. Haines, Journal of Business Venturing, Vol. 16[6], pp. 595-612.
Objective of policy To increase the availability of loans for the purpose of the establishment, expansion, modernisation and
improvement of small business enterprise.
To encourage lenders to provide debt financing to SMEs, that the lenders would otherwise consider too small
or risky.
Key findings The SBLA is an extraordinary effective means of stimulating job creation and assisting small firms to survive
and grow.
The Canadian default rates are well below those for other countries.
Sophistication of evaluation 232 000 loans were examined.
Comparisons made with other OECD countries.
Comments Very helpful international review of LGS findings.
Table 2.4. Assistance to new enterprises started by young people, Italy
Available online www.informaworld.com/smpp/content~content=a713693784
Country Italy
Title of report The Life Duration of Small Firms born within a Start-up Programme: Evidence from Italy.
Date of report 2001
Author/details A. Del Monte and D. Scalera, Regional Studies, 35.1, pp. 11-21.
Objective of policy
Law 44 supports the creation of new firms by young people [< 35 years old] in Southern Italy in three ways:
Subsidy of up to 60% of initial investment.
Loans at 30% of market rates.
For first three years a subsidy on variable costs.
Key findings
Subsidies significantly affected the creation and survival of assisted firms established between 1988
and 1993.
About half the firms would not have started without the subsidy.
Sophistication of evaluation No control groups used but careful use of hazard modelling.
Comments
Table 2.5. Grant assistance and small firm performance, Ireland
Available online http://ideas.repec.org/p/eri/niercp/38.html
Country Ireland
Title of report Grant Assistance and Small Firm Development in Northern Ireland and the Republic of Ireland.
Date of report 2001
Author/details Stephen Roper and N. Hewitt-Dundas, Scottish Journal of Political Economy, Vol. 48, No. 1, pp. 99-117.
Objective of policy To enhance the performance of SMEs in Northern Ireland and the Republic of Ireland by the provision of
different forms of grant support.
Key findings Grant support for SMEs in both Northern Ireland and the Republic of Ireland has a positive effect on
employment growth.
No effect is observed on either sales or profitability.
Sophistication of evaluation Step 6: Sophisticated selection and assistance effects taken into account.
Comments Exemplar Study.
41
Both are technically very good studies with the latter being one of the few
Step 6 studies available for any of the policy areas.
Tax reliefs to business angels
Two studies of the provision of tax reliefs to support the business angel
market are presented. Table 2.6 shows a UK evaluation of two different
methods for providing tax relief to wealthy individual to encourage
investment in small companies. Table 2.7 shows a second review of one of the
schemes Enterprise Investment Scheme (EIS).
Table 2.6. Public subsidies to business angels: EIS and VCT, UK
Available online www.hmrc.gov.uk/research/report.pdf
Country United Kingdom
Time period of study 1994/95 onwards
Title of report Research into the Enterprise Investment Scheme [EIS] and Venture Capital Trusts [VCT].
Date of report 2003
Author/details Nick Boyns et al, PACEC for Inland Revenue.
Objective of policy Provide incentives to individuals to encourage the supply of equity finance to smaller unquoted companies.
This is achieved by improving the post-tax return to investors.
Thereby encouraging them to invest more in the company.
Leading to better access to funds by companies.
Leading to the UK having a more competitive small firms sector.
Key findings There is a lack of external benchmarks for effectiveness assessment.
For EIS, for every GBP 1m in tax forgone, there is a rise in sales of GBP 3.3m and employment rise of 65 jobs.
For VCT for every EIS tax forgone, sales increase by GBP 0.6m and employment by 9 jobs.
Surveys of EIS investors, companies and controls.
Surveys of VCT investors, companies and controls.
Plus survey of advisors.
Comments Sound evaluation.
Table 2.7. Public subsidies to business angels: EIS, UK
Available online No
Time period of study 1994/95 onwards
Title of report The Enterprise Investment Scheme: Why Investors and Companies do not use the Scheme.
Date of report December 1999
Author/details Confederation of British Industry, London.
Objective of policy Provide a targeted incentive for new equity investment in growth trading companies to overcome the
problems faced by such companies in raising small amounts of equity finance.
To encourage business angels by enabling them to take an active part in the management of the company
as paid directors without losing entitlement to relief.
Key findings The scheme is too complicated and has too many rules.
Sophistication of evaluation No Survey: Report based on hypothetical examples and personal experience.
Comments Not really an evaluation.
42
The value of showing two evaluations of the same scheme is to illustrate
two very different approaches to evaluating the same programme.
Enterprise culture
Table 2.1 showed that three types of programme evaluations are to be
reviewed in this section. The first deals with those programmes that seek to
encourage enterprise amongst disadvantaged groups. The second are those that
seek to encourage enterprise amongst university graduates. Finally an evaluation
of a programme to enhance the investment readiness of enterprises is included.
Disadvantaged groups
Table 2.8 is an example of a Step 6 evaluation of a programme seeking to
provide both access to finance and business advice to disadvantaged young
people to enable them to start their own business.
Programmes to encourage university graduates to start businesses
Table 2.9 is an evaluation of a programme that seeks to influence the
career choices of graduates by making them more aware of the benefits of
working in, and perhaps starting, a small firm after graduating from college.
Enhancing investment readiness
Table 2.10 shows an evaluation of a programme which seeks to raise the
awareness of SME owners of being investment ready.
Table 2.8. Assisting young disadvantaged people to start up businesses, UK
Available online http://intl-ner.sagepub.com/cgi/content/abstract/186/1/59
Title of report An Evaluation of Business Start-Up Support for Young People.
Date of report 2003
Author/details Nigel Meager, Peter Bates and Marc Cowling, National Institute Economic Review, No. 186, pp. 70-83.
Objective of policy Not clearly specified in the paper but the objectives of the scheme (the Princes Youth Business Trust
Initiative) are:
To stimulate self-employment amongst young people aged 18-30.
To focus upon those who are disadvantaged.
Key findings There is no evidence that participation self-employment has any significant impact on subsequent
employment or earnings.
Use of selection bias and control groups.
Comments Contentious findings given the high profile of the Princes Trust.
43
Advice and assistance
Evaluations of three forms of advice and assistance provided to SMEs are
reviewed. These are marketing advice, general business advice and export
promotion assistance. The evaluation in Table 2.11 shows a Step 6 evaluation
of the impact of the provision of marketing advice and assistance to SMEs
Tables 2.12 to 2.16 show, in a variety of very different countries,
evaluations of the impact of the provision of general business advice upon
SMEs. These countries are: Belgium, Bangladesh and the UK.
Table 2.9. Graduates into business, UK
Available online www.ingentaconnect.com/content/routledg/tepn/2001/00000013/00000002/art00004
Title of report Outcomes reported by students who participated in the 1994 Shell Technology Enterprise
Programme [STEP].
Date of report 2001
Author/details P. Westhead et al., Entrepreneurship and Regional Development, Vol. 13, No. 2, pp. 163-185.
Objective of policy To provide undergraduates with experience of working in an SME over 8 weeks in the summer.
To raise awareness of the possibility of working in, or starting a SME after graduation.
To give SMEs experience of having graduates in their firm.
Key findings Students and SMEs found the projects very helpful.
Upon graduation students were less likely than the control group to get a job in an SME, but more
likely to enter quickly into employment.
This is because their STEP project made them more attractive to larger companies.
STEP students were matched with other similar non-STEP students.
Comments
Table 2.10. Investment readiness, New Zealand
Available online www.med.govt.nz/templates/MultipageDocumentTOC____1246.aspx
Country New Zealand
Title of report BIZ Investment Ready Scheme.
Date of report 2003
Author/details Nick Davis, Ministry of Economic Development.
Objective of policy Provide entrepreneurs with improved understanding of raising external equity.
Provide entrepreneurs with awareness of investment readiness.
To achieve this by providing workshop based training.
Key findings 99% of respondents from the workshops were satisfied with content and relevance in 2002/3.
Slightly more than 1 in 10 participants actually raised external capital.
Does track workshop participants.
Comments
44
Table 2.11. Impact of marketing advice, UK
Available online http://oep.oxfordjournals.org/cgi/content/abstract/54/2/334
Title of report Evaluating the effects of soft business support upon small firm performance.
Date of report 2002
Author/details C.M. Wren and D.J. Storey, Oxford Economic Papers, Vol. 54[2], pp. 334-365.
Objective of policy To enhance the performance of SMEs by providing subsidised consultancy advice by marketing consultants.
Key findings Overall the policy does not have impact.
But in the target group of 10-80 employees participation raises longer run survival rates by 4% and sales by
up to 10%.
Applicants are compared with non-applicants, then Stage 2 applicants are compared with Stage 3.
Comments
Table 2.12. Impact of business advice, Belgium
Available online www.informaworld.com/smpp/content?content=10.1080/0898562042000338598
Country Belgium
Title of report An Evaluation of Public Support Measures for Private External Consultancies to SMEs in the Walloon Region of
Belgium.
Date of report 2005
Author/details Johan Lambrecht and Fabrice Pirnay, Entrepreneurship and Regional Development, Vol. 17[2], pp. 89-108.
Objective of policy To provide three person days of a free diagnostic advice.
Subsequently different forms of advice are provided with different levels of subsidy.
Key findings Consultants recorded favourable evaluations from SMEs.
However they have no significant influence on net job creation, turnover or financial indicators.
Sophistication of evaluation Step 4 approach but careful analysis.
Distinguishes usefully between supply and demand effects.
Identified adverse selection issue.
Comments An important recent study
Table 2.13. Impact of advisory support, Bangladesh
Available online http://findarticles.com/p/articles/mi_hb177/is_199704/ai_n5784454
Country Bangladesh
Time period of study 1993
Title of report The Importance of Support Services to Small Enterprises in Bangladesh.
Date of report 1997
Author/details J.H. Surder, D. Ghosh and P. Rosa
Journal of Small Business Management, April, Vol. 37, No. 2, pp. 26-36.
Objective of policy To enhance the performance of SMEs by the provision of business support services.
Key findings Significantly better performance is shown by SMEs receiving support services.
Sophistication of evaluation Step 5: Authors recognised that their results may reflect non-random selection process by support agencies.
Comments Despite being an early study it is well conducted and reaches appropriate conclusions.
45
Table 2.17 is an important methodological table which takes the use of
business advice as its example. It shows that, although the vast majority of firms
that received business advice were happy with that advice, there was little
evidence that advice provision lead to enhanced performance amongst recipient
firms.
Export promotion programmes are a favoured way of assisting SMEs.
Evaluations of the impact of such programmes seem rare, but the Swedish
evaluation presented in Table 2.18 is helpful.
Table 2.14. Bank customers receiving business advice, UK
Available online No
Title of report Profiting from Support.
Date of report December 2001
Author/details Barclays Bank SME Team.
Objective of policy To provide advice and assistance to individuals before or at the time of starting a business.
The purpose is to enhance their chances, post start-up, of survival and growth.
Key findings Those seeking advice, compared with the match control group have:
Higher survival.
Fewer closures in financial distress.
Faster and more sustained growth amongst surviving firms.
Signs of better overall financial management.
Comparison between clients of the National Federation of Enterprise Agencies.
Matched control sample of Barclays Bank clients.
Comments Real risk of selection bias as those approaching Enterprise Agencies may differ from the norm.
Table 2.15. Assistance and advice for mature SMEs, UK
Available online www.informaworld.com/smpp/content~content=a739469376~db=all
Title of report The use of external assistance by mature SMEs in the UK: some policy implications.
Date of report 1993
Author/details David Smallbone, David North and Roger Leigh, Entrepreneurship and Regional Development,
Vol. 5, pp. 279-295.
Objective of policy To encourage the use of consultants to assist SMEs.
Key findings 45% of firms made no use of external assistance over 10 years.
Mature firms have different requirements from newly established firms.
Banks and accountants have an important role to play as a point of entry for firms into the support
system.
Not concerned with individual elements of policy.
Comments
46
Table 2.16. Use and impact of business advice, UK
www.ingentaconnect.com/content/routledg/raef/2000/00000032/00000013/art00005
Title of report The use and impact of business advice by SMEs in Britain: an empirical assessment using logit
and ordered logit models.
Date of report 2000
Author/details P.J.S. Robson and R.J. Bennett, Applied Economics, Vol. 32, pp. 1675-1688.
Objective of policy To encourage the use of external business advice.
To link business advice to SME performance.
Key findings The factors influencing the likelihood of firms seeking external advice are:
Firm size [+].
Rate of growth [+].
Innovation [+].
The impact of advice depends on the field of that advice but for business strategy advice the
significant effects were:
Firm age [-].
Firm size [+].
Export [-].
Firms are asked to assess the impact of advice on a scale of 1-5.
No verification on selection issues is addressed, but a multi-variate framework is utilised.
Comments
Table 2.17. Evaluating entrepreneurial assistance programs, US
Available online www.questia.com/googleScholar.qst?docId=5002434866
Country United States
Time period of study Not specified
Title of report Some Problems in Using Subjective Measures of Effectiveness to Evaluate Entrepreneurial Assistance
Programs.
Date of report 2001
Author/details Ed McMullan, J.J. Chrisman and K. Vesper, Entrepreneurship Theory and Practice, Fall, pp. 37-54.
Objective of policy The paper reviews these programs:
Western Economic Network [WEN] which is 13 evening class sessions each of 3 hours.
SBDC activity, consisting mainly of counselling and consulting, rather than coursework.
They argue that the economic development impact sought from entrepreneurial assistance
programs is the creation of tacit and explicit knowledge that will lead to competitive advantage.
Key findings The subjective views of course clients [Step 2] are uncorrelated with the attribution and objective
measures [Step 3].
The authors question the merits of the Step 2 approach.
Sophistication of evaluation Step 2
Clients opinions are compared with Step 3.
Client estimates of impact suggest the Step 2 approach has limited value.
Comments
47
Technology
Evaluations of two types of programmes for assisting new technology
based firms are reviewed. The first are specific programmes that seek to
enhance individual firms such as the SBIR programme in the United States
and the SMART Programme in the United Kingdom. The second are more
general programmes such as those promoting the development of science
parks.
Subsidies to new technology based firms
Tables 2.19 and 2.20 are two evaluations of the United States SBIR
programme. The value of these evaluations is that very similar programmes
have been implemented elsewhere so it should be possible to undertake
similar evaluations. Table 2.21 outlines an evaluation of the United Kingdom
SMART programme.
Science parks and university-SME links
Tables 2.22 and 2.23 show two evaluations of the impact on SME
performance of a science park location. The Sweden study is more
comprehensive partly because there are more parks and they have been
established longer than in Greece. Table 2.24 shows a New Zealand evaluation
of a programme to encourage regional Polytechnics to establish stronger links
with the SMEs in their locality.
Table 2.18. Encouraging partnerships amongst SMEs, Sweden
Available online www.fsf.se/eng/publications.htm
Country Sweden
Title of report Interprise DECIDE V Expectations, Experiences and Results.
Date of report 1999
Author/details A. Lundstrom, Swedish Foundation for Small Business Research.
Objective of policy To restructure the Defence Industry.
To link Swedish firms with those from at least three other countries.
To diversify the customer base of Swedish firms.
Key findings 8/9 months after DECIDE V, SMEs are satisfied with the outcomes.
A quarter has a formal contract that stemmed from DECIDE.
Given the costs incurred by the SMEs it is less clear whether DECIDE yields value for money.
Sophistication of evaluation Important follow up study.
Valuable comparison between two meetings.
No control group.
Comments
48
Management training
This section examines the evaluations that have been undertaken of
programmes that seek to enhance managerial skills in SMEs by the provision
of training for business owners. Table 2.25 and Table 2.26 show two
evaluations of the impact of management training programmes in SMEs. The
evaluation methods used vary considerably with the Small Firm Training Loan
Scheme being assessed primarily through interviews with recipients.
Conclusion
This section identified five areas of SME policy that are to be found in most
developed countries. It then takes examples of policy evaluations in each of
Table 2.19. Technology assistance to small firms, US
Available online http://ideas.repec.org/p/nbr/nberwo/5753.html
Time period of study Not specified.
Title of report The Government as Venture Capitalist: The long run impact of the SBIR Program.
Date of report 1999
Author/details Josh Lerner, Journal of Business, Vol. 72, No. 3, pp. 285-318.
Objective of policy Objective of SBIR not specified in article.
Key findings Awardees grew significantly faster than a matched sample over a decade.
Awardees were more likely to attract venture financing.
Multiple awards did not increase performance.
However there is likely to be both selection bias and application bias.
Comments Reservations about the selection bias must raise a question mark over the findings.
Table 2.20. The SBIR program, US
Available online www.springerlink.com/content/nnap7wyb7e1r/?p=ba199ec0f06c48e0af015d1011f3c3ff&pi=38
Title of report Purpose and Performance of the SBIR Program.
Date of report 2003
Author/details R.S. Cooper, Small Business Economics, Vol. 20, No. 2, pp. 137-151.
Objective of policy The purpose of the program is to strengthen the role of the small innovative firms in federally
funded research and development as a base for technological innovation.
To meet agency needs and contribute to the growth and strength of the nations economy.
Key findings SBIR addresses each of the dimensions of the financial gap. It provides:
Recognition.
Small amounts of capital.
Assistance to a wide variety of technologies.
Invited opinions of recipients and reports the results of other studies using this methodology,
together with that of Lerner [1999] reported in Table 2.19.
Comments
49
the five areas and presents a summary of both the methods used and the
key findings. Its conclusions are the following: sophisticated quantitative
evaluations are the exception rather than the norm. In many countries it
has been difficult to obtain any [English language] evaluations of policy
undertaken by government that are in the public domain. Evaluations
Table 2.21. The UK SMART programme
Available online www.berr.gov.uk/files/file22002.pdf
Title of report Evaluation of SMART.
Date of report 2001
Author/details Public and Corporate Economic Consultants.
Objective of policy Increase and improve technology use and adaptation and research and development by individuals
and SMEs.
Contribute to a climate which encourages investment in innovative technology by individuals, firms
and financial institutions and which stimulates a market in technological advice.
Key findings SMART is generally effective in relation to its objectives and provides value for money.
There is scope to increase benefits by increasing market penetration and improving the integration
of the support delivered by the scheme with the wider business support infrastructure.
Compared the performance of SMART award winners and unsuccessful applicants after accounting
for selection effects, i.e. factors that distinguish successful from unsuccessful programme
applicants.
Included in-depth qualitative interviews with scheme administrators and grant recipients on their
view of the impact of the scheme and possible improvements.
Comments Combines quantitative and qualitative evidence. Qualitative stakeholder views on the usefulness
of different aspects of the scheme and how its operation might be improved was critical to the
recommendations on adjustments to the scheme. However, the recommendation to retain the
scheme was based on value for money evidence from the quantitative evaluation.
Table 2.22. Impact of science parks, Greece
Available online www.ingentaconnect.com/content/els/01664972/2002/00000022/00000002/art00087
Country Greece
Title of report Science Parks, a high-tech fantasy? An analysis of the Science Parks of Greece.
Date of report 2002
Author/details Bakouras, Y.L., Mardas, D.C. and Varsakelis, N.V., Technovation, Vol. 22 [2], pp. 123-128.
Objective of policy To promote the development of high technology firms.
To promote this through establishing formal and informal links between NTBFs and Universities.
Key findings Greek Science Parks have more small firms than is typical of EU Science Parks.
The formal links between Science Park Firms and Universities exists only in one Science Park in
Greece.
Informal links do exist.
Details are collected from 17 firms on three Greek Science Parks.
No controls are used.
Comments
50
undertaken by outsiders academics or consultants are more frequently
accessible, but they are still far from commonplace. Even when they are
available the level of sophistication is mixed.
The central finding is that, taken as a whole, sophisticated evaluations of
SME support are on balance less likely to provide evidence of policy impact
than the evaluations using less sophisticated approaches. The implication
of this important finding is that policy makers, in principle, can influence
the outcome of an evaluation by their methodological choices.
Table 2.23. Impact of science parks, Sweden
Available online www.springerlink.com/content/100338/
Country Sweden
Title of report Science Park Location and New Technology Based Firms in Sweden implications for Strategy
and Performance.
Date of report 2003
Author/details P. Lindelof and H. Loftsen, Small Business Economics, Vol. 20 [3], pp. 245-258.
Objective of policy To promote the economic development of high technology firms.
To promote the development of research.
To promote firm/university linkage.
Key findings No statistically significant differences between Science Park Firms and off-Park Firms.
The economic performance of the on-Park and off-Park firms do not differ.
Collaboration is less with on-Park than with off-Park firms.
10 established Science Parks. Matches 134 on Park Firms with 139 not on Park.
Comments Sophisticated study.
Table 2.24. University/SME links, New Zealand
Available online www.med.govt.nz/templates/MultipageDocumentTOC____13694.aspx
Country New Zealand
Title of report Review of Polytechnic Regional Development Fund.
Date of report 2004
Author/details Ministry of Economic Development.
Objective of policy Enable and encourage institutes of technology and polytechnics to collaborate with regional
industry.
To build capability in ITPs to be responsive to regional training needs aligned with regional
comparative advantage.
Improve regional economic development.
Key findings NZD 3.8 million was distributed to ITPs.
ITPs and industry have significantly increased collaboration following the project.
Participants expect this to continue beyond the funding stream.
ITPs in receipt of funding and those not receiving funding were contacted.
Industry Partners only 20% of which were SMEs were also contacted.
Comments
51
A second important finding is that larger, longer duration programmes are
more likely to be assessed using sophisticated methods, than is the case of
short duration, small scale programmes. However, this also presents
problems since it means that a different currency is being used to evaluate
programmes of different scales, perhaps to the disadvantage of larger
programmes.
Table 2.25. Impact of management training on SMEs, UK
Available online www.envplan.com/abstract.cgi?id=c110331
Title of report The impact of government assisted management training and development on SMEs in Britain.
Date of report 1993
Author/details J.N. Marshall, N. Alderman, C. Wong and A. Thwaites, Environment and Planning C, Vol. 11,
pp. 331-348.
Objective of policy The policy aims at business growth through management training.
The policy provides financial assistance to firms with 25-500 employees to employ consultants to
help them develop their management.
Firms are provided with up to half the costs, maximum GBP 15 000 of the consultants.
The employment of consultants would result in improvements in firm organisation and
improvement.
Key findings There is a major boost to management training, combined with a smaller but continuing increase
in the number of firms carrying out management training and development.
Less evidence that human resource development thereby improves business performance.
This may be because the evaluation was undertaken too early.
49 Option 3 firms were matched with 100 firms similar in terms of size, industry and location.
Comments
Table 2.26. Small firms training loans, UK
Available www.dfes.gov.uk/research/programmeofresearch/
projectinformation.cfm?projectid=12837&resultspage=1
Title of report Evaluation of Small Firms Training Loans.
Date of report 1999
Author/details Kevin Maton, UK Research Partnership, DfEE Research Report, No. 97, RR 1999.
Objective of policy Small Firm Training Loans (SFTL) are loans offered to small firms (< 50 employees) to invest in
training.
They are financed through banks but include repayment holidays and preferential rates of interest.
Key findings In terms of client satisfaction the scheme scores very highly, with 81% of businesses saying it
works well, as a way of funding training for small firms.
The study interviewed 92 SFTL clients by telephone and more in-depth interviews with others.
It also interviewed support agencies.
Comments No control group included.
52
We therefore emphasise again the points made above that the choice of
evaluation approach needs to be carefully assessed.
This inevitably raises the question of whether there is merit in undertaking
what is referred to in the Six Steps approach as monitoring. Our view is
that policy makers need to be aware that there is a risk that low grade
evaluations monitoring can lead to misleading pictures of programme
effectiveness.
ISBN 978-92-64-04008-3
OECD 2007
53
Section 3
Evaluation of Regional and Local Programmes
3. EVALUATION OF REGIONAL AND LOCAL PROGRAMMES
54
Introduction
Regional and local governments and development agencies often have
significant responsibilities for SME and entrepreneurship policies and
programmes in their own right, alongside national governments and agencies.
Just like national organisations, they require evaluations for their
accountability and policy learning. However, the evaluation techniques
highlighted in this Framework are essentially generic to both the national and
regional/local scales and the main purpose of this section is therefore to
extend the examples of evaluation practices to the sub-national level to offer
local policy makers some further examples that may better match their own
policy environments.
There are certainly a few technical evaluation issues that are distinct at
the local and regional scale and should be taken into account in sophisticated
evaluations. One of these issues concerns the treatment of displacement and
leakages. Thus whereas in national evaluations the displacement of economic
activity from one locality to another is often assumed to have no national
benefit, one of the purposes of regional and local interventions may be
precisely to redistribute activity towards disadvantaged places in order to
promote spatial equity or to release supply-side benefits that promote
national growth. Therefore a different interpretation needs to be given to
displacement at different geographical scales. In the case of leakages, there
may be benefits from the implementation of a programme in one region or
locality that are felt in another. Whereas evaluations of national programmes
would not pick up these regional and local leakages explicitly, being
considered simply as part of the national benefits, they may be an important
focus of attention at the regional and local level.
Another distinct issue that arises concerns area-selection bias. We
discussed above how at national level evaluations should seek to account for
participant selection bias. At the regional and local level, there is potentially
an additional selection bias because apparently similar places may vary in the
degree to which they favour entrepreneurship, complicating comparison
between areas with different scales or types of support. For example, two
disadvantaged areas may differ with the one offered support representing
either a more or less favourable institutional environment for entrepreneurship,
depending on whether government seeks to reward success or remediate
failure.
55
However, the main distinction between national and sub-national
evaluation is that the latter often involves smaller scale programmes and
evaluation budgets and potentially weaker evaluation support structures.
Thus, following the principle of proportionality (under which evaluation
budgets should not exceed a certain proportion of programme expenditure),
sub-national evaluations will often need to be less sophisticated on the Six
Steps to Heaven metric and may put more emphasis on qualitative
techniques, although there are some excellent local and regional examples of
high quality quantitative evaluations at Steps 5-6.
Overall, then, the differences between national and subnational evaluation
are not major. Given the close correspondence between national and regional/
local evaluation requirements, this section therefore reports examples of regional
and local evaluations using the same format as Section 2. The majority of these
evaluations can be consulted online for further information on the evaluation
approaches used.
The evaluations are categorised by the following topics:
Advice, consultancy and assistance.
Clusters and local innovation systems.
Support to areas of geographical disadvantage.
Table 3.1 shows the policy areas discussed under these headings.
Advice, consultancy and financial assistance
Examples of evaluations of regional programmes to provide consultancy,
export assistance and financial support to SMEs are provided below.
Advice and consultancy
The evaluation in Table 3.2 shows a Step 4 evaluation of the impact of the
same subsidised consultancy to SMEs in Wallonia, Belgium that was reviewed
in Table 2.12. Our purpose in reproducing it here is to show that a high quality
evaluation of the same programme can both be of interest to national and
local/regional policy-makers, but that each will focus upon different aspects.
Table 3.1. Regional/local policy areas covered
Advice, consultancy and financial
assistance
Clusters and local innovation
systems
Support to areas of geographical
disadvantage
Advice and consultancy. Regional innovation and technology
transfer strategies.
Distressed urban areas.
Export assistance. Business networking. Lagging regions.
Business start up and investment
finance.
Rural policy.
56
Tables 3.3 and 3.4 show typical regional-level evaluations of consultancy
support to SMEs.
Export assistance
Table 3.5 presents an evaluation of export information and advice centres
in Quebec, Canada. Information on impacts is claimed but the report really
presents monitoring rather than evaluation evidence. Table 3.6 shows an
evaluation of export support to enterprises in a Swedish region by brokering
partnerships with entrepreneurs in other countries.
Table 3.2. Subsidised consulting, Belgium, Wallonia
Available online www.informaworld.com/smpp/content?content=10.1080/0898562042000338598
Country and region Belgium, Wallonia
Title of report An Evaluation of Public Support Measures for Private External Consultancies to SMEs in the
Walloon Region of Belgium.
Date of report 2005
Author/details J. Lambrecht and F. Pirnay, Entrepreneurship and Regional Development, Vol. 17[2], 89-108.
Objective of policy To provide consultancy that would not be provided on a commercial basis.
Key findings Strong SME satisfaction with subsidised consultancy but no significant impact on net job
creation, turnover or financial indicators.
Sample of 200 users and 100 randomly selected non-users.
Comments Argues that the matching required for steps 5 and 6 is very difficult. These steps could be
omitted for proportionality reasons.
Table 3.3. Business advisory services, UK, South West England
Available online www.southwestrda.org.uk/downloads/document.asp?lang=&documentid=1775
Country and region South West England, UK
Title of report Interim Evaluation of the South West Manufacturing Advisory Service.
Date of report 2007
Author/details DTZ Consulting & Research.
Objective of policy To improve manufacturing business productivity by providing advice, diagnostic support, shared
learning events and implementation-focused consultancy.
Key findings A high proportion of businesses rated support as excellent or good. Deadweight was high, estimated
at 50-70%. Displacement was low because of the propensity of beneficiaries to export. The ratio of
additional local Gross Value Added to local public sector cost was estimated at between 1.4 and 3.7.
Survey of a sample of 189 beneficiary firms exploring services used, satisfaction with the
programme and recipient views on its impact.
Comments Usefully assesses the programme on the criteria of economy, effectiveness and efficiency. A range of
effectiveness measures are used including measures of the delivery process quality of service,
adaptation to business needs, networking with other providers and the business impact efficiency,
profitability, adoption of innovative processes and enhanced management capabilities.
57
Financial support
Table 3.7 presents an evaluation of business start-up finance in North
East England, UK, whilst Table 3.8 presents a small firm grant programme
from a regional development agency in the west of Ireland.
Clusters and local innovation systems
Support to SMEs is often provided indirectly by seeking to improve the
regional and local business environment in which SMEs operate. In particular,
cluster and local innovation systems policies have the aim of improving the
Table 3.4. Enhancing the capability of the SME owner through use of consultants,
UK, Scotland
Available www.envplan.com/abstract.cgi?id=c9868
Country Scotland
Title of report Developing Expertise in SMEs: An Evaluation of Consultancy Support.
Date of report 2000
Author/details I. Turok and M. Raco, Environment and Planning C, Vol. 18, pp. 409-427.
Objective of policy The Expert Help Scheme operates in Scotland.
It seeks to make SMEs more competitive by improving their general competence and technical
ability through the provision of advice and subsidies to enable them to access external advice.
Key findings The companies were generally very satisfied with their consultants.
Some additionality in 82% of cases.
Full additionality in 23% of cases.
Less than one job per company additionality.
45 companies were interviewed out of a total of 1936 participants.
25 consultants were also interviewed.
No control group.
Comments Difficult to assess impact using this methodology.
Table 3.5. Export information and advice, Canada, Quebec
Available online www.dec-ced.gc.ca/Complements/Publications/Evaluation-EN/Rapport_final_ORPEX_Anglais.htm
Country and region Canada, Quebec
Title of report Evaluation of ORPEX Funding Programmes.
Date of report 2003
Author/details Canada Economic Development for Quebec Regions.
Objective of policy To promote the development of regional businesses exports by providing information and advice
through 22 ORPEX centres.
Key findings The programme should be retained because of high client satisfaction.
to establish the counterfactual.
Comments No question on impact on exports.
58
innovative environments in which SMEs operate. It is expected that this will
increase the competitiveness and performance of SMEs in the supported
clusters and innovation systems. Evaluation of indirect support to SMEs is
Table 3.6. Enterprise partnerships for exporting, Sweden, rebro
Available online www.fsf.se/eng/publications.htm
Country and region Sweden, rebro
Title of report Interprise DECIDE V Expectations, experiences and results.
Date of report 1999
Author/details Lundstrm, A., Swedish Foundation for Small Business Research.
Objective of policy To promote exports by brokering international partnerships among enterprises in the rebro
region through meetings with other firms brokered at an international enterprise fair hosted in
the region.
Key findings On average 19 meetings were held. Nearly half of participating enterprises considered that the
programme was a good opportunity to establish collaborations. After 9 months one-quarter of
the enterprises had established new contracts.
Operated through a questionnaire to participant enterprises on their satisfaction, experiences in
establishing partnerships, obstacles to success and new export contracts won.
The counterfactual is not explicitly addressed in terms of whether any of the contracts would have
been won anyway without participation in the fair.
Comments In additional analysis, impacts were compared between different types of enterprises, e.g. sector
and previous experience of internationalisation, and the success of the fair was compared with
that of one in another region (Lulea, Sweden).
Table 3.7. Small business grants, UK, North East England
Available online www.ignite-ne.com/ignite/DTIEvaluations-hvstr.nsf/LookupUNID/
CDEE2D147E62AD2280257149004984EF?OpenDocument
Country and region United Kingdom, North East England
Title of report Tyne and Wear Business Start-up Support (Small Business Grant) Project Evaluation.
Date of report 2005
Author/details ARK 19
Objective of policy To increase levels of business start-up and business survival in Tyne and Wear by providing
small capital grants to start-up businesses as part of an integral package of advice, information
and support.
Key findings 498 new jobs created and 117 new businesses established.
Cost per new job created in 2003-4 was GBP 3 000 and cost per new business created was
GBP 13 000.
Review of monitoring data, including outputs reported by beneficiaries.
Survey of beneficiaries on job creation and business start-up outputs.
Relies on the implicit counterfactual reported by the beneficiary.
Comments The evaluation pointed to potential unreliability of self-reported job creation outcomes in
monitoring data declaration forms. A telephone survey enabled further exploration of recipients
own views on their job creation rates. Job creation may be double counted with other initiatives
supporting the same enterprises.
59
more difficult than is the case for direct support, whilst it also must be
recognised that SME promotion is often not the sole, or sometimes even the
principal, aim of the activity. However some assessments are needed of this
important policy activity.
Table 3.9 presents the findings of a major European Commission
programme, which unfortunately does not proceed to a full assessment of
impacts on SMEs. Table 3.10 presents an evaluation of an initiative to enable
SMEs to learn about best practice processes through networking with other
SMEs and larger host companies.
Support to areas of geographical disadvantage
One of the objectives of many SME and entrepreneurship programmes at
regional and local level is to increase incomes and employment in
disadvantaged regions and localities. Evaluations of three types of
programmes for tackling geographical disadvantage through small firm
promotion are reviewed: Distressed urban areas, lagging regions and
distressed rural areas.
Distressed urban areas
Tables 3.11 and 3.12 describe sophisticated evaluations of state-led
Enterprise Zone initiatives in distressed urban areas in the USA. Table 3.13
Table 3.8. Regional development agency grants, Ireland, Shannon
Available online www.envplan.com/abstract.cgi?id=c170303
Country and region Ireland, Shannon
Title of report An evaluation of a regional development agencys grants in terms of deadweight and
displacement.
Date of report 1999
Author/details Lenihan, H., Environment and Planning C: Government and Policy, 17.3, 303-318.
Objective of policy Support the growth and competitiveness of new and existing firms in the Shannon regional
development agency area by providing investment grants, with particular focus on indigenous
firms.
Key findings 53% of firms reported full deadweight i.e. the policy was judged to have made no difference to
their investment project and 79% of associated jobs would have occurred anyway.
Only 4% of sales from assisted firms displaces output and jobs from other Shannon region
companies.
In-depth face-to-face interviews with managing directors of 73 firms that received grants in 1995.
Data are then used to obtain estimates of deadweight and displacement in the region.
Relies on the implicit counterfactual reported by the beneficiary, which may be biased or unreliable.
Comments Provides an in-depth exploration of the extent to which policy succeeds in avoiding two key
problems namely deadweight (supporting projects that would go ahead anyway) and
displacement (indirectly reducing activity in other firms by competition for markets and
resources). Here displacement is estimated only in the product market.
60
reviews a major evaluation of Enterprise Zone policies in the UK that included
matched firm comparisons, but in a less sophisticated way than the two
presented US evaluations. Table 3.14 shows a combined quantitative and
qualitative evaluation of a UK programme that aimed to encourage the
development of more innovative and specialist business support to enterprise
in disadvantaged areas.
Lagging regions
Table 3.15 describes an evaluation of enterprise support in lagging
regions of the UK. The support was available for inward investment projects,
for local large companies and for new and small firms. However important
direct and indirect (through linkages and multipliers) impacts on SMEs and
entrepreneurship are picked up.
Table 3.16 focuses on a more sophisticated evaluation of SME-specific
financial support in areas of industrial decline in regions of northern Italy.
Rural policy
Table 3.17 shows an evaluation of a rural entrepreneurship initiative.
Although presenting findings on employment impact this is based on analysis
of management information rather than an evaluation as such. Table 3.18 sets
Table 3.9. Local innovation system policy, EU regions
Available online ftp://ftp.cordis.europa.eu/pub/innovation-policy/studies/
studies_regional_technology_transfer_strategies.pdf
Country and region Regions in European Union countries
Title of report Assessment of the Regional Innovation and Technology Transfer Strategies and Infrastructures
(RITTS) Scheme: Final Evaluation Report.
Date of report 2000
Author/details Charles, D., Nauwelaers, C., Mouton, B. and Bradley, D.
Objective of policy To upgrade the regional technology and innovation infrastructure in 42 participating European
Union regions with reference to the needs for support shown by regional SMEs.
Key findings The programme brought more strategic thinking, created a regional dialogue, helped develop a
broader concept for innovation and supported many regions in developing the scene of
innovation support infrastructure and to develop actions to rationalise, better define and
augment the visibility of the infrastructure.
Questionnaires were sent to project managers and members of the steering committee in each
region on regional characteristics, programme actions and results.
A sample of case studies of projects were undertaken to identify a set of good practices through
interviews with key policy actors and firms.
Comments The focus was more on the impact on innovation infrastructures and policies than on the impact
of these changes on SMEs. It offers a participatory and formative approach but summative
evidence is weak.
61
Table 3.10. Business networking, UK, North East England
BB1025FD7910342B8025715C0053CE91?OpenDocument
Country and region North East England, UK
Title of report Evaluation of the RIF-funded Business to Business Project.
Date of report 2003
Author/details Public and Corporate Economic Consultants.
Objective of policy To support innovative sectoral/geographic networks and promote access to and application of
new technologies by SMEs through visits and mentoring of SME networks by larger host
companies.
Key findings The objectives of the programme were appropriately specified. The majority of project
participants felt their objectives were largely satisfied. Around half the benefits companies
ascribed to the programme would not have accrued without participation. No adverse comments
were made on the management and administration of the project. There is concern over potential
duplication of effort with other programmes.
Sophistication of evaluation The evaluation is both qualitative and quantitative, Step 3.
The qualitative assessment includes desk based research on project and strategic documentation,
face-to-face consultations with project partners and a telephone survey of beneficiaries. The focus
is on assessment of the appropriateness of the project rationale, the management and
administration of the programme, and its relationship with similar initiatives. Questions are included
on the satisfaction of participants with the programme.
The quantitative assessment included beneficiary self-assessments of impacts on their performance
and an assessment of the counterfactual through asking whether participants would have otherwise
taken alternative actions to achieve the same ends.
Comments This is an example of a small-scale evaluation at a relatively early stage of the project life cycle.
It demonstrates that useful information can be achieved from relatively low cost evaluation
approaches, although sophisticated estimates of impact could not be obtained.
Table 3.11. Enterprise Zone evaluation, US, Indiana
Available online http://papers.ssrn.com/sol3/papers.cfm?abstract_id=476122
Country and region United States of America, Indiana
Title of report Tax policy and urban development. Evidence from the Indiana enterprise zone program.
Date of report 1994
Author/details L. Papke, Journal of Public Economics, Vol. 54, pp. 37-49.
Objective of policy To induce investment and enhance employment opportunities for residents in 15 distressed
urban areas in Indiana state by providing tax preferences to capital and labour.
Key findings The programme permanently increased the value of inventories by about 8% in the zones and
reduced the value of machinery and equipment by about 13%. Unemployment claims decline by
about 19%.
Estimates impact using changes in capital stock and unemployment before and after zone
designation.
External influences on performance are controlled for by using a panel of zones and non-zones.
The panel approach also controls for non-random selection of areas for zone designation.
Comments Tackles the issue of area selection bias.
62
out a broad evaluation of the appropriateness of business support
arrangements in a largely rural county of the United Kingdom.
Table 3.12. Enterprise Zone evaluation, US, Five States
Available online www.sciencedirect.com/science/article/B6V89-40WDSVD-3/2/
4a74e2a35255bf9e1f30f2acb114dc63
Country and region United States of America, five states
Title of report Enterprise zones and local employment: evidence from the states programs.
Date of report 2000
Author/details D. Bondonio and J. Engberg, Regional Science and Urban Economics, Vol. 30, pp. 519-549.
Objective of policy To generate employment through business development in distressed urban areas in five states:
California, Kentucky, New York, Pennsylvania and Virginia.
Key findings The analysed Enterprise Zone programmes do not have a significant impact on local
employment. Programme impact does not depend on the monetary amount of the incentives or
on specific features of programme design.
Compares zone and non-zone performance pre- and post-designation.
A first method controls for selection bias in the areas receiving designation by including in the
regression model an area-specific fixed effect and an area-specific growth rate.
A second method controls for selection bias by estimating the designation probability of each area
based on pre-designation characteristics and controlling for differences in these characteristics by
including the predicted probability in the model.
Comments Tackles the issue of area selection bias by including information on the characteristics of the
areas selected in econometric models. Uses the diversity of state programmes to estimate the
impact of different programme designs.
Table 3.13. Enterprise Zone evaluation, UK
Available online www.communities.gov.uk/archived/general-content/citiesandregions/finalevaluation
Country and region United Kingdom
Title of report Final Evaluation of Enterprise Zones.
Date of report 1995
Author/details PA Cambridge Economic Consultants.
Objective of policy To overcome economic and environmental problems in zones of urban and industrial economic
decline by promoting business development and physical regeneration.
Key findings Approximately 60 000 additional jobs were created in supported businesses at an estimated cost
per job year of GBP 1 700 pounds per year (1994/95 prices). The most important benefit for
business development was business rates relief.
Sophistication of evaluation Steps 3-5:
The main estimates were based on recipients views of the difference made by the assistance.
A comparison with matched enterprises outside the zones was used to verify the results.
Comments Provides explicit treatment of displacement and leakages.
63
Table 3.14. Evaluation of enterprise support in disadvantaged areas, UK
Country and region United Kingdom
Title of report Evaluation: The Phoenix Development Fund.
Date of report 2005
Author/details Peter Ramsden, Freiss Ltd.
Objective of policy To tackle social exclusion by supporting innovative projects providing business support to
enterprise in disadvantaged geographical areas and to groups currently underrepresented
among business owners.
Key findings The Fund has encouraged new innovative approaches to providing outreach services. Projects
were successful at reaching underrepresented groups. The Fund has not succeeded in
transforming the mainstream provision of business support.
The main impact assessment was through a client survey that measured the support received and
beneficiary views on the impact of that support.
The client survey was accompanied by 24 case studies of supported service providers to understand
the delivery methods and services that were developed and their rationales. The focus was on
uncovering interesting practice.
Comments There was particular focus on assessing whether the programme encouraged fresh thinking in
the types of outreach services provided and the demonstration effects on mainstream providers.
Insufficient information was available on costs of different approaches used by projects to reach
a judgement on value for money.
Table 3.15. Regional policy evaluation, UK
Country United Kingdom, Assisted Areas
Title of report Evaluation of Regional Selective Assistance 1991-1995: Final Report.
Date of report 2000
Author/details Arup Economics and Planning.
Objective of policy To create and safeguard employment in Assisted Areas in a cost-effective manner.
Key findings Almost 40 000 net discounted permanent job equivalents were supported in Assisted Areas over
the period 1991-1995. The policy has operated in a cost effective manner, with a central estimate
of GBP 17 500 per net discounted job in 1995 prices.
Estimates of additionality and impact are based on recipients views of the difference made by
assistance.
Displacement effects are estimated from respondent information on the location of their
competitors. Displacement outside of the assisted areas was not considered.
The timing and duration of jobs created are considered important. The number of job years the
support created was estimated and job years closer to the present were assumed to be more
valuable through discounting to present value job years.
Comments A survey of recipient firms provided information on expected job life, additionality, displacement
and linkages, which was used to adjust gross employment information provided to government
by all recipient firms.
64
Conclusion
Thi s secti on has set out a number of sub-nati onal SME and
entrepreneurship policy evaluations that show the range of evaluation
options. It is hoped that the methods outlined and the results demonstrated
will build the demand for more and better evaluation. Some of the evaluations
reported lack the sophistication required to provide robust estimates of
programme impact, although there are also examples of very good
quantitative evaluations falling into the categories of Steps 4-6 of the Six Steps
Table 3.16. Regional policy evaluation, Italy
Available online http://eur.sagepub.com/cgi/content/abstract/13/3/225
Country and region Italy, EC Objective 2 regions
Title of report Do Business Investment Incentives Promote Employment in Declining Areas? Evidence from EU
Objective 2 Regions.
Date of report 2006
Author/details D. Bondonio and R. Greenbaum, European Urban and Regional Studies, Vol. 13.3, 225-244.
Objective of policy Increase economic development in regions of industrial decline through packages of SME
incentives.
Key findings 24,000 additional jobs between 1995 and 1998 can be attributed to the programme at an
average cost of 21,000 euros. Incentives were more successful when rewarding growing
industrial sectors.
Longitudinal parametric model analysing firm employment data aggregated by province and
industrial sector.
Exploits the fact that firms in non-target areas receive incentives as well as firms in target areas, but
that Objective 2 increases the overall programme budget (and hence the number of firms
supported) in target areas.
Trends in supported and non-supported firms are included in target regions and non-target regions.
Comments Tackles area selection bias.
Table 3.17. Rural policy evaluation, Canada, Quebec
Available online www.dec-ced.gc.ca/Complements/Publications/Evaluation-EN/eval_entr_rurale_en/css/
eval_entr_rurale_en.htm
Country and region Canada, Quebec
Title of report Evaluation of the Rural Enterprise Program. A Component of the Bas Saint Laurent and Gaspsie/les
de la Madeleine Regional Strategy.
Date of report 2000
Author/details Crac Inc.
Objective of policy To support economic activity in rural communities by financing support to new and small enterprises
Key findings 59 grants of between CAD 11,000 and CAD 25,000 were associated with the creation of 1.7 jobs and
protection of 3.0 jobs per project approved.
Analysed project take up and associated outputs declared in project applications
Comments No survey of beneficiaries.
65
approach. Other examples are interesting because of their more qualitative
focus on policy options and stakeholder involvement. Ideally, quantitative and
qualitative approaches should be brought together in comprehensive
programme evaluations where budgets permit and there is an opportunity for
significant policy adjustment.
Table 3.18. Rural enterprise support, United Kingdom, Northumberland
EEF952CCD3B762B68025716B003E72CE?OpenDocument
Country and region United Kingdom, Northumberland
Title of report Business Support in Northumberland: A Review.
Date of report 2005
Author/details SQW Ltd.
Objective of policy To support business start-up and development through a range of services provided by the
Northumberland Strategic Partnership, mainly operated through Business Link for Northumberland.
Key findings In reorganising services it is important to build on existing strengths and support infrastructures.
However, there is a need to develop more extensive geographical coverage and better support
businesses with high value added potential. Two-thirds of respondents believed the support had a
significant effect on increased sales and increased productivity. Over 80% found services vital,
important or useful.
Sophistication of evaluation The evaluation combines quantitative (Step 3) and qualitative elements.
There is an assessment of quantifiable outputs through a telephone survey of 60 businesses that
received support.
There is also assessment of qualitative results such as improving access to services and the
perceived quality of services, drawn from views of companies, Business Link for Northumberland
staff and staff in partner organisations.
Comments The evaluation has a broad focus addressing a package of funding streams and providers. It makes
recommendations on how to organise future delivery based on what has worked well. It is largely
qualitative in focus.
ISBN 978-92-64-04008-3
OECD 2007
67
Section 4
The Role of Peer Review in Evaluation
4. THE ROLE OF PEER REVIEW IN EVALUATION
68
Introduction
The Six Steps method is particularly appropriate for the evaluation of
large scale programmes for which information is needed for decisions on the
future of the programme (e.g. continue at current scale, scale-up, scale-down
or close, adjust content and delivery methods). However, whilst it is possible
to identify many such specific programmes on which such decisions are
required, the reality is that SME and entrepreneurship instruments are not
operated on their own but are part of a patchwork quilt of interventions
addressing issues such as finance, management training, provision of
premises, or provision of information. In these circumstances it is difficult to
isolate the impact of any single programme since, the success of one policy,
depends upon the presence of the others.
In this environment, the difficulty with assessments of individual
programmes is that they tend to lack the overall coherence emphasised as
being vital in the COTE framework (Appendix B). Often, policy makers need to
go beyond individual programme evaluations to assess successes and
challenges in developing broader entrepreneurship and SME strategies that
represent more comprehensive and integrated packages of measures. In this
case, the Framework sees a valuable role for a helicopter view of the policy
environment.
The peer review method is discussed here as a valuable tool for the
development of such big picture visions for entrepreneurship and SME
policies in countries or regions. In addition to assuring coherence, the peer
review has the second key advantage of supporting policy development in the
light of information on best practices and lessons learned elsewhere through
an exchange with policy makers and experts from other places.
Both the OECD Working Party on SMEs and Entrepreneurship (WPSMEE)
and the Local Economic and Employment Development (LEED) Programme of
the OECD undertake peer reviews in the field of entrepreneurship and SME
policies. However, peer reviews can be organised by any organisation with
like-minded counterparts.
The peer review methodology
The essential feature of the peer review methodology is that exchanges of
experiences are organised amongst a group of reviewers from different
organisations in different places, but who are seeking to address similar
69
challenges in comparable contexts. They may either focus on reviewing
performance in one organisation that calls for the review or on reviewing all
the participating organisations.
Key advantages of the approach are that:
Peer reviews are part of a learning process, with the specific aim of
developing better strategies.
Information on good practices and innovations in policies and programmes
can be shared with other participating organisations.
The peer review provides a mechanism to build consensus and momentum
behind a policy approach in the participating areas.
There is no standardised peer review mechanism, but peer reviews tend
to share the following common structural elements: a basis for proceeding; an
agreed set of principles, standards and criteria against which performance
will be reviewed; designated actors to carry out the review; and a set of
procedures leading to the final result. Reviews also tend to consist of the three
following core phases: preparation, consultation and assessment.
Usually, each reviewer will examine the development challenges and
current policy responses in the case study organisations under review and
contribute an assessment of the opportunities for further policy development
based on their knowledge of practices in their own environment, supported
with illustrative material on approaches used elsewhere that may be of benefit
in the case study area.
For a review to be effective, it should meet certain conditions (OECD, 2003):
Value sharing Participating organisations should share the same views on
the standards or criteria against which to evaluate performance. A strong
common understanding on these elements will prevent uncertainty or
backtracking during the process.
Mutual trust Since peer review is, by its nature, a co-operative, non-
adversarial process, mutual trust is important for its success. While the peer
review process itself can contribute to confidence building, a large degree of
trust among the participants should be present from the beginning to facilitate
the disclosure of data, information and documentation which are essential to
the process.
An adequate level of commitment by participating organisations This
means not only supplying enough resources to carry it out, but also being
fully engaged in the process, whether reviewing or being reviewed.
The independence, transparency, accuracy and analytic quality of the
review panels work are essential to the effectiveness of the process.
The credibility of the peer review process is essential to its effectiveness
There is a strong link between the credibility of the process and its capacity
to influence. The involvement of the reviewed organisation in the process
70
and its ownership of the outcome is the best guarantee that it will
ultimately endorse the final report and implement its recommendations.
However, this involvement should not go so far as to endanger the fairness
and the objectivity of the review. For example, the organisation under
review should not be permitted to veto all or part of the final report.
OECD national SME reviews
The OECD WPSMEE undertakes national peer reviews on the theme of SME
Issues and Policies. The reviews result in a published report that assesses
accomplishments, spells out shortfalls and makes recommendations for the
reviewed country [e.g. Turkey in 2004 (OECD 2004b) and Mexico in 2006 (OECD
2007)].
The process works in the following way for an SME review of a given country:
The preparatory phase defines the outline of the report which includes
generally:
Main features of the SME sector in the country under review: This chapter
provides an overview of the size and structure of the SME sector and its
main characteristics.
Strengthening the entrepreneurial business environment: This chapter provides
the macroeconomic background which explains how SME performance and
development are affected by economic events and policy at the national
level and by framework conditions in which all businesses operate.
SME policy and programmes: This chapter analyses the SME policy carried out by
authorities at national and sub-national levels through specific programmes
and measures.
Policy evaluation and recommendations.
At each of the following stages the country review team which includes,
in addition to the Secretariat staff, experts from several reviewing countries, is
carefully consulted:
A questionnaire is developed by the Secretariat, covering the various topics
identified in the outline. The questionnaire is sent out to the country
authorities in charge of the review and of the co-ordination of responses at all
levels. The quality of the answers and the national authorities involvement
and co-operation with the Secretariat are central to the effectiveness and
success of the review.
A preliminary draft based on the countrys responses and the Secretariats
own research is prepared.
In the fact-finding/consultation phase, the Secretariat carries out a fact-
finding mission, on the basis of the preliminary draft, in order to have an
intensive dialogue with the country concerned: policymakers in charge of
programmes, as well as academics and representatives of industry, NGOs,
71
etc. It also conducts on-site visits to places such as clusters, training
centres, etc. The assistance of the national authorities in setting up, to the
extent possible, the most complete and cost-efficient fact-finding mission
is also crucial to the quality and reliability of the review.
The following phase is dedicated to the preparation by the Secretariat of a
complete draft report which includes conclusions and recommendations.
During a second mission in the country, the Secretariat discusses the draft
thoroughly with the national authorities and the draft is amended
consequently. Experts from several reviewing countries can also take part in
discussions with the reviewed country.
In the assessment phase, the body responsible for the review in this case the
WPSMEE grouping all 30 OECD member countries discusses the draft report.
The few countries which are the lead examiners launch the debate, while the
rest of the group participates actively in the final discussion. A delegation from
the reviewed country (civil servants from ministries and agencies and at
different levels of government, and in some cases ministers in charge of SMEs),
answers questions from the member countries. The WPSMEE then amends the
conclusions and recommendations in the light of the discussion and approves
the review. The report is published under the responsibility of the OECD
Secretary-General.
Publication and follow-up include in the country a press conference on the
final report and its recommendations.
OECD regional and local entrepreneurship reviews
The OECD LEED Programme provides a similar forum for peer review work
on entrepreneurship, employment and economic development issues affecting
cities and regions in OECD member and non-member countries. Participation is
organised by city and region representatives from governments, development
agencies and their partners who approach the LEED Secretariat to signal their
interest.
There are four thematic series of reviews in the field of SME and
entrepreneurship policies.
Local Entrepreneurship Reviews. This series examines the challenges and
opportunities for further development of the entrepreneurial culture,
framework conditions and policies and programmes in participating cities
and regions.
Reviews on Entrepreneurship and Local Innovation Systems. The focus
here i s on how l ocal governments and agenci es can promoted
entrepreneurship by strengthening of the local innovation systems in
which business operates.
72
Reviews on Innovation and the Internationalisation of Local Economies.
These reviews examine how cities and regions can promote entrepreneurship
and innovation by internationalising knowledge-intensive indigenous
firms, attracting and embedding knowledge-intensive foreign direct
investment, attracting highly-skilled labour and building the international
innovation connections of research organisations.
National Reviews of the Local Dimension of Entrepreneurship Policy.
Whereas the above reviews focus on regional and local agencies, this series
is designed for national governments and development agencies. It focuses
on how they can achieve appropriate tailoring of policy to differing local
geographical contexts (e.g. distress urban, rural, high technology cluster,
etc.) and an appropriate balance between central and local design and
delivery of policy.
The approach used for these reviews usually includes the following
elements:
Local diagnostic work, including assessment of existing evidence and
documentation and in some cases local survey work with companies or
research organisations.
A one-week visit by an international review panel to discuss policy
challenges and options with key local stakeholders.
Preparation of a draft report and discussion with local stakeholders.
Follow-up visits to deepen the evidence base and explore new issues.
Seminars and workshops to debate, validate and explain findings and
recommendations and move to a locally-generated policy action plan.
The review reports include discussion of:
International trends, issues and best practices and their implications for the
participating organisations.
Assessment of the policy challenges and opportunities in the reviewed
organisations.
Recommendations for policy development in the review organisations.
Illustration of the recommendations with learning model descriptions from
other countries.
OECD evaluation guidance
The OECD also uses the peer review approach to provide evaluation
guidance for local and regional development. Among these evaluation support
activities figures the Local Economic and Employment Development
Programmes work to support the creation and strengthening of strategic
evaluation frameworks. These may be frameworks for SME and entrepreneurship
73
programmes, for other economic and employment development policies or for
broader regional and local development programmes. Support is provided
to national governments on creation of an evaluation plan that will enable
assessment of progress towards national objectives and the provision of
mentoring and guidance to its local and regional partners to support capacity in
evaluation. Work is also undertaken with regional and local governments and
agencies to create evaluation plans covering: priorities for evaluation; setting up
an organisational structure for evaluations of local development programmes;
identifying the information and data to collect and monitor and how to
categorise, store and use it; deciding what to evaluate, when and how to carry out
the evaluation; benchmarking evaluation results; communicating results to
stakeholders; and feeding results back into future policy design.
ISBN 978-92-64-04008-3
OECD 2007
75
Section 5
Reviewing the Aggregate Impact
of Public Policies
5. REVIEWING THE AGGREGATE IMPACT OF PUBLIC POLICIES
76
Introduction
Sect i on 2 di scussed t he eval uat i on of i ndi vi dual SME and
Entrepreneurship programmes delivered by national governments.
Programmes delivered by local and regional authorities were covered in
Section 3.
However, it is widely recognised that total impact of government upon
SME and entrepreneurship performance may be either greater or less than the
summation of the impact of the individual programmes, depending upon
whether they complement, duplicate or even destructively compete with one
another. Where the impact of the sum of the programmes exceeds that of
their individual impacts then policy is regarded as coherent. It is important
for SME and entrepreneurship policy makers to be able to assess the issue of
policy coherence and make decisions that will increase synergies. The first of
the two main themes of the current section then focuses on how we might
assess the coherence of sets of SME and entrepreneurship policies.
Its second key theme is that the total impact of SME and Entrepreneurship
policies is strongly influenced by public policies which are not delivered by the
main SME department of government. Indeed such policies could be more
influential than the programmes delivered by the SME department.
Two examples of such policies are:
Public policies that target SMEs but where the policy is formulated and
delivered by an organisation within government other than the main SME
department of government.
Policies which influence SMEs/entrepreneurship that are delivered by
departments of government that do not view themselves as having SMEs/
entrepreneurship as their prime, or perhaps even their partial, focus.
It is important for policy makers responsible for SME and entrepreneurship
issues to understand the impact of such mainstream policies on SMEs to help
them in their role of SME-proofing of mainstream government policies and
in influencing decisions on such policies so that they take into account the
SME impacts. However, this is a difficult area for evaluation.
This is a more exploratory section of the Framework, since evaluations on
of policy coherence and the impact of mainstream policies on SMEs are rare
and guidelines on how to undertake such evaluation are not yet in place.
77
Nonetheless, this Section discusses some relevant research literature that can
be referred to in helping pushing forward evaluation practice in this area.
Our purpose here is to review the evaluation challenge and provide
examples of promising approaches that have been used to assess the impact
of non-SME specific or aggregate public policies upon SMEs.
Impact of mainstream policies on SMEs
The most striking quantification of policies targeting SMEs that are
delivered and funded from budgets separate from the lead SME department
or agency of government comes from the UK. In 2002, the Small Business
Service published a cross cutting review of business support which estimated
the total public expenditure on services for small businesses
1
within England.
It concluded that, excluding payments from the Common Agricultural Policy
(CAP), expenditure on direct support to small businesses was about
GBP 2.5 billion. In addition, GBP 2.6 billion was provided in the form of tax
measures that benefited small firms.
A second review in 2005 attempted a more detailed analysis and mapping
of expenditure on support for small businesses (PACEC, 2005). It included an
estimate of the share of expenditure on vocational training from which small
businesses benefited, but which had not been included in the original review.
This was calculated to be 1.7 billion GBPs on vocational training, implying that
the combined cost to the taxpayer, excluding CAP payments was 6.8 billion
GBPs which was broadly similar to taxpayers expenditure on the police force.
Until April 2007, the UK had a separate agency responsible for SMEs the
Small Business Service (SBS).
2
In 2002 its budget was approximately
GBP 400 million, but by 2005/6 this had fallen to GBP 213 million following a
decision to transfer business advisory services Business Link to the
Regional Development Agencies. The key point is that SBS was at that time
responsible for setting the national enterprise policy framework. However it
only directly controlled around 5% of public funds provided to assist the small
business sector.
The second key point is that other departments of government which, at
first glance might not seem to have SMEs as their core responsibility, spend
considerably more public money in 2003/4 than SBS on activities and
programmes that benefit small firms. Such departments or public
organisations include:
Department of Culture Media and Sport;
Department for Environment, Food and Rural Affairs;
Department for Work and Pensions; and
Learning and Skills Council.
78
Finally, the disparate nature of the funding and delivery of Entrepreneurship
and SME policy is reflected in the finding that there were 265 activities supporting
small businesses provided by 15 central government departments and
agencies. In addition, there were 800 services for small businesses provided by
three Regional Development Agencies examined in the Review. Given that
there are nine such Agencies, [if the London Development Agency is included],
it is reasonable to assume that a national figure might be in the region of
2 500 GBPs (PACEC, 2005).
Of course there is nothing, in principle, bad about diversity. Different
SMEs have different needs depending upon their location, sector,
circumstances and motivation. So, if policies were tailored to reflect these
different sections of the marketplace, then this might be an expected outcome
if policy was being delivered efficiently.
However, central to demonstrating such efficiency, is the existence of a
committee or organisation within government charged with comparing, for
example, the impact upon SMEs of the GBP 2.6 billion of tax reliefs with the
GBP 1.7 billion expenditure by the Learning and Skills Council, or with the
GBP 2.5 billion of expenditure by the 15 different central government
departments. Without formal oversight of the total spending package there
must be a risk such expenditure is not joined up. This implies the strong risk
of duplication, or a focus upon departmental priorities, rather than those of
the SME and/or the taxpayer. In response to the complexity of business
support provision within the UK the Small Business Service established the
Business Support Simplification Project (BSSP) which has brought together key
departments, delivery organisations and other stakeholders. BSSP has put in
place a framework and underpinning processes to reduce duplication of
support programmes including a rigorous economic appraisal of existing and
proposed programmes drawing on available evaluation evidence and wider
economic literature.
The evidence, presented in the above paragraphs, of a diversity of funding
and delivery organisations for SME policy is taken from the UK, but it is likely
to apply to many other countries. The emphasis is likely to differ, but
countries for example that very clearly distinguish between the role of federal
or national government and state or provincial governments are perhaps
particularly likely to encounter such issues. This is because entrepreneurship/
SME policy is a topic likely to straddle the interests of many organisations
within all forms of government.
The central issue is that it is of little concern either to the taxpayer or to
the SME which organ of government delivers a service. What are of concern is
whether the right service is provided and whether such services are provided
at the lowest possible cost to the taxpayer. Where the services are provided in
79
a disparate manner without a central strategic or co-ordinating body taking
overall responsibility there is a real risk of combining duplication with
omission, implying inefficiency.
Policies which influence SMEs/entrepreneurship that are delivered
by departments of government that do not view themselves as having
SMEs/entrepreneurship as their prime, or perhaps even their partial,
focus
We now turn to examine a range of public policies that impact directly
upon SMEs. They influence the start up of new firms and the performance of
small firms once established, and yet they are frequently not policies over
which the main SME department of government exercises any direct control.
However, if the main SME department is given targets relating to the start-up
and performance of new and small firms, it is these policies which may have
an equally strong, or perhaps even stronger, influence upon whether these
targets are achieved than the SME departments own policies.
The role of this sub-section is to identify these policies and then to review
the evidence on the impact of these policies upon SMEs and entrepreneurship.
The policies that will be reviewed are:
macro policies to create a stable economic environment, with low inflation,
interest rates and unemployment;
taxation;
unemployment benefits;
business regulation policies (licensing policies);
immigration/emigration policies; and
competition and public procurement policies.
Macro policies to create a stable economic environment, with low
inflation, interest rates and unemployment.
There is evidence that variations in entrepreneurship are linked to
changes in the macro economy. Shane (1996) shows that rates of
entrepreneurship, in the United States over the period 1899-1988, were clearly
lower during times when interest rates were high. Cullen and Gordon (2002)
also find the same result for a later period in the United States. Both studies
attribute this to small business owners having to incur higher borrowing costs
at times of high interest rates. However, a different specification of Shanes
measure of entrepreneurship leads to interest rates being non-significant, but
being replaced as significant by economic growth implying that
entrepreneurship is higher in times of prosperity.
3
80
The role of macro economic conditions upon self employment rates have
also been examined by Parker (1996). He also found that self employment was
inhibited by higher real (excluding inflation) interest rates. Interestingly,
Parker also implies that self employment is higher during periods of economic
stability, since he finds the self employed to be deterred by risk.
But it is not only the absolute scale of macroeconomic variables such as
inflation, interest rates and growth. Their predictability also influences the
actions and behaviour of the small business owner. The former Chief
Economist at the World Bank, Joseph Stiglitz, viewed macroeconomic
predictability as delivering a valuable boost to entrepreneurship in the
developing world.
4
The above appears to imply that entrepreneurship is enhanced by benign
macroeconomic conditions, and it might therefore have been expected that
UK small firms would have performed better during the period 1997-2005,
than during the more turbulent economic times of 1991/7. However Cosh et al.
(2006) do not find this to be the case. Six year survival rates of SMEs are
marginally higher (59.5% compared with 56.4%) in the earlier period and
employment growth rates amongst survivors were significantly higher (3.4%,
compared with 2.2%).
Despite the Cosh et al. findings, the bulk of the evidence suggests that a
benign and predictable macroeconomic environment of high aggregate
demand, low inflation and low interest rates is one in which entrepreneurship
is most likely to prosper.
Taxation
A key distinction is made between the impact of income or personal
taxes and that of business taxes. In matters relating to taxation on income,
employees and the self employed differ in three important respects. First,
whilst employees have their tax removed at source by their employer, the
self-employed declare their income to the tax authorities. Second, the
employee pays tax immediately, whereas the self-employed pay in arrears,
normally at the end of a financial year. Thirdly the self employed are able to
claim expenses against their income on a scale not normally available to the
employee. These three differences offer potential financial benefits to the self-
employed that are not available to the employee and might therefore
influence an individuals choice in favour of self employment.
Given that most individuals would choose not to pay taxes but still
receive the public benefits it is expected that lowering tax rates for the self-
employed or raising taxes on employees leads individuals to a shift to self
employment. In support of this, Schuetze (2000) finds that increases in average
income tax rates have large and positive effects on self employment in Canada,
81
1983-94. For Sweden, Folster (2002) finds that reducing the tax burden by 10% (of
GDP) would increase the share of the self employed in employment by about 3%.
Davis and Henrekson (1999) argue that Swedens high business tax
regime explains that countrys low rates of new firm formation over several
decades. They argued that the combination of high corporate tax rates
combined with generous provisions for accelerated depreciation meant the
key beneficiaries were large firms with heavy capital investment. By
implication, those penalised were new and small firms with modest tangible
assets. They also support the view of Schuetze that high personal taxes
depress rates of entrepreneurship.
However both Robson and Wren (1999) and Bruce (2000) draw an
important distinction between changes in the average, and changes in
marginal, rates of taxation. They find that lowering marginal rates of tax
increases the efforts of the self employed. This is because the self employed
have more opportunity to vary their input, in terms of effort or hours worked,
than employees and so retain more for themselves by working the marginal
hour. In contrast, lowering average tax rates reduces the potential gains to the
self employed from evasion and so reduces the differential between paid and
self employment. It is because of the offsetting influences of these two factors
that more recent empirical evidence of changes in income taxes upon self
employment has become less clear.
Robson and Wren (1999) show, using simple bi-variate analysis, that
countries with high marginal and high average tax rates Denmark,
Netherlands, Finland and Germany had low rates of self employment in the
1978-92 period. However when a range of other factors are taken into account
they generate their key result that high marginal rates decreases self
employment because this leads to reduced effort, whereas high average
rates increase self employment because of the greater opportunities for
evasion.
Schuetze and Bruce (2004) conclude that that the evidence of the impact
of taxation on self employment is now less clear. They identify several studies
pointing to higher (marginal) taxes increasing self employment because of
risk. The logic underlying such a finding is that an individual may be more
willing to enter high risk self employment if they know that, in the event of
failure, they will be able to offset any losses against tax.
5
They conclude:
The fact that self-employment seems to increase with income tax rates calls into
question the common view that higher taxes hamper self-employment
[Schuetze and Bruce (2004), pp. 28-29].
The above discussion has focussed exclusively on the impact of income
taxes, but governments also impose business, sales and inheritance taxes, all
82
three of which may influence the behaviour of small businesses and their
owners. For example, Michaelas et al. (1999) showed that taxes which were levied
on small company profits were likely to lead to lower growth rates, since the
retained profits were the prime source of funding for small company investment.
A second review is by Bjuggren and Sund (2005). They speculate on the
impact of a policy change in Sweden in 2005 to abolish gift and inheritance
tax. They report that the previous system was slow and expensive to
administer
6
and encouraged practices which were on the margins of legality.
This might suggest that the economic case for inheritance tax is weak if there
were advantages to the economy as a whole of inter-generational business
transfers. Unfortunately this case has yet to be made [Westhead and Cowling
(1998)]. Early analysis pointed towards no significant differences between the
performance of first and second generation family businesses [Westhead,
Cowling and Storey (1997)] implying there were no spill over gains from
lowering inheritance taxes.
Overall, the above suggests that a simple relationship between low rates
of taxation and an entrepreneurial economy does not exist. Instead, the
nature of taxation influences the behaviour of individuals both in their choice
of employee or self employment status, and in their choice as business
owners. It also influences their effort as a self employed person.
7
The key decisions on tax policy are rarely made by the department of
government responsible for enterprise. Such decisions are almost always made
by the Ministry of Finance, with often only a modest contribution from enterprise.
Unemployment benefits
A third example of macro policies influencing entrepreneurship and small
businesses is the decision made by government on the real value of state
support provided to individuals who are unemployed. The broad evidence is
that in those economies where individuals who become unemployed receive a
high proportion of their formal wages, new firm formation rates tend to be
lower. Nickell (1997) found that, in the mid 1990s, the benefit replacement ratio
was 90% in Denmark, compared with 20% for Italy and 30% for New Zealand.
Parker and Robson (2004) show this replacement ratio had a significant impact
upon self employment in OECD countries 1972-1993.
However, the factors influencing the choice of an appropriate
unemployment benefit ratio are likely to be driven primarily by concerns of
social justice. In such debates, factors such as entrepreneurship are likely to
play a comparatively small role. For this reason, the SME department is likely
to have only a very modest voice, even though the decision reached exercises
a powerful influence on new firm formation rates.
83
Business regulation policies (licensing policies)
The influential work by Djankov et al. (2002) has shown there are wide
differences between countries in terms of both the money and time taken to
establish a standard business. Djankov et al. then linked the cost and time of
starting a business to economic variables and concluded that:
Countries with heavier regulation of entry have higher corruption and larger
unofficial economies, but not better quality of public or private goods.
The inference was clear: that lowering the costs and time taken for
starting a business would yield significant economic and political benefits.
Both developed and less developed economies have responded to these
findings. The World Bank now has a Doing Business Website which provides
comparable data on this topic for 155 countries.
8
The data collected is shown in
Table 5.1.
Based upon these indicators a country ranking in 2007 is produced for all
175 countries. Table 5.2 reproduces this ranking, but only for OECD countries.
The number on the left hand side shows the countries ranking in terms of the
175 countries.
Table 5.1. The indicators
Starting a business
Procedures, time, cost and minimum capital to open a new business.
Dealing with licenses
Procedures, time and cost of business inspections and licensing.
Hiring and firing workers
Difficulty of hiring index, rigidity of hours of index, difficulty of firing index, hiring cost and firing cost.
Registering property
Procedures, time and cost to register commercial real estate.
Getting credit
Strength of legal rights index, depth of credit information index.
Protecting investors
Indices of the extent of disclosure, extent of director liability and ease of shareholder suits.
Paying taxes
Number of taxes paid, hours per year spent preparing tax returns and total tax payable as share of gross profit.
Trading across borders
Number of documents, number of signatures and time necessary to export and import.
Enforcing contracts
Procedures, time and cost to enforce a debt contract.
Closing a business
Time and cost to close down a business and recovery rate.
Source: World Bank Doing Business database, 2007.
84
From this it is clear that OECD countries generally are more likely to have
conditions in which it is easier to do business than those in non-OECD
countries. This is reflected in the finding that eight out of the top ten countries
are OECD members the exceptions being Singapore and Hong Kong, China.
However the lowest ranking OECD country is Greece in position 109, implying
that the remaining countries are not OECD members.
The impact of this research has been considerable, with many developed
countries reducing both the number of permits required to start a standard
business, and the minimum time needed to obtain such permits. These
changes are reflected in Table 5.3. It shows the 1999, 2004 and 2007 and the
data for the above two measures for France, Spain, and Italy.
All three countries have substantially reduced the duration and
complexity of starting a business. The number of days taken to start a
business in France in 2007 was eight, compared with fifty-three previously. A
similar substantial reduction has also taken place in both Italy and Spain.
Nevertheless it is also interesting to note that, despite the improvement
in Spain, Italy and France, all three of these countries continue to have
significantly more procedures, which take longer to complete than, for
example, Canada. Currently the latter has two procedures and the business
may be established in three days.
Given that the procedure for business creation has been accelerated
considerably in many countries it is unsurprising that such countries have
seen a rise in the number of new firms that have been created.
9
Table 5.2. Ease of Doing Business ranking
2 New Zealand 22 Netherlands
3 United States 23 Korea
4 Canada 30 Austria
6 United Kingdom 35 France
7 Denmark 36 Slovakia
8 Australia 39 Spain
9 Norway 40 Portugal
10 Ireland 43 Mexico
11 Japan 52 Czech Republic
12 Iceland 66 Hungary
13 Sweden 75 Poland
14 Finland 82 Italy
15 Switzerland 91 Turkey
20 Belgium 109 Greece
21 Germany
Source: World Bank Doing Business database, 2007
85
On the face of it, speeding and simplifying the procedures for business
creation seems to be a highly beneficial policy. If businesses are more easily
created these new firms constitute a competitive threat to existing businesses,
compelling the latter to compete or to go out of business. A second advantage
is that, in some instances, the business licensing procedure is the focus of
corruption, with those paying bureaucrats obtaining their licenses more
quickly.
10
The elimination of such corruption has considerable and wide
ranging social and economic benefits.
However, as with most legislative change, there are some important
contrary arguments.
First, the prime function of the business registration process is to provide
the basis for the protection of consumers from fraudulent or incompetent
business owners. Nobody would suggest that untrained doctors or surgeons
should be allowed to practice, and registration provides the basis for the
minimum check of the qualifications and track record of such individuals.
The wide acceptance of the need for doctors to register is presumably
because their incompetence could kill people. But the same might also be said
for an individual wishing to establish a business as a driving instructor.
Similar arguments might then be applied to electricians who could electrocute
themselves or other people if they made errors. Like doctors, both driving
instructors and electricians could kill those in their charge, so raising the
question of whether the same business licensing/regulations are appropriate
for all three types of business. If not, then this raises the supplementary
question of the nature of appropriate differences, and on what basis should
such differences exist.
Extending the argument, we might also all wish to be protected from the
financially unscrupulous. We might view it to be the role of government to
ensure that rogues are identified and prohibited from trading. Again,
registration is a minimum condition for identifying such individuals. In short,
whilst all might agree that superfluous business licensing is undesirable, there
might be less consensus on the balance between the interests of those in the
business community and consumers or others in society.
Table 5.3. Starting a business in 1999, 2004 and 2006
Number of procedures [1999 data,
followed by 2004, followed by 2007]
Duration in days [1999 data, followed
by 2004, followed by 2007]
France 15, 7, 7 53, 14, 8
Italy 16, 9, 9 62, 13, 13
Spain 11, 10, 10 82, 47, 47
Source: Data in parenthesis from Djankov et al. (2002). Other data taken from Doing Business Website,
2007
86
Second, the impact of reducing the time and cost of business licensing may
be less than might appear from an examination of official data on new firm
formation. The reason for this is that regulation is likely to have a much bigger
impact upon the number of registered, than the number of actual, new firms.
William Baumol (1990) famously argued that the total number of
individuals acting in an entrepreneurial manner varies little over time and space.
Where it does vary is in its distribution between productive and unproductive
activities. Baumol argued that there was never an absolute shortage of
entrepreneurs merely that different countries at different times generated
incentive structures that encouraged unproductive activities. The prime effect
therefore of making it easier to start a business was to convert those individuals
currently in business in the informal economy to become part of the formal
economy. So, observed changes in the number of official new firms will include
many that were previously contributing to economic welfare, but as part of the
informal economy. Observing only the number of firms in the formal economy
may therefore lead to inflating the significance of reducing the licensing burden.
There is some evidence for this outcome. The first draws upon
comparisons between England and Spain. Capelleras et al. (2008) show there are
marked differences between heavily regulated Spain
11
and lightly regulated
England. They show that, when only registered firms are examined, the Spanish
firms, as would be expected, start larger
12
and grow more slowly than the
English firms. Inclusion of all firms, however, leads to these differences
disappearing. This implies that the prime impact of licensing is to influence the
distribution of enterprises between the formal and the informal sectors, rather
than influencing the total number of enterprises.
The evidence overall therefore is that most developed countries have
sought to reduce the bureaucratic burdens and costs of starting a business.
Where this serves to generate competition and to reduce corruption it is clearly
highly desirable. However in making these changes politicians have to be aware
of the importance of adequately protecting the consumer. They also have to
recognise that such actions may have their prime impact upon converting
businesses from the informal to the formal sectors, rather than adding totally
new businesses to the economy.
13
Again, business licensing is not normally the responsibility of the
enterprise department. It is even less likely that the SME Department will be
responsible for enterprise compliance with environmental or employment
legislation. Understandably the prime focus of the departments of government
that are responsible for this legislation is to introduce and enforce it. Yet, the
consistent finding is that unit compliance costs are much higher for firms with
less than 20 workers (OECD, 2005) than for larger firms, and that the average
time spent by small businesses normally the owner on administration and
87
compliance activities is 16 hours per week
14
and considerably more than the
top person in a larger firm. Yet this is of secondary significance to those
departments of government charged with the responsibility to enforce
legislation. Indeed smaller firms, partly because of their higher compliance
costs, are likely to be a prime target for the enforcing departments.
Public policy in this area therefore has to balance the interests of three
parties: consumer, (small) business and society as a whole. There are
occasions when each of the three parties views themselves as disadvantaged,
but the minimum condition for addressing these issues is the presence of a
forum at the top level in government at which such matters are debated. The
critical lesson from this policy area is that evaluation has to seek to highlight
evidence leading to more informed decisions. What is clear is that, whilst
business de-regulation is favoured by the business community, the magnitude
of the overall economic benefits is still being questioned by some scholars.
Immigration/emigration policies
The movement of people across national borders has major influences
upon entrepreneurship in both host and donor countries. The impact upon
the host countries has been more extensively studied than the impact upon
the donor countries.
For centuries, people have travelled across national borders to start
businesses. Often the motivation was negative, in the sense of escaping from
religious, or other forms of, persecution.
Whilst it is not always the case, there are many instances where immigrants
are a self-selected sample of enterprising individuals prepared to leave their
homeland to seek an improved life for themselves and sometimes their
families. New immigrants often enter self-employment because of the
difficulties of obtaining, and low pay in, waged employment. Andersson and
Wadensjo (2004), provide evidence of this wage gap between immigrants and
natives, even taking account of human capital differences, for both Sweden
and Denmark although their result was particularly strong for Sweden.
Todays migrants, as has been the case throughout history, are a mixture
of the highly skilled and the unskilled. But, for the skilled, what is clear is that
many have channelled their entrepreneurial energies into starting high tech
enterprises, which draw upon the skills these individuals have acquired from
education. A study by Saxenian (1999) shows that about one third of the high
skilled workforce in Silicon Valley, California, USA was born in either India or
China, and came to the area initially to study. She says:
In 1998 Chinese and Indian engineers were senior executives at one quarter of
Silicon Valleys new technology businesses. These immigrant-run companies
collectively accounted for more than USD 16.8 billion in sales and 58 282 jobs
88
in 1998. Moreover Chinese and Indian immigrants started companies at an
increasing rate in the 1990s.
It is likely that similar economic activity is currently taking place in the
regions around many other large universities, perhaps more actually in the
English speaking world.
This emphasises yet again, that there are a wide range of public policies
this time education and immigration that play a key role in influencing the
scale and nature of entrepreneurship in a country. Again this causes serious
problems for departments of government responsible for delivering the
entrepreneurship agenda, but where the policy instruments are under the
control of another department.
As noted earlier there may also be benefits from entrepreneurship to the
donor countries. Those leaving will often either send home remittances to
their family or may return home to establish a business after a period abroad.
McCormick and Wahba (2001) show that amongst literate Egyptians, business
ownership upon return is more likely amongst those with savings
accumulated overseas and those who have spent a longer period abroad. On
the basis of their evidence, they argue that losses to the donor countries of
enterprising individuals who migrate to work in more prosperous economies
are perhaps only temporary. Indeed, it is the overseas work experience which
provides individuals with a range of skills that they would never have
accumulated by staying in their home country. It should be noted that their
results do not apply to individuals without reading or writing skills.
This finding explains the active campaigns run by governments seeking
to persuade those who have emigrated to return home, often with the
intention of starting a business. High profile campaigns have, for example,
been run in countries such as Scotland and New Zealand. What is less clear is
the performance of those businesses started by returnees. In the absence of
any carefully collected evidence it seems likely that many will be life-style
returners since, if they were highly successful in their host country, it is much
more likely that they would have stayed there. However that evidence has yet
to be presented.
In lower income countries remittances can play an important role in
supporting home businesses. This appears to be confirmed in a study of
remittances in Moldova. Here Ghencea and Gudumac (2004) find that, whilst
the number of business activities practiced by the migrant families has not
increased, their scale has risen. They are larger and more profitable than
before the migration took place.
Overall it is clear that the migration of individuals across national borders
is, and always has been, a major source of entrepreneurship in the host
countries. Recent research suggests it may also exert a positive influence upon
89
entrepreneurship and economic development in the donor countries, but this
is more ambiguous.
However, whilst immigration generally exerts a positive influence on
entrepreneurship in the host countries, decisions made on immigration take a
wide range of, frequently non-economic, factors into account. This often
means that immigration rates are lower and much more selective than if only
issues of entrepreneurship were taken into account.
Competition and public procurement policies
Governments can also exercise a very powerful direct and indirect
influence upon SMEs through their competition and markets policies. We now
discuss each of these in turn.
Most governments have policies to ensure that markets are competitive.
Competition, however, must be fair
15
and not be an abuse of market power.
Given the nature of SMEs, it is much more likely that they will be the abused
party than is the case for larger firms. In this sense SMEs might seem to be the
obvious beneficiary from policies intended to ensure markets are fairly
competitive.
In practice, however, many SME owners are unlikely to be easily
persuaded that competition policy is in their best interests for the following
reasons:
Many recognise that enhancing competition could undermine the viability
of their own business. Instead they may view the objective of competition
policy as seeking to improve benefits for consumers rather than for
producers.
Many SME owners are fundamentally suspicious of all aspects of
government, and will view any policy changes as yet another burden.
Many SME owners, even if affected by unfair competition, will be unaware
of their rights of appeal or may be aware of their rights but cannot afford the
costs or the time out of the business to pursue their claim.
Many SME owners will not be able to distinguish between competition
which is fair, but damaging to them, and that which is unfair.
The crucial word in the above paragraph is many because it emphasises
the difficulty of making generalisations about the heterogeneous SME sector.
The evidence
16
shows that SME owners who are educated, well informed and
running growing businesses in modern sectors are likely to respond to
issues of fair and unfair competition in a different way from owners without
these characteristics. The former are more likely to be competitive, to be able
to distinguish between fair and unfair competition and to be prepared to take
action where that competition is thought by them to be unfair. Those without
90
these characteristics are much more likely to ignore such practices and accept
them as part of the cut and thrust of business life.
However it is not only by examining the competitive framework in which
SMEs themselves operate that competition policy can provide benefits for
smaller firms. Instead, SMEs also need to be reassured that there is
competition amongst their suppliers.
In this context, a key role for government is to ensure that the market for
SME finance is competitive. The UK government, for example, has been
particularly concerned to ensure to monitor and review the marketplace for
the provision of banking services for SMEs. This is because the market is
dominated by four large financial institutions whose market share for some
services, and in some geographical areas, can be close to 90%. In that
situation, there could be collusion to drive up prices and raise profit levels.
Whatever the merits of the case made by the financial institutions, it is
undoubtedly the case is that government policy exerts a powerful influence to
ensure that SMEs benefit from competition policy in the market for finance.
Finally, governments themselves are a major market for some SMEs and
could be a major market for many more. Morand (2003) states that, whilst
SMEs provide about 65% of private sector sales in the EU, their share of public
contracts won directly (excluding sub-contracting) is less than 25%.
It can be argued that this imbalance is a natural outcome of the
marketplace. Hence the reasons for SMEs low share are firstly that many
projects are too big for SMEs to bid for. Second that, since governments place
a greater premium on reliability than (some parts of) the private sector, this
also places new and small firms at a disadvantage. Thirdly, the tendering
procedure reflects the objective of the government to minimise any risk to
itself. By imposing high, but almost fixed, application costs this provides a
competitive advantage to large firms. This is because they are more likely to be
able to spread the fixed costs over many applications an option which is less
likely to be available to the small firm.
The effect is that it is more risky for governments to commission SMEs for
public sector work. To address this, government seeks to provide itself with
safeguards, such as firms having to register and be vetted, yet it is these
safeguards that disadvantage the SME. A decision therefore has to be made,
taking account of the interests of the taxpayer, whether to seek to increase the
role of SMEs in public procurement. To achieve this would require a sharp fall
in application costs for SMEs and recognition that this may increase the risk
that some contractors will fail to deliver and others may be dishonest. The
extent of that risk is difficult to determine in advance.
After undertaking such reviews, governments in many countries have
sought to ease the barriers which SMEs experience in bidding for government
91
contracts. To achieve this, a number of policies have been implemented. First,
countries such as Canada, India, and South Africa have targets or set asides
by which a proportion of public expenditure is delivered by SMEs. Alternative
approaches include pressures on public agencies to ensure that public
programmes are delivered in smaller chunks to enable SMEs to bid for them
more easily. Thirdly, governments have sought to reduce both the complexity
of the application forms and to ensure that similar information only needs
to be logged once with government, irrespective of the department
commissioning the work. All these initiatives, whilst seemingly mundane, do
make a real difference to the ability of the SME to make informed tenders for
government work and seek to reduce their disadvantage relative to larger
firms.
Overall therefore governments, through their competition and markets
role, can be strong influences upon the performance of the SME sector.
Capturing the total policy package
There are numerous macroeconomic policies that can influence dimensions
of entrepreneurship. Van Stel et al. (2006) take one entrepreneurship policy
objective to be the conversion of nascent to actual entrepreneurs. Using GEM
data, their conversion rates are shown in Table 5.4. Amongst OECD countries,
those with high rates are Sweden, Korea, and Netherlands. Those with low
conversion rates are France, Belgium, Germany and the United States. In
seeking to explain these country differences they confirm the earlier findings
on the influence of social security expenditure, but find limited support for the
business licensing argument.
From the above it is clear there are numerous influences upon
entrepreneurship and small business performance in a country. It is also clear
that decisions on taxation, immigration, legislative burdens and enforcement,
that strongly influence SMEs, are normally made by departments of
government other than the main SME department. Furthermore, the main
SME department of government almost certainly exerts only a very modest
influence upon these decisions. A key challenge for SME policy makers is
therefore to identify the most important of these policy influences on
entrepreneurship and then to influence key decision-making departments
elsewhere in government.
An interesting example of this broader approach to policy formulation
comes from Denmark. Table 5.5 identifies 16 policy areas that influence SME
performance, and challenge is for Denmark to decide where it should focus its
efforts.
The Danish policy target is that by 2015 Denmark should be one of the top
three countries in the world in terms of both start up rates and proportion of
92
Table 5.4. Average conversion rates young businesses/nascent
entrepreneurs, 2000-2004
Mexico 0.35 (3) Finland 0.74 (6)
Poland 0.37 (4) Singapore 0.75 (9)
Slovenia 0.38 (5) Russia 0.77 (3)
Peru* 0.41 (1) Jordan* 0.80 (1)
Croatia 0.44 (5) Portugal 0.81 (2)
South Africa 0.45 (7) Switzerland 0.82 (3)
France 0.47 (9) Spain 0.82 (9)
Venezuela* 0.51 (1) Iceland 0.83 (5)
Germany 0.56 (9) Greece 0.85 (3)
Belgium 0.57 (9) Hong Kong 0.87 (5)
Argentina 0.57 (9) Australia 0.89 (9)
United States 0.58 (9) Denmark 0.89 (9)
Hungary 0.58 (4) United Kingdom 0.89 (9)
Japan 0.59 (6) India 0.90 (5)
Italy 0.60 (7) Netherlands 0.98 (7)
Chile 0.64 (3) Brazil 1.03 (9)
Canada 0.64 (9) Uganda 1.18 (3)
Ecuador* 0.66 (1) Sweden 1.20 (7)
Ireland 0.69 (7) Korea 1.23 (5)
New Zealand 0.70 (7) China 1.43 (3)
Norway 0.72 (9) Israel # 2.05 (6)
Thailand* 0.72 (1) Taiwan* 2.41 (1)
Notes: Average of current conversion rate (average 2000-2004) and lagged conversion rate
(average 2001-2004). In case of missing observations the average conversion rates are computed over
the available years. The number of observations on which the statistic is based is given in
parentheses. For the countries marked with:
* The current conversion rates shown are actually computed for just one year.
# The relatively high conversion rate for Israel is mainly due to a very low nascent rate in 2001.
Table 5.5. Selecting policy areas
Low correlation with
performance
Significantly correlated with
performance
High Priority among Top 3 Corporate Tax
Bankruptcy Legislation
Wealth and Bequest Tax
Administrative Burdens
Entrepreneurship Education
Restart possibilities
Personal Income Tax
Venture Capital
Labour Market Regulation
Access to Foreign Markets
Low Priority among Top 3 Exit
Capital Taxes
Government Programmes
Traditional business education
Loan capital
Technology Transfer
Culture/Motivation
Entry Barriers
93
SMEs experiencing rapid growth. Its intermediate target is to become one of
the European elite, according to these criteria, by 2010.
The analysis on which Table 5.5 is based takes data for 20 OECD countries
on start up and growth, and examines the extent to which this links to the
policy areas in the Table. A total of 61 measures are identified for the 16 policy
areas and these are set out in full in Appendix E where that link is significant
it is shown in bold in the final column.
The rows of Table 5.5 show those policy areas where the top three
countries currently (USA, Canada and Korea) which Denmark aims to join
by 2015 place their policy emphasis. By implication these are the policy areas
where Denmark will seek to place more (or less) emphasis in its efforts to
match these countries. Based on statistical analysis, the columns of Table 5.5
show the links between policy activity and measures of entrepreneurship. So,
for example, bankruptcy legislation or wealth taxes in the left hand columns
are not strongly related to entrepreneurial performance. In contrast,
entrepreneurship education or programmes to support access to foreign
markets are characteristic of entrepreneurial economies.
The key box in Table 5.5 is then the one in the top right, since these
policies are both significantly correlated with performance and also where the
top three countries currently place more policy emphasis than Denmark. It
implies that Denmark, if it wishes to achieve its 2015 ambition, would be
expected to focus more heavily on entrepreneurship education, restart
possibilities and the lowering of personal income tax.
The value of the Denmark analysis is that it takes a set of policy priorities
and then conducts an analysis upon its competitors, in terms of those policy
areas. From this emerge a set of policies that appear to explain good
performance, but where Denmark devotes less policy attention than the
exemplars. Using this approach the coherence of the total policy can be
reviewed.
Of course such an approach does not solve all problems, or the need for
further political discussion and debate. What it does do is provide the basis of
a better informed debate. An example might be over such issues as focussing
upon entrepreneurship education. Those favouring such policies will point
to Denmark being deficient in comparison with the high performing
entrepreneurial countries. Those who do not favour such a focus would point
out that entrepreneurial education programmes cannot be expected to impact
upon entrepreneurship in the short run. Hence they are likely to fail to deliver
results by the target date of 2015. A separate debate might also be about
whether the lowering of taxes across the board is politically acceptable. So,
whilst it does not lead directly to problem solution it frames, in an informed
manner, the debate about options in SME and entrepreneurship policy.
94
Conclusion
This section has examined the macroeconomic context for SME and
entrepreneurship policies. It concludes that:
Total public funds devoted to encouraging entrepreneurship and supporting
SMEs are likely to be substantial in many countries. The total sum is rarely
quantified but, where it has been in the UK, it is approximately equivalent
to total spending on the police force.
Control, or influence, over that total expenditure is rarely exercised by the
department of government responsible for SME policy. Instead, other
departments or organisations of government often have considerably larger
budgets, but may have very different priorities from that of the main SME
department. Hence there is a strong case for considering a cross-government
group to decide spending priorities on SME and entrepreneurship policy.
Whilst SME and entrepreneurship policies are important in influencing the
creation of new firms and the development of SMEs, so also are other
government policies which do not have an explicit SME/entrepreneurship
focus. These include: macro policies to create a stable economic
environment, with low inflation, interest rates and unemployment;
taxation; unemployment benefits; business regulation policies (licensing
policies); immigration/emigration policies; competition and public policies.
The evidence reviewed points to high interest rates and macro economic
uncertainty depressing rates of new firm formation. The impact of
unemployment has a more mixed effect, with some people being
stimulated, and others inhibited, from starting new businesses.
The evidence on the impact of the tax regime on entrepreneurship is perhaps
even less clear. Early work asserted that high taxes depressed enterprise on
the grounds that business owners would be less prepared to work harder
and longer if they were subject to high taxes. Recent research is more
ambiguous since business ownership becomes more attractive under a high
tax regime since it provides opportunities for tax evasion and/or avoidance.
High unemployment benefits appear to be associated with low rates of new
firm formation.
Reducing the time and cost to start a new business has been the focus of
policy in several countries. The evidence is that lowering these barriers
does lead to more new businesses being registered, but how many of these
are informal businesses that convert to being formal is less clear.
There is no doubt that many groups of immigrants to a country make a major
contribution to enterprise in the host country. There is also evidence that
educated returning migrants can play an important role in enterprise
creation.
95
Finally competition and public procurement policies can also powerfully
influence enterprise creation and development.
The challenge for SME/entrepreneurship policy makers is therefore to
identify these macro policies and their links to enterprise. It is then to
ensure that macro policies which do not have enterprise as their central
aim nevertheless work in a way which is congruent with the objectives of
enterprise support. The current approach of the Danish government can be
considered to be an example of this.
The Framework has emphasised the importance of factors such as the tax
regime, the regulatory environment, immigration and emigration as well as
macro policies seeking to achieve low inflation, interest rate and unemployment.
These policies, perhaps significantly more than specific SME and
entrepreneurship policies, influence rates of new firm formation and the
growth of the SME sector in a region or country.
The policy implication of this are that the main department responsible for
SME policy has to work closely in conjunction with other departments of
government responsible for matters relating to economic policy, the tax regime,
or immigration/emigration. It is also recognised that these departments of
government are likely to have expenditures relating to SMEs that dwarf the
main SME department. Hence the development of strong relationships to
ensure policy coherence is a pre-condition for successful policy delivery.
Overall the evidence presented in Section 5 has demonstrated that the
evaluation techniques appropriate for assessing the impact on SMEs of
macro policies differs radically from that for SME policies and programmes
discussed in Section 2-4. Evaluation of macro-policies such as competition
policy, immigration policy, licensing policy as well as traditional macro-
policies such as the control of inflation and aggregate demand, rely heavily
upon the availability of long established time series information and the
ability to make valid international comparisons. Here the OECD is well
positioned to play a key role in the future in this form of evaluation.
Notes
1. Cross Cutting Review of Government Services for Small Businesses, DTI, London,
September 2002.
2. Responsibility for enterprise policy now resides with the Enterprise Directorate
which is part of the Department of Business, Enterprise and Regulatory Reform.
3. The difference is explained by Shane using the change in the ratio of businesses to
the population for each year, rather than the ratio itself. It emphasises the possible
sensitivity of such findings to the choice of dependent variable.
4. He was significantly less persuaded that the other two elements of what he called
the Washington consensus privatisation and trade liberalisation had made
96
much contribution to the economic development of low income countries. These
comments are reported in an interview with Multinational Monitor, April 2000
(http://multinationalmonitor.org/mm2000/00april/interview.html).
5. To develop this point slightly: high tax societies are also likely to have high
exemptions. Hence in a high tax society the individual knows that in the event
of a business failure s/he will be able to write off any losses against either current
or future tax. This may make them more willing to take risks, in the knowledge
that their personal downside losses will be minimal.
6. Bjuggren and Sund (2005), for example estimate that a lifetime succession plan
would cost between 50 000 and 500 000 euros.
7. A classic example is reported in Storey (1994). He reports on the work of Rees and
Shah (1994) who found that whilst income tax rates fell sharply following the
election of a Thatcher government in Britain in 1979, this was associated with the
self employed working fewer hours the opposite to that which Thatchers aides
had forecast. The reason was that most self-employed individuals had a target
level of after-tax pay. Since they achieved this target when taxes were lower by
working fewer hours, the outcome should have been entirely predictable.
8. www.doingbusiness.org/economyrankings/.
9. For example the 2005 OECD SME and Entrepreneurship Outlook, page 222-3 reports
that, in 2004 there were 225 000 new start ups in France. This is the highest number
in history and compares with an average of 175 000 in the late 1990s. The sharp rise
follows the Economic Initiative Act of 2003, one key theme of which is making
enterprise creation fast, simple and available to all. A second example is provided
by Haggarty et al. (2006) who reports that, following a major simplification of
business licensing in Lima, Peru, in the first four months of operation the
municipality has licensed nearly as many firms as in the previous two years.
10. The much quoted study of this type of procedures is by De Soto (1990).
11. The new firms examined in the comparison were created and identified before the
change in the Spanish legislation.
12. The reason we would expect the registered Spanish firms, on average, to start
larger than the English is because licensing is an (almost) fixed cost. So, if the cost
of licensing is high and the sales of the business are expected to be small then
registration is unlikely, and it is the elimination of the very small firm firms that
raises the average size of the registered Spanish firms.
13. It might be argued that the willing transfer of businesses from the informal to the
formal sector itself has economic benefits. For example, it may mean that an
entrepreneur who previously ran a business in the informal sector is no longer
apprehensive of expanding for fear of being discovered by the relevant authorities.
This expansion could provide considerable economic benefits, but it is difficult to
quantify its significance.
14. This is taken from the Australian Small Business De-Regulation task force and is
reported in the 2005 OECD SME and Entrepreneurship Outlook.
15. See Vickers (2005).
16. Storey (2005) shows that although one third of small firms are aware of anti-
competitive practices and one quarter of SMEs have experienced such practices
themselves, their most frequent reaction is to ignore it. Those SME owners most
likely to report it is an educated male networker owning a somewhat larger
business outside the traditional sectors.
97
References
Andersson, P. and E. Wadensjo (2004), Self Employed Immigrants in Denmark and Sweden:
A Way to Economic Self Reliance?, Discussion Paper No. 1130, IZA Bonn.
ARK 19 (2005), Tyne and Wear Start-up Support (Small Business Grant) Project Evaluation,
April.
Arup Economics and Planning (2000), Evaluation of Regional Selective Assistance 1991-1995:
Final Report Arup Economics and Planning, London.
Bakouras, Y.L., Mardas, D.C. and Varsakelis, N.V. (2002), Science Parks, a high-tech
fantasy? An analysis of the Science Parks of Greece, Technovation, Vol. 22[2],
pp. 123-128.
Barclays Bank (2001), Profiting from Support, London.
Baumol, W.J. (1990), Entrepreneurship: Productive, Unproductive and Destructive,
Journal of Political Economy, Vol. 98, No. 5, pp. 893-921.
Bjuggren, P.-O. and R.-G. Sund (2005), Organisation of Transfers of Small and Medium
Sized Enterprises within the Family: Tax Law Considerations, Family Business
Review, Vol. 18, No. 4, pp. 305-320.
Bondonio, D. and J. Engberg (2000), Enterprise Zones and Local Employment: Evidence
from the States Programmes, Regional Science and Urban Economics, Vol. 30, 519-549.
Bondonio, D. and R. Greenbaum (2006), Do business investment incentives promote
employment in declining areas? Evidence from EU Objective 2 regions, European
Urban and Regional Studies, Vol. 13.3, 225-244.
Bonner, K. and S. McGuiness (2007), Assessing the Impact of Marketing Assistance on
the Export Performance of Northern Ireland SMEs, International Review of Applied
Economics, Vol. 21, No. 3, pp. 361-379.
Boyns, N., Cox, M., Spires, R. and Hughes, A. (2003), Research into the Enterprise
Investment Scheme and Venture Capital Trusts. PACEC Report for Inland Revenue.
Bruce, D. (2000), Effect of the US Tax System on transitions into Self Employment,
Labour Economics, Vol. 7, Issue 5, pp. 545-574.
Canada Economic Development for Quebec Regions (2003), Evaluation of ORPEX
Funding Programmes, Canada Economic Development for Quebec Regions.
Capelleras, J.-L., K. Mole, F.G. Greene and D.J. Storey (2008), Do more Heavily regulated
Economies have poorer performing new ventures: Evidence from Britain and
Spain?, Journal of International Business Studies [forthcoming].
Crac Inc. (2000), Evaluation of the Rural Enterprise Program. A Component of the Bas
Satin Laurent and Gaspsie/Ile de la Madeleine Regional Strategy.
Charles, D., C. Nauwelaers, B. Mouton and D. Bradley (2000), Assessment of the Regional
Innovation and Technology Transfer Strategies and Infrastructures (RITTS) Scheme, Final
Evaluation Report. Centre for Urban and Regional Development Studies,
Newcastle-upon-Tyne.
Chen, D., F. C. Lee and J. Mintz (2002), Taxation, SMEs and Entrepreneurship, OECD
Science, Technology and Industry Working Papers, 2002/9, OECD Publishing,
doi: 10.1787/013245868670.
Confederation of British Industry (1999), The Enterprise Investment Scheme: Why Investors
and Companies do not use the Scheme, London.
98
Cooper, R.S. (2003), Purpose and Performance of the Small Business Innovation and
Research (SBIR) Program, Small Business Economics, Vol. 20, No. 2, pp. 137-151.
Cosh, A., A. Hughes and D.J. Storey (2006), SME Happiness under the Tories and
Labour, presented at CBR Summit on Innovation and Governance, University of
Cambridge, 29-30 March.
Cullen, J.B. and R.H. Gordon (2002), Taxes and Entrepreneurial Activity: Theory and
Evidence from the United States, National Bureau of Economic Research, Working
Paper, 9015.
Davis, N. (2003), BIZ Investment Ready Scheme, Ministry of Economic Development.
Davis, S.J. and M. Henrekson (1999), Explaining National Differences in the Size and
Industry distribution of Employment, Small Business Economics, Vol. 12, No. 2,
pp. 59-83.
Del Monte, A. and D. Scalera (2001), The Life Duration of Small Firms Born within a
Start-up Programme: Evidence from Italy, Regional Studies, Vol. 35, No. 1, pp. 11-21.
De Soto, H. (1990), The Other Path, Harper and Row, New York.
Department of Trade and Industry, Cross Cutting Review of Government Services for Small
Businesses, September 2002, URN 02/1324, London.
Djankov, S., R. La Porta, F. Lopez-de-Silanes and A. Schleifer (2002), The Regulation of
Entry, Quarterly Journal of Economics, Vol. CXVII, February, No. 1, pp. 1-37.
DTZ Consulting & Research (2007), Interim Evaluation of the SWMAS Strategic
Management Programme, South West of England Regional Development Agency.
EIM (2004), Review of methodologies to measure the effectiveness of state aid to SMEs, EIM,
Netherlands.
Folster, S. (2002), Do Lower taxes stimulate self-employment?, Small Business
Ghencea, B. and I. Gudumac (2004), Labour Migration and Remittances in the Republic of
Moldova, Soros Institute, Moldova.
Haggarty, L., Rada, K. and Dohm, E. (2006), Cutting Red Tape Simplifying Regulation at
the Municipal Level in Latin America: Lessons from IFCs Experience in Peru and
Nicaragua, International Finance Corporation.
Heckman, J.J., H. Ichimura, and P. Todd (1997), Matching as an Econometric Evaluation
Estimator: Evidence from Evaluating a Job Training Programme, Review of Economic
Studies, 64(4), pp. 605-654.
Heckman, J.J. (2001), Micro Data, Heterogeneity, and the Evaluation of Public Policy: Nobel
Lecture, Journal of Political Economy, August, Vol. 109, Issue 4, pp. 673-748.
Lambrecht, J. and F. Pirnay (2005), An evaluation of public support measures for
private external consultancies to SMEs in the Walloon Region of Belgium,
Entrepreneurship and Regional Development, Vol. 17, No. 2, pp. 89-108.
Lenihan, H. (1999), An evaluation of a regional development agencys grants in terms
of deadweight and displacement, Environment and Planning C: Government and
Policy, 17.3, pp. 303-318.
Lenihan, H., M. Hart and S. Roper (2007), Industrial Policy Evaluation: Theoretical
Foundations and Empirical Innovations: New Wine in New Bottles, International
Review of Applied Economics, Vol. 21, No. 3, pp. 313-319.
99
Lerner, J. (1999), The Government as Venture Capitalist: The Long-Run Impact of the
SBIR Program, Journal of Business, 1999, Vol. 72, No. 3, pp. 285-318.
Lindelof, P. and Loftsen, H. (2003), Science Park Location and New Technology Based
Firms in Sweden implications for Strategy and Performance, Small Business
Economics, Vol. 20 [3], pp. 245-258.
Lundstrom, A. (1999), Interprise DECIDE V Expectations, experiences and results,
Swedish Foundation for Small Business Research, 1999:6.
Marshall, J.N., N. Alderman, C. Wong and A. Thwaites (1993), The impact of
government-assisted management training and development on small and
medium-sized enterprises in Britain, Environment and Planning: Government and
Policy, Vol. 11, 1993, pp. 331-348.
Maton, K. (1999), Evaluation of Small Firms Training Loans, DfEE Research Report No. 97,
January.
McCormick, B. and Wahba (2001), Overseas Work Experience, Savings and
Entrepreneurship amongst return migrants to LDCs, Scottish Journal of Political
Economy, Vol. 48, No. 2, pp. 164-178.
McMullan, E., J.J. Chrisman and K. Vesper (2001), Some Problems in Using Subjective
Measures of Effectiveness to Evaluate Entrepreneurial Assistance Programs,
Entrepreneurship Theory and Practice, Vol. 26, No. 1, pp. 37-54.
Meager, N., P. Bates and M. Cowling (2003), An Evaluation of Business Start-up
Support for Young People (2003), National Institute Economic Review, No. 186,
October 2003, pp. 70-83.
Michaelas, N., F. Chittenden and P. Poutziouris (1999), Financial Policy and Capital
structure Choice in UK SMEs: Empirical Evidence from Company Panel Data, Small
Business Economics, Vol. 12, No. 2, pp. 113-130.
Ministry of Economic Development (2004), Review of the Polytechnic Regional Development
Fund, New Zealand.
Morand, P.-H. (2003), SMEs and Public Procurement Policy, Review of Economic Design,
Vol. 8, No. 3, pp. 301-318.
National Audit Office (2006), Supporting Small Business, London, 19 May.
Nickell, S. (1997), Unemployment and Labour Market Rigidities: Europe verses North
America, Journal of Economic Perspectives, Vol. 11, No. 3, pp. 55-74.
Nitani, M. and Riding, A. (2005), Promoting Enterprise Development or Subsidising
Tradition: The Japanese Credit Supplementation Scheme, International Small
Business Journal, Vol. 23[1], pp. 48-71.
OECD (2003), Peer Review: An OECD Tool for Co-operation and Change, OECD, Paris.
OECD (2004a), Evaluation of SME Policies and Programmes, Final Report prepared for the
2nd OECD, Ministerial Conference on SMEs on Promoting Entrepreneurship and
Innovative SMEs in a Global Economy Towards a More Responsible and Inclusive
Globalisation, OECD, Paris.
OECD (2004b), Small- and Medium-Sized Enterprises in Turkey: Issues and Policies, OECD,
Paris.
OECD (2005), Small and Medium Enterprise and Entrepreneurship Outlook, OECD, Paris.
OECD (2007), SMEs in Mexico, Issues and Policies, OECD, Paris.
100
PA Cambridge Economic Consultants (1995), Final Evaluation of Enterprise Zones.
HMSO, London.
PACEC (2001), Evaluation of SMART (including SPUR).
PACEC (2003), Evaluation of the RIF-funded Business to Business Project.
PACEC (2005), Mapping of Government Services for Small Businesses. A report for the
Small Business Service by Public and Corporate Economic Consultants.
Department for Business, Enterprise and Regulatory Reform: London.
Papaconstantinou, G. and Polt, W. (1997), Policy Evaluation in Innovation and Technology:
An Overview, OECD Proceedings, Policy Evaluation in Innovation and Technology
Towards Best Practices, OECD, Paris.
Papke, L. (1994), Tax policy and urban development: Evidence from the Indiana
enterprise zone program, Journal of Public Economics, Vol. 54, 37-49.
Parker, S.C. (1996), A Time Series Model of Self Employment Under Uncertainty,
Economica, Vol. 63, No. 251, pp. 459-475.
Parker, S.C. and M.T. Robson (2004), Explaining International Variations in Self-
Employment: Evidence from a Panel of OECD Countries, Southern Economic Journal,
Vol. 71, No. 2, pp. 287-301.
Ramsdem, P. (2005), Evaluation: The Phoenix Development Fund, Freiss Ltd.
Ramsey, E. and D. Bond (2007), Evaluating Public Policy Formations and Support
Mechanisms for Technological Innovation, International Review of Applied
Rees, H. and A. Shah (1994), The Characteristics of the Self-Employed: The Supply of
Labour, in J. Atkinson and D.J. Storey (eds.), Employment, the Small Firm and the
Labour Market, Routledge, London.
Riding, A.L. and Haines, G. (2001), Loan Guarantee: Cost of Default and Benefit to
Small Firms, Journal of Business Venturing, Vol. 16[6], pp. 595-612.
Robson, M. and C.M. Wren (1999), Marginal and Average Tax Rates and the Incentive
for Self-Employment, Southern Economic Journal, Vol. 65, No. 4, pp. 757-773.
Robson, P.J.A. and R.J. Bennett (2000), The use and impact of business advice by SMEs
in Britain: an empirical assessment using logit and ordered logit models, Applied
Economics, Vol. 32, pp. 1675-1688.
Roper, S. and N. Hewitt-Dundas (2001), Grant Assistance and Small Firm
Development in Northern Ireland and the Republic of Ireland, Scottish Journal of
Political Economy, Vol. 48, No. 1, February 2001, pp. 99-117.
Saxenian, A. (1999), Silicon Valleys New Immigrant Entrepreneurs, Public Policy
Institute, San Francisco.
Schuetze, H.J. (2000), Taxes, Economic Conditions and recent trends in male self-
employment: a Canada US comparison, Labour Economics, Vol. 7, Issue 5, pp. 507-544.
Schuetze, H.J. and D. Bruce (2004), Tax Policy and Entrepreneurship, Paper presented at
conference on Self-Employment, March 22, Stockholm Economic Council for
Sweden.
Shane, S. (1996), Explaining Variation in Rates of Entrepreneurship in the United
States: 1899-1988, Journal of Management. Vol. 22, No. 5, pp. 747-781.
101
Sheikh, S. and Steiber, N. (2002), Evaluating Actions and Measures Promoting Female
Entrepreneurship A Guide, Austrian Institute for Small Business Research (IfGH) and
European Commission Enterprise Directorate General.
Small Business Service (2002), Cross Cutting Review of Government Services for Small
Businesses, DTI, London.
Smallbone, D., D. North and R. Leigh (1993), The use of external assistance by mature
SMEs in the UK: some policy implications, Entrepreneurship and Regional
Development, Vol. 5, pp. 279-295.
Smith, J. (2000), A Critical Survey of Empirical Methods for Evaluating Active Labor
Market Policies, Zeitschriftfur Volkwirtschaft und Statistik, Vol. 136, No. 3, pp. 1-22.
SQW Ltd. (2005), Business Support in Northumberland: A Review. Final Draft Report to
Northumberland Strategic Partnership, Leeds.
Storey, D.J. (1994), Understanding the Small Business Sector, ITP, London.
Storey, D.J. (2000), Six Steps to Heaven: Evaluating the Impact of Public Policies to Support
Small Businesses in Developed Economies, in D.L. Sexton and H. Landstrom [eds.],
Handbook of Entrepreneurship, Blackwell, Oxford.
Storey, D.J. (2005), The Competitive Experience of UK SMEs: Fair and Unfair, Report to the
Competition Commission, London.
Surder, J.H., Gosh, D. and Rosa, P. (1997), The Importance of Support Services to Small
Enterprises in Bangladesh, Journal of Small Business Management, April, Vol. 37,
No. 2, pp. 26-36.
Turok, I., and Raco, M., (2000), Developing expertise in small- and medium-sized
enterprises: an evaluation of consultancy support, Environment and Planning C:
Government and Policy 18[4], pp. 409-427.
Van Stel et al., A. Van Stel, D.J. Storey and R. Thurik (2006), The effect of business
regulations on nascent and actual entrepreneurship, Small Business Economics
(forthcoming).
Vickers, J. (2005), Abuse of Market Power, Economic Journal, Vol. 115, June, pp. F244-F261.
Westhead, P. and M. Cowling (1998), Family Firm Research: The need for a
Methodological Re-Think, Entrepreneurship Theory and Practice, Vol. 23, Issue 1,
pp. 31-56.
Westhead, P., M. Cowling and D.J. Storey (1997), The Management and Performance of
Unquoted Family Companies in the UK, Stoy-Hayward Centre for Family Business,
London.
Westhead, P., Storey, D.J. and Martin, F. (2001), Outcomes reported by students who
participated in the 1994 Shell Technology Enterprise Programme, Entrepreneurship
and Regional Development, Vol. 13, No. 2, pp. 163-185.
Wren, C. and D.J. Storey (2002), Evaluating the effect of soft business support upon
small firm performance, Oxford Economic Papers, Vol. 54, pp. 334-365.
ISBN 978-92-64-04008-3
OECD 2007
103
APPENDIX A
The OECD Istanbul Position
This Framework document stems directly from the 2004 Istanbul
Ministerial Declaration, the outcome of the 2nd OECD SME Ministerial Conference
on Promoting Entrepreneurship and Innovative SMEs in a Global Economy,
Towards a More Responsible and Inclusive Globalisation organised in Istanbul
in June 2004. The Istanbul Ministerial Declaration recognised:
The need to develop a strong evaluation culture, in ministries and
agencies responsible for SME policies and programmes. Evaluation provides a
means of ensuring that SME programmes remain cost-effective and adapt to
changing conditions in a dynamic world. Ideally, evaluation would be
mandated and budgeted for when programmes are designed, would be carried
out by independent but informed evaluators, and would generate
recommendations for improving and strengthening those programmes that
should be retained.
The Ministerial Declaration then went on to invite the OECD to consider:
Proactively disseminating the work it has carried out on best practices for
the evaluation of SME policies and programmes, for example by working with
OECD members and interested non-member economies and organisations
to develop and test a handbook* of best practices for evaluation of SME
policies and programmes.
The conclusions of the technical report prepared for the Istanbul Ministerial
Conference on SMEs [OECD (2004a)] are reproduced below:
Evaluation cannot take place adequately until the objectives and targets of
SME policy are clearly specified. The paper suggests using the COTE
framework for policy formulation.
* In subsequent discussion the more generic term framework, rather than the more
prescriptive term handbook was thought to be appropriate. This is designed to
reflect the very different circumstances in which evaluation takes place in different
countries.
APPENDIX A
104
In this, the letter C stands for both policy clarity and policy coherence.
Clarity, in this context, means that the policy is fully understood by those
delivering it and those expected to benefit from it. Coherence, however, is
a slightly more sophisticated concept. It refers to the need to ensure that all
parts of government seek to collectively work in the interests of SMEs.
Specifically SME policies are unlikely to be coherent where they are
delivered by many different government departments with inadequate co-
ordination or where some departments focus upon regulation of SMEs and
others focus upon support, without an adequate dialogue between them.
The letter O stands for Objectives of policy. To ensure clarity the
objectives of policy must be clearly stated; in practice such objectives might
be to increase the number of firms, or to increase the formation rate of
firms, or to increase the value added of firms. It might also include seeking
to raise business ownership amongst groups defined in terms of ethnicity,
gender, disadvantage or location.
The letter T stands for Targets which need to be specified in a manner
that enables an assessment to take place of whether or not they are
achieved. An example of converting an objective into a target would be
the objective of to increase the formation rate of firms. Here the target
might be to raise new firm formation by 10% over the next three years.
Only then is it possible to be clear whether or not the target is met.
The letter E stands for Evaluation, which is the prime focus of this paper.
However the first key conclusion of the paper is that evaluation cannot be
undertaken unless targets exist. This is because evaluation can only take
place in a framework in which the expected policy impacts are clearly
specified.
A second key conclusion of the paper is that evaluation needs to become
more central to the policy-making process. It should not be undertaken
solely as a historic accounting exercise to determine whether public money
has been spent wisely, although that role is of value. Instead of being at the
end of the line evaluation should be used to inform current policy, so that
the objectives and targets may be modified in the light of evidence of policy
effectiveness. Considerations of how policy is evaluated should therefore be
incorporated into policy formulation when new ideas are being developed.
Ensuring that evaluation is given a higher priority means that all, rather
than merely some, programmes are assessed. The paper is however less
clear about the extent to which all programmes should be assessed with the
same level of sophistication. The key argument in favour of a consistent
level of sophistication is that this enables valid comparisons to be
undertaken between one programme and another. The argument against is
APPENDIX A
105
that it is inappropriate to have similar evaluation budgets for programmes
of widely differing scales.
Nevertheless the paper makes it clear that it is now technically possible to
provide accurate measures of programme impact. It shows, through its Six
Steps to Heaven approach that, although there are examples of the most
sophisticated approach being used, these are the exception rather than the
rule. The cause for concern is that the less sophisticated approaches tend to
provide misleading answers, most frequently over-estimating policy
impact. The third key conclusion of the paper is that, where appropriate,
the most sophisticated Step 5 or Step 6 approaches are to be used.
In undertaking these evaluations it is necessary to bear in mind some
potential conflicts. The first is that ideally those undertaking the
evaluations need to be independent of those responsible for the
programmes, as in any audit role. But, if a key role of evaluation is to
contribute to making continuous improvements in the policy, then the
evaluators need the active co-operation and involvement of both policy-
makers and deliverers. This may be more difficult to achieve if the evaluator
is viewed as an outsider. On balance, the paper concludes that the
independence of the evaluator is of the greater importance, but this is a
close call.
APPENDIX B
106
APPENDIX B
Six Steps to Heaven: Methods for Assessing
the Impact of SME Policy
The Six Steps procedure recognises that there is a gradation of
sophistication in the assessment procedures. The Six Steps show a gradation,
with Step 1 being the least sophisticated procedure and Step 6 being the most
sophisticated. Here sophistication is intended to reflect the confidence the
policy maker has in being able to attribute changes in businesses supported
under the programme often called the treatment group to participation
in the programme. Alternatively expressed, it may be considered as the
confidence the policy maker has that all other influences are held constant.
Table B.1 shows that a distinction is made between Monitoring and
Evaluation. Steps 1 to 3 are not considered to be Evaluation, but instead are
referred to as Monitoring. The difference between Monitoring and Evaluation
is that the former relies exclusively upon the views of the recipients of the
policy. Evaluation however seeks, by some means, to contrast these views or
actions with those of non-recipients in order to present the counter factual.
The difference between actual changes and the counter factual is viewed as
the impact of the policy or its additionality.
Table B.1. Six Steps to Heaven: Methods for assessing the impact
of SME policy
Monitoring
STEP 1 Take up of schemes
STEP 2 Recipients opinions
STEP 3 Recipients views of the difference made by the Assistance
Evaluation
STEP 4 Comparison of the Performance of Assisted with Typical firms
STEP 5 Comparison with Match firms
STEP 6 Taking account of selection bias
APPENDIX B
107
A full review of Steps 1 to 4 are provided in Storey (2000), but for current
purposes it is sufficient to note that, despite the frequency with which they
are used, these approaches either do not attempt to make a comparison with
a control group, or do so in a clearly inadequate manner. It is not until Step 5
that an attempt is made to compare treatment with non-treatment firms,
where the latter do not differ in terms of observable factors such as
ownership, size, sector and geography. This is normally undertaken through a
matching process. Once the matching is complete then differences in the
performance of the two groups is attributed to participation in the programme.
Whilst the Step 5 approach reflects a real attempt to address the issue of
the counter-factual, even it cannot be considered as best practice. The
results are still somewhat ambiguous because observed differences in
performance may also reflect the presence of un-observables. The most
obvious of these may be the dynamism or motivation of the SME owner, with
this leading to various forms of selection bias in the estimation techniques.
Two examples of selection bias are self-selection and committee
selection. Self selection bias may occur where, for example, programmes seek
to provide support for rapidly growing businesses. Here those businesses that
are seeking growth may be more likely to apply to become participants in the
programme than those with only modest or with no growth ambitions. The
problem arises where this is only partly reflected in observable factors, since
the latter under-estimate how the firm would have got on even if the
programme had not existed. The counter-factual is specified too low, and
the programme impact is consequently over-estimated. The clearest example
is where applicant firms, although apparently no different from non-applicant
firms in terms of the controls, in fact have more motivated owners. These
differences in owner motivation which is very difficult to observe mean
these firms would be expected to perform better than the matches, even if
there had been no programme. It is therefore unreasonable to attribute all
differences in performance between the programme firms and the matches to
programme participation. Failure to take account of selection, however, leads
to precisely this error.
A second example is committee selection. This refers to programmes
where only a proportion of applicants are successful. Here a committee or
similar group makes a judgement, with the better firms/applications
obtaining the funding. If the committee is effective it will eliminate those
firms likely to perform poorly, so that the programme participants are clearly
non-randomly drawn from applicants. As before, even if the programme did
not exist, the selected firms would be expected to have outperformed the
other firms. Hence observed differences in performance between programme
participants and matched firms cannot be attributed solely to programme
participation.
APPENDIX B
108
There are now a number of statistical techniques that address the issue
of selection bias with these being closely associated with James Heckman.
[Heckman et al. (1997), Heckman (2001)]. It is a curiosity that, although these
techniques are now well established in the evaluation of labour market and
welfare programmes, they are much less frequently used in estimating the
impact of business support. A helpful review of these techniques is provided
by Smith (2000), who identifies five statistical techniques to identify
programme impact when users differ from non-users. These begin with tests
to control for observable differences but then move on, using greater
sophistication, to account for both programme selection and self-selection. He
concludes that using some instrumental variables that predict programme
use, but are uncorrelated with the observable values that affect the economic
outcomes of the programme, is ideal [Angrist and Kruger (2001)]. The problem,
of course, is that good instruments are hard to find.
APPENDIX C
109
APPENDIX C
Examples of Evaluation Guidance
OECD MEMBER COUNTRIES
Canada: Canadian International Development Agency (CIDA) Evaluation Guide
evaluating development co-operation policies, programmes and projects
(www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Performancereview5/$file/
English_E_guide.pdf).
Denmark: Evaluation Guidelines, Ministry of Foreign Affairs, Danida, 1999
(www.um.dk/NR/rdonlyres/4BA486C7-994F-4C45-A084-085D42B0C70E/0/
Guidelines2006.pdf).
Japan: Framework of JICA Project Evaluation (www.jica.go.jp/english/evaluation/
guides/guideline.html) and (www.jica.go.jp/english/evaluation/guides/pdf/
guideline01-01.pdf).
Japan: ODA evaluation guidelines, Economic Cooperation Bureau, Ministry of
Foreign Affairs, Japan, 2003 (www.mofa.jp/policy/oda/evaluation/guideline.pdf).
Sweden: Swedish International Development Cooperation Agency (SIDA)
Looki ng Back, Movi ng Forward: SI DA Eval uati on Manual , 2004
(www.tsunami-evaluation.org/NR/rdonlyres/95FD78B5-DAC0-4339-AECC-
B25EFD284797/0/evaluation_manual_sida.pdf).
UK: DFID Department of International Development (www.dfid.gov.uk/
aboutdfid/performance/files/guidance-evaluation.pdf).
US: USAID Performance Management Toolkit (http://pdf.usaid.gov/pdf_docs/
PNACL702.pdf).
INTERNATIONAL ORGANISATIONS
European Commission: Evaluating European Union Activities (http://
ec.europa.eu/budget/documents/evaluation_en.htm#guides).
European Commission: Review of methods to measure the effectiveness of state
aid to SMEs (http://ec.europa.eu/enterprise/library/lib-competition/libr-
competition.html).
APPENDIX C
110
OECD: Development Assistance Committee, Evaluation Resource Centre
(www.oecd.org/document/63/0,2340,en_35038640_35039563_35067327_
1_1_1_1,00.html).
OECD: Development Assistance Committee Glossary of Key Terms in
Evaluation and Results Based Management, OECD, 2002 (www.oecd.org/
dataoecd/29/21/2754804.pdf) and Joint Evaluations (www.oecd.org/dataoecd/
28/9/35353699.pdf).
Organization for Security and Co-operation in Europe (OSCE): Forthcoming
publications on Systems of Indicators for Evaluation of State-supported
SME Development Programs and Monitoring and Evaluation Handbook.
UNDP/Global Environment Facility (GEF): Measuring and Demonstrating Impact
Resource Kit, 2005 (www.undp.org/gef/05/documents/me/UNDP_GEF_Measuring_
and_Demonstrating_Impact_Mar05.doc).
UNICEF: Standards (www.unicef.org/evaluation/files/Evaluation_standards.pdf) and
(www.unicef.org/evaluation/index_13486.html).
World Bank: Framework for Evaluating Framework the Impact of SME Programs
Kris Hallberg Operations Evaluation Department, World Bank (http://
info.worldbank.org/etools/docs/library/86503/Session%201%20Hallberg.pdf).
World Bank: Ten Steps to a Results-Based Monitoring and Evaluation System: a
Handbook for Development Practitioners, 2004 (www-wds.worldbank.org/
servlet/WDSContentServer/WDSP/IB/2004/08/27/000160016_20040827154900/
Rendered/PDF/296720PAPER0100steps.pdf).
OTHERS
Evaluate Europe: (www.evaluate-europe.net/handbook/) and (www.evaluate-
europe.net/tools/eval_of_elearning/sme_enviro/).
Nexus Associates: Framework for Evaluating the Impact of Small Enterprise
Initiatives (Eric Oldsman, Kris Hallberg), 2003 (www.enterprise-impact.org.uk/
pdf/FrameworkforEvaluatingImpact-Oldsman2003.pdf).
Performance Assessment Resource Centre (PARC): provides support services
on performance management combined with the development of an
information resource centre on performance assessment (www.parcinfo.org).
Enhancing Organizational Performance: a Toolbox for Self-assessment, 1999
(www.idrc.ca/en/ev-9370-201-1-DO_TOPIC.html).
APPENDIX D
111
APPENDIX D
Assessing the Quality of an Evaluation
Although the quality of an evaluation is often judged on the sole basis of
the final report, it largely depends on the way in which the entire evaluation
process was managed, from the commissioning of the evaluation to the
conduct of the work, the drawing of conclusions, the formulation of
recommendations and the communication of results. It is necessary to assess
the quality of an evaluation in order to verify whether the evaluation team
satisfied the commissioners requirements and to ensure that the evaluation
can stand up to criticism that may arise when judgements on success and
failure are made. In particular, it has to be decided whether the collection and
analysis of data is sufficient or must be improved, whether the conclusions
drawn from the evaluation are acceptable in view of the proof gathered, and
whether the report answers all the questions mentioned in the terms of
reference or whether further work needs to be under taken. A grid for
assessing the quality of an evaluation is provided in the following table.
APPENDIX D
112
Table D.1. Grid for a synthetic assessment of the quality of evaluation work
With regard to this criterion, the evaluation report is: Unacceptable Acceptable Good Excellent
1. Meeting needs: Does the evaluation adequately address
the requests for information formulated by the
commissioners and does it correspond to the terms of
reference?
2. Relevant scope: Have the rationale of the programme, its
outputs, results, impacts, interactions with other policies
and unexpected effects been carefully studied?
3. Defensible design: Is the design of the evaluation
appropriate and adequate for obtaining the results
(with their limits of validity) needed to answer the main
evaluative questions?
4. Reliable data: Are the primary and secondary data
collected or selected suitable? Are they sufficiently reliable
compared to the expected use?
5. Sound analysis: Are the quantitative and qualitative data
analysed in accordance with established rules and are they
complete and appropriate for answering the evaluative
questions correctly?
6. Credible results: Are the results logical and justified by the
analysis of data and by interpretations based on carefully
presented explanatory hypotheses?
7. Impartial conclusions: Are the conclusions just and
non-biased by personal or partisan considerations, and
are they detailed enough to be implemented concretely?
8. Clear report: Does the report describe the context and
goals, as well as the organisation and results of the
evaluated programme in such a way that the information
provided is easily understood?
In view of the contextual constraints bearing on the
evaluation, the evaluation report is considered to be
Source: European Commission (1999).
APPENDIX E
113
APPENDIX E
Framework Condition Indicators:
Entrepreneurship Conditions in Denmark
in 2005
The National Agency for Enterprise and Construction in Denmark
identified indicators of entrepreneurship conditions which were selected on
the following four grounds:
relevance;
accuracy;
availability; and
coherence.
The indicators are shown below in groups.
a) Technology transfer
University/industry research collaboration measures business
executives perceptions of the scope of R&D collaboration between their firm
and local universities. Source: World Economic Forum (WEC), Global
Competitiveness Report.
1
Technological co-operation measures business executives assessment of
the level of technological co-operation between companies. Source: International
Institute for Management Development (IMD), World Competitiveness
Yearbook.
2
b) Entry barriers
Barriers to competition consists of two indices that measure national,
state or provincial laws or other regulations that restrict the number of
competitors allowed to operate, and scope of exemptions to competition law
for public enterprises or government mandated behaviour. Source: OECD.
APPENDIX E
114
Public ownership measures the scope of publicly owned enterprises
across multiple product and service sectors, the size of state-owned
enterprises, and the existence of government special voting rights in
privately-owned enterprises. Source: OECD.
Public involvement in business operation reflects the extent to which
government use coercive or incentive-based regulation in specific sectors.
Source: OECD.
c) Access to foreign markets
Share of new enterprises with exports measures the share of exporting
companies among new enterprises. Source: FORA, Denmark.
Access to capital markets measures to what extent business executives
assess domestic access to capital markets. Source: IMD.
Export credit and insurance measures top managements perceptions of the
availability of export credits and insurance at reasonable prices. Source: IMD.
d) Loan capital
The scope of loan guarantees measures the maximum extent of loan
guarantees as a per cent of a given loan. Sources: EU Commission, OECD, SBA,
SMBA.
Private credit (average 1997-2001) measure the ratio of credit going to the
private sector from deposit-taking financial institutions relative to GDP.
Sources: World Bank, Doing Business.
Interest rate spread measures the lending rate minus the deposit rate
based on the average annual interest rates for each country. Source: World
Bank, Doing Business.
Cost to create collateral measures creditor costs incurred when creating
and registering collateral (as a percentage of income per capita). Source: World
Legal Rights Index measures the extent to which legislation pertaining to
collateral and bankruptcies are beneficial to the entrepreneur. Source: World
Country Credit Rating measures country credit ratings. The rating is
based on an assessment by the Institutional Investor Magazine. Source: IMD,
World Competitiveness Yearbook.
e) Venture capital
Venture capital (early stage) measures the average total annual venture
capital investment (early stage) as a share of GDP. Source: OECD Venture
Capital Database.
APPENDIX E
115
Venture capital (expansion stage) measures the average total annual
venture capital investment (expansion stage) as a share of GDP. Source: OECD,
OECD Venture Capital Database.
f) Exit opportunities
Capitalisation of secondary stock market (average 1999-2002) measures
the capitalisation (the total value of issued shares) in the secondary stock
market in percentage of GDP. Source: OECD.
New companies quoted in secondary stock market (average 1999-2002)
measures the number of new companies quoted in the secondary stock
market per million inhabitants. Source: OECD.
Capitalisation of primary stock market (average 1997-2001) measures the
capitalisation (the total value of issued shares) in the primary stock market in
percentage of GDP. Source: World Bank.
Turnover in primary stock market (average 1997-2001) measures the
value of total shares traded on the stock market exchange in percentage of
GDP. Source: World Bank.
Buy outs (average 1999-2002) measures the total annual value of buy-outs
as a share of GDP. Source: OECD Venture Capital Database.
g) Wealth and bequest tax
Revenue from bequest tax measures the revenue from bequest tax
as per cent of GDP 3-year moving average. Source: OECD.
Revenue from wealth tax measures the revenue from wealth tax
as per cent of GDP 3-year moving average. Source: OECD.
Top marginal bequest tax rate measures the top marginal bequest tax
rate for spouses or children to the deceased. Source: Chen, Lee and Mintz,
OECD.
h) Capital tax
Taxation of dividends top marginal tax rate measures the top marginal
tax rate of dividend income. Source: Chen, Lee and Mintz, OECD.
Taxation of dividends top marginal tax rate for the self-employed
measures the top marginal tax rate of dividend income for the self-employed.
Source: Chen, Lee and Mintz, OECD.
Taxation of stock options measures the effective tax rate of stock options
for a hypothetical tax payer with certain assumptions regarding income,
family situation and portfolio development. Source: EU Commission.
APPENDIX E
116
Taxation of capital gains on shares short term measures the level of the
tax rate when shares are issued and sold in the short term. The indicator is
comprised of two sub-indicators; one tax rate for companies and one tax rate
for individuals. Source: Chen, Lee and Mintz, OECD.
Taxation of capital gains on shares long term measures the level of the
tax rate when shares are issued and sold in the long term. The indicators are
comprised of two lower-level indicators; one tax rate for companies and one
tax rate for individuals. Source: Chen, Lee and Mintz, OECD.
i) Restart possibilities
Time that a creditor has claims on a bankrupts assets measures the time
it takes to settle old debt after bankruptcy. The indicator is measured by the
number of years. Source: OECD.
j) Entrepreneurship education
Entrepreneurship education at primary and secondary levels measures
the perception of entrepreneurship experts of the quantity and quality of
entrepreneurship education in primary and secondary education. Source: GEM
and Schtt (2005c).
Entrepreneurship education at higher levels measures the perception of
entrepreneurship experts of the quantity and quality of entrepreneurship
education in higher education. Source: GEM and Schtt (2005c).
k) Traditional business education
Quality of management schools measures business executives
assessment of the quality of management or business schools. Source: WEF,
Global Competitiveness Report.
l) Government programmes
Government programmes measures entrepreneurial experts assessment
of government counselling programmes implemented to promote the creation
and subsequent growth of new enterprises. Source: GEM.
m) Personal income tax
Highest marginal income tax rate plus contributions measures the
highest rate of taxation for single, high income person without children in
percentage of the gross wage. Source: OECD.
Average income tax plus social contributions measures the average tax
rate for a single person without children in percentage of the gross wage.
Source: OECD.
APPENDIX E
117
n) Corporate tax (including fiscal incentives)
SME tax rates measure the corporate tax rate for small and medium-sized
businesses. Source: Chen, Lee and Mintz, OECD.
Revenue from corporate taxes measures the revenue from corporate
income tax as a percentage of GDP (3-year moving average). Source: OECD.
o) Bankruptcy legislation
Actual cost to close a business measures the cost of closing a business
in per cent of the estate, based on a standard business closure. Source: World
Actual time to close a business measures the actual time to close a
business recorded in calendar years. The indicator is based on a standard
business closure. Source: World Bank, Doing Business.
Recovery rate measure the effectiveness of bankruptcy legislation by
estimating the pay back rate, i.e. how many cents per dollar that creditors are
paid from an insolvent estate. Source: World Bank.
p) Administrative burdens
Number of procedures for starting a business records the generic
procedures that are officially required for an entrepreneur to start an
industrial or commercial business. Source: World Bank, Doing Business.
Average number of days for starting a business measures the average
time spent during each enterprise start-up procedure in calendar days per
start-up procedure. Source: World Bank, Doing Business.
Costs incurred to start a business measures the official costs when
starting a business. Source: World Bank, Doing Business.
Regulatory and administrative opacity measures the simplicity in
obtaining permits and licenses from the government as well as the regulatory
and administrative opacity in business-related rules and administrative
practices. The measure is an index of questions covering the process of
obtaining permits, license and authorisations, as well as the extent to which
authorities make concerted efforts in making business-related regulation
intelligible and simple. Source: OECD.
Enforcing contracts number of procedures measures the number of
procedures mandated by law or court regulation that require interaction
between the concerned parties or between them and the judge, administrator
or court officer. Source: World Bank, Doing Business.
Enforcing contracts time measures the average number of days from the
moment the plaintiff files the lawsuit until the settlement or payment. Source:
World Bank, Doing Business.
APPENDIX E
118
Enforcing contracts cost in percentage of debt measures the official
costs of going through court procedures in percentage of the debt value.
Source: World Bank, Doing Business.
Required capital for starting a business with limited liability measures
the required capital in a company with limited liability. Source: World Bank,
Doing Business.
q) Labour market regulation
Flexibility of hiring index measures whether legislation or other
regulations have implications for the difficulties of hiring a standard worker in
a standard company. Source: World Bank, Doing Business.
Flexibility of firing index measures whether legislation or other
regulation have implications for the difficulties of firing a standard worker in
a standard company. Source: World Bank, Doing Business.
Flexibility of overtime work is an index with five components that
measure the rigidity of rules pertaining to overtime work. Source: World Bank,
Doing Business.
Number of administrative procedures when recruiting the first employee
measures all mandatory contracts that employers have to make with
administrative units in the recruitment process for the first employee. Source:
European Commission.
Number of administrative procedures when recruiting additional
employees measures all mandatory contracts that employers have to make
with administrative units in the recruitment process of additional employees.
Source: European Commission.
Cost of firing measures the costs related to firing employees; advance
notice requirements, severance payments and penalties due in terms of
weekly wages in a standard company. Source: World Bank, Doing Business.
r) Culture/motivation
Cultural and social norms measures entrepreneurial experts perception
of cultural and social norms based n five questions. Source: GEM and Schtt
(2005b).
Entrepreneurial motivation. The indicator measures entrepreneurial
experts assessment of entrepreneurial motivation. The indicator is based on
four questions. Source: GEM and Schtt (2005b).
Self-employment preferences measures individuals preferences towards
being self-employed or being a regular employee. Source: Eurobarometer.
Desirability of becoming self-employed short term measures peoples
desire to become self-employed in the short term. Source: Eurobarometer.
APPENDIX E
119
Desirability of becoming self-employed long term measures peoples
desire to become self-employed within the next five years. Source:
Eurobarometer.
Risk of business failure measures peoples perception of being willing to
start a business if a risk exists that it might fail. Source: Eurobarometer.
Notes
1. The World Economic Forums Global Competitiveness Report is based on an
Executive Opinion Survey as well as data collected from other sources. In the
Executive Opinion Survey managers from across the business world assess the
quality of national framework conditions conducive to competitiveness.
2. The applied indicators IMDs Competitiveness Yearbook is based on an Executive
Opinion Survey, as well as data collected from other sources. In the Executive
Opinion Survey managers from across the business world assess the quality of
national framework conditions conducive to competitiveness.
APPENDIX F
120
APPENDIX F
Summary of the Evaluation of State Aid
to SMEs in the Member States,
European Economic Area
and the Candidate Countries
Country Q A E Short description
Austria y y s Some schemes are evaluated, partly internally, partly outsourced. Evaluations are mainly
quantitative (using questionnaire surveys) with attention for qualitative aspects (e.g.
improvement of market position of the supported firms). Outputs include sets of ratios and
multipliers that measure the effects in terms of money value granted. An example of this
would be the ratio additional sales prompted per 1 assistance, or the subsidy amount per
job created. Austria applies both ex-nunc monitoring and ex post evaluations.
Belguim y y s Aid and evaluation are the responsibility of the regions. Some regions do evaluate. For
example, the Walloon region recently commissioned an evaluation of consultancy aid. This
extensive study describes the (entry) problems SMEs experience, the support provided, and
the effects of the support. The study employs various tools and techniques for data collection
(among users and non-users of the support) such as literature research, file research,
telephone survey and face-to-face interviews. Also, they use various methods for data
analysis, such as graphic representations, statistical analyses and control groups. There is a
separate chapter on synthesis and recommendations.
Bulgaria n n ?
Cyprus n y ?
Czech
Republic
y n s The Ministry of Trade and Industry evaluates some schemes. A major evaluation of the Czech-
Moravian Guarantee and Development Bank (whose objective it is to stimulate the SME
sector) is currently underway.
Denmark y y s Denmark does evaluate state aid, but not specific SME schemes. Some of the evaluations are
outsourced. Reports are in Danish.
Estonia n y ?
Finland y y s Ex ante, ex-nunc and ex post evaluations re mandatory by law, but in practice they do not
always take place. As a result, there is an impressive record of Finnish evaluation studies.
However, a government working group judged the valuation efforts unsystematic. Therefore,
the working group has made 10 recommendations so that evaluation results will be integrated
in the policy process in a systematic manner.
France n y ?
APPENDIX F
121
Germany y y s Germany evaluates some of its state aid. Part of these evaluation studies are performed
internally, other studies are outsourced. Evaluation has helped Germany to improve state aid
objectives considerably during the last 5 years. In general, German evaluations of state aid to
SMEs use quantitative approaches, where the results of assisted firms are compared with the
results of non-assisted firms. German evaluators also apply time series analysis.
Greece y y n In Greece, state aid measures are not evaluated yet. A method for the appraisal of aid granted
is being developed.
Hungary y y s Some Hungarian state aid schemes are evaluated, partly by external consultants. Evaluations are
often based on statistical analyses, but an interesting good practice in Hungary is the analysis of
80 randomly chosen micro-credit documents (including the micro-credit contracts and the
recipients business plans).
Ireland y y a
Italy y y s Italy evaluates some of its aid schemes. Examples include a quantitative analysis of the net
impacts of SME subsidies on employment. This evaluation uses literature data, data from the
tax and social security offices and a questionnaire. Based on this, indicators of the support
schemes effects are calculated. The indicators are applied to measure the extent to which the
support objectives have been reached.
Latvia y y n Latvia does not evaluate its state aid to SMEs.
Lithuania y y ?
Luxembourg n n ?
Malta n n ?
The
Netherlands
y y a It is stated in the law that:
For each new or adjusted policy objective or instrument it should be considered whether ex-
ante evaluation is useful.
Each existing support measure should be evaluated, at least once per five years. For ex post
evaluations, the Dutch Ministry of Economic Affairs uses one standard format. The main
purpose of evaluation is to check whether the policy instrument contributes to the main goal
for which it was created.
Norway y y s In Norway state aid is targeted at horizontal objectives, most of which concern SMEs. Besides
evaluation results, political and administrative priorities have driven changes in the Norwegian
state aid policy.
Poland y y n Poland does not evaluate state aid to SMEs.
Portugal y y s A large proportion of Portuguese state aid to SMEs is included in the Community Support
Framework. An ex ante evaluation of this framework was carried out and in 2003 the support
measures (including state aid) under this framework were subject to mid-term evaluations using a
combination of qualitative (case studies) and quantitative (financial data, physical indicators,
beneficiary questionnaires) tools, based on the MEANS quality criteria. Only state aid schemes that
are not included in the Community Support Framework are not evaluated.
Romania n n ?
Slovakia n y ? The State Aid Act obliges the Slovak government to evaluate all state aid. This is done using one
standard evaluation technique for all state aid. This technique consists of statistical (input-output)
analyses of economic characteristics of aid recipients and a (total population) control group,
(turnover, value added, profits, taxes, exports and number of employees) The evaluation focuses
on four topics: technology transfer, innovation, exports and competitiveness. The outputs of the
analyses are data, such as: number of jobs created through the scheme, number of companies
established due to the measure, etc. The analyses are performed both on micro and macro levels.
Aided SMEs are followed for three consecutive years to see how they develop. Data about the
assisted companies are obtained through questionnaire surveys. Data about the control group are
obtained from the Tax Office and the Statistical Office.
Slovenia n y ?
APPENDIX F
122
Spain n y ?
Sweden y y s The Swedes use quantitative matched sample control group approaches, total population
control group approaches, cost-benefit analyses, supported with more qualitative interviews.
They use these methods to measure the effect of state aid on the survival and sales of firms,
employment, regional development, etc. They also assess the influence of other variables,
such as sector, firm size and region. Evaluations have yielded important insights into the
effects of support. The Swedes do not use the outcomes of evaluation studies for re-focussing
state aid objectives.
Turkey n n ?
United
Kingdom
y y s Recently, evaluation of all DTI services acted as inputs for a new strategy to reduce the
number of services to 10-14. All evaluations are out-sourced and follow DTI evaluation
guidelines. Where possible they use a set of standard indicators.
Notes:
Q = questionnaire received from the country y: yes; n: no
A = attended the member State meeting on 25 May 2004 y: yes; n: no
E = perform state aid evaluation a: all state aid schemes are evaluated
s: some state aid schemes are evaluated
n: no state aid scheme is evaluated
? no information available
Source: EIM (2004).
OECD PUBLICATIONS, 2, rue Andr-Pascal, 75775 PARIS CEDEX 16
PRINTED IN FRANCE
(85 2007 04 1 P) ISBN 978-92-64-04008-3 No. 55919 2006
P
O
L
IC
Y
E
V
A
L
U
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T

ENTREPRENEURSHIP PO
LICY EVALUATIO
N FIRST FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH FO
URTH THRID SECO
ND FIRST FIF
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
T
H
IR
D
F
O
U
R
T
H
F
IF
T
H
F
IR
S
T
S
E
C
O
N
D
SM
E PO
LICY EVALUATIO
N ENTREPRENEURSHIP D FIRST FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH FO
URTH FIFTH FO
URTH THIRD SECO
ND FIRST FIFTH
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E

PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
I
SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATI
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O

PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N ENT
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N
ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
I
SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATI
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O

PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
SM
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N ENT
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O
L
IC
Y
E
V

PO
LICY EVALUATIO
E PO
LICY EVALUATIO
N ENTREPRENEUR
P
O
L
IC
Y
E
V
A
L
U
A
T
IO
N
S
M
E
E
N
T
R
E
P
R
E
N
E
U
R
S
H
IP
P
O

ENTREPRENEURSHIP SM
E PO
LICY EVALUATIO
N
S
M
E
E
N
T
R
E
P
OECD Framework
-:HSTCQE=UYUU]X:
The full text of this book is available on line via this link:
www.sourceoecd.org/industrytrade/9789264040083
Those with access to all OECD books on line should use this link:
www.sourceoecd.org/9789264040083
SourceOECD is the OECDs online library of books, periodicals and statistical databases.
For more information about this award-winning service and free trials, ask your librarian, or write to
us at SourceOECD@oecd.org.
ISBN 978-92-64-04008-3
85 2007 04 1 P
OECD Framework for the Evaluation
of SME and Entrepreneurship Policies
and Programmes
This Framework provides policy makers with a concrete, explicit, practical and accessible
guide to best practice evaluation methods for SME and entrepreneurship policies
and programmes, drawing upon examples from a wide range of OECD countries.
It examines the benets of evaluation and how to address common issues that arise
when commissioning and undertaking SME and entrepreneurship evaluations. Key
evaluation principles are set out, including the Six Steps to Heaven approach, and
illustrated with examples of evaluations of national, regional and local programmes
that can be explored further by the reader. The publication focuses not only on the
evaluation of individual policies and programmes but also on bigger picture peer
review evaluations and assessment of the impact on SMEs and entrepreneurship
of mainstream programmes that do not have business development as their principal aim.

O
E
C
D

F
r
a
m
e
w
o
r
k

f
o
r

t
h
e

E
v
a
l
u
a
t
i
o
n

o
f

S
M
E

a
n
d

E
n
t
r
e
p
r
e
n
e
u
r
s
h
i
p

P
o
l
i
c
i
e
s

a
n
d

P
r
o
g
r
a
m
m
e
s
852007041cov.indd 1 11-Dec-2007 10:43:50 AM

OECD Framework For The Evaluation of SME and Entrepreneurship Policies and Programmes

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

OECD Framework For The Evaluation of SME and Entrepreneurship Policies and Programmes

Загружено:

Авторское право:

Доступные форматы

P

Вам также может понравиться