Академический Документы
Профессиональный Документы
Культура Документы
Evaluation
Quantitative
Qualitative
Review
Evaluation
Evaluation can be of policy, initiatives or interventions and can be formative or
summative in nature. An evaluation may utilise both sub categories of assessment /
research, Qualitative and Quantitative. In terms of assessment interventions (or
instruments) evaluations allow you to assess the instruments themselves.
This approach can be sub categorised into the following discrete types of
assessment:
Descriptive (non-experimental)
Quantitative - Quasi-Experimental
(A quasi-experiment can allocate some treatment, to part of a population, but not
another, but without any kind of randomisation. It is obviously not as good as a real
experiment from a theoretical standpoint, since you can never be sure that the groups
have not been assigned according to some more or less systematic plan. The classic in
this was the Lanarkshire milk experiment, supposed to observe the effect of free milk on
children’s growth, where the teachers scuppered the project by giving the milk to the
children who needed it most)
Quantitative - Experimental
(An experiment compares two (or more) programmes of instruction, schools etc,
individuals are randomly assigned to one or other arm of a study, and the results
compared at the end.)
Nota Bene: Experiments and even quasi-experiments can be difficult to set up and
one may be tempted to use pre-existing groups and compare their performance. This
is an ex post facto design. However ex post facto designs have the problem that the
groups involved may be intrinsically different. So some pupils might attend grammar
schools and some comprehensives, and if on the whole pupils in the grammar
schools did better, than it could be suggested that this was due to differences in
attainment at entry. If we have information on such type of factor (e.g. achievement
at entry), and can make allowance for it statistically
Quantitative or Qualitative
The distinctions between the two approaches are shown in the comparative table
below:
Quantitative Qualitative
Objective Subjective
Research questions: How many? Strength of Research questions: What? Why?
association?
Facts are value-free and unbiased Facts are value-laden and biased
Reduction, control, precision Discovery, description, understanding, shared
interpretation
Measurable Interpretive
Mechanistic: parts equal the whole Organismic: whole is greater than the parts
Report statistical analysis. Report rich narrative, individual; interpretation.
Basic element of analysis is words/ideas.
Highly controlled setting: experimental setting (outcome Flexible approach: natural setting (process
oriented) oriented)
For each of these sub categories there are different approaches. It should be
remembered that these approaches to assessment are often used in association
with each other rather than in isolation. Each has its own data collection and data
analysis methods. It is, therefore, imperative to define at the outset what the final
outcome (or report) requires as proof and then determine, in reverse, what data
collection method will allow that analysis to take place. This in turn will determine the
approach (or combination of approaches). The data gathering instruments, or
“assessments” can then be constructed to best deliver the required data. Open the
embedded PDF spreadsheet (fig1) for greater detail on the different approaches
Figure 2, flow diagram ,shows the relationships between the assessment approach
and the analyses that follow from the data collected. The outcomes can be used
either in a formative or a summative manner depending on the construct of the test.
Enter project data detail incl ITT Updated "Proj mgmnt system" with Enter project data detail incl ITT Updated "Proj mgmnt system" with
dissemination process decision* auto Dissemination spc for auto dissemination process decision* auto Dissemination spc for auto
costings from stndard print cost table costings from stndard print cost table
Prepare document structure and Style guide Skeleton ITT response document Prepare document structure and Style guide Skeleton ITT response document
standard text templates standard text templates
Set up Document Management Doc Mgmnt guide Correctly structured Doc Mgmnt Set up Document Management Doc Mgmnt guide Correctly structured Doc Mgmnt
system files and assign rights system file with correct access system files and assign rights system file with correct access
permissions permissions
Health and safety review H&S Policy report Set up Document Management H&S Policy report
system files and assign rights
Project Risk assessment Risk Policy report Set up Document Management Risk Policy report
system files and assign rights
Compile standard text entries CVs & other standing texts including Current data included in updated ITT Set up Document Management CVs & other standing texts including Current data included in updated ITT
dissemination text response system files and assign rights dissemination text response
Write Technical response Text files supplied by contributory Updated ITT Set up Document Management Text files supplied by contributory Updated ITT
Specialist Services system files and assign rights Specialist Services
Prepare costings Departmental costings supplied in Costing calculations to match Clients Set up Document Management Departmental costings supplied in Costing calculations to match Clients
response to ITT owners request for requirements, an Updated ITT a system files and assign rights response to ITT owners request for requirements, an Updated ITT a
services profiled budget for Finance services profiled budget for Finance
Data Services costs (total includes O'head, staff costs Project specific profiled costings Set up Document Management O'head, staff costs Project specific profiled costings
O'head and staff costs) system files and assign rights
Statistical analysis costs given in O'head, staff costs Project specific profiled costings Set up Document Management O'head, staff costs Project specific profiled costings
"ave" statistician days) system files and assign rights
Dissemination Freelance and Dissemination methods & Project specific profiled costings Set up Document Management Dissemination methods & Project specific profiled costings
Print/publishing costs specification, Print cost matrix system files and assign rights specification, Print cost matrix
O'head costs including Fixed costs and overhead allocations NONE Set up Document Management Fixed costs and overhead allocations NONE
Communications costs - where system files and assign rights
appropriate
Staff costs Staff rates by Grade NONE Set up Document Management Staff rates by Grade NONE
system files and assign rights
Dept Staff time alloc Current resource allocations calenderised projected staff Set up Document Management Current resource allocations calenderised projected staff
deployment for resource planning and system files and assign rights deployment for resource planning and
management reporting management reporting
Produce schedules / Timetable ITT / Draft response to ITT, specialist Project schedule including Set up Document Management ITT / Draft response to ITT, specialist Project schedule including
services sub-process schedules Checkpoints & product delivery to system files and assign rights services sub-process schedules Checkpoints & product delivery to
clients as a minimum clients as a minimum
Produce response to ITT / Proposal ITT / Draft response to ITT, specialist Proposed Response to Client - incl Set up Document Management ITT / Draft response to ITT, specialist Proposed Response to Client - incl
services sub-process schedules system files and assign rights services sub-process schedules
Peer review Proposed Response to Client Annotated comments on Response Set up Document Management Proposed Response to Client Annotated comments on Response
to Tender system files and assign rights to Tender
Stakeholder review Proposed Response to Client Annotated comments on Response Set up Document Management Proposed Response to Client Annotated comments on Response
to Tender - impact analysis of system files and assign rights to Tender - impact analysis of
alternatives agreed & supplied alternatives agreed & supplied
Tests Specification Specification template Project specific Specification Select/specify Sample Specification template Sample specification - Draft
Early development (Text Selection) Project specific Specification Item specification - Text maps (Eng) Write Interview Schedule (Prompts ) NONE Project specific instrument
Writers Meeting Item specification - Text maps (Eng) NONE Interview schedule (prompts) peer Project specific instrument Amendments to be made to the
review instrument
Item Writing Item specification - Text maps (Eng) Draft Items (Eng internal) (Maths / Interview schedule (prompts) revised Project specific instrument Project specific instrument amended
Science external)
Develop mark scheme Items Draft item level marking scheme Instrument produced Project specific instrument amended Digital version of the instrument
Items entered by DTP Item specification DTP'd items in dummy tests Interview schedule / prompts agreed Project specific instrument amended Agreed interview schedule / prompts
by client
Informal Trialing DTP'd items in dummy tests Item performance in trials report Inform RDS of subjects used Agreed Sample specification Notiification
Internal Review Meetings Item performance in trials report Draft Performance report for client Timetables (Schedules etc) produced contract timetable Planned range of dates for fieldwork
review /reviewed
Expert review Items & Performance report for client Draft Performance report for client Contact subjects & arrange visits Planned range of dates for fieldowork Updated timetable of fieldworker
review review visits
Teacher Panel Items & Performance report for client Performance report for client review Fieldworkers meeting DTP'd instrument NONE
review
(Clients') Review Meetings Draft Performance report for client Agreed items for inclusion in pre-test Fieldwork visits DTP'd instrument Draft interview / observation notes,
review 1 - with amendments recordings etc
Items Revised Agreed items for inclusion in pre-test Final items for inclusion in pre-test 1 Write up fieldwork notes Draft interview / observation notes, Fieldworker report
1 - with amendments recordings etc
NONE NONE Agree report structure Contractural requirements - Report structure
Fieldworker report
Specify Sample for Data Services Specification template Sample specification - Draft analysis Fieldworker's reports Technical analysis
Sample Agreed Sample specification Agreed specification Synthesis Fieldworker reports - and digital compiled materials
materials, literature reviews etc
Specify Tests Specification template Test specification Write report Synthesis, report Templates - where Draft report
appropriate
DTP Produce Tests Test specification DTP'd test instruments Internal Review Draft report Agreed Draft report
Other related materials - instructions Templates - instructions, manuals Finished "related materials" Draft Report to Client Agreed Draft report NONE
Questionnaires, instruments etc etc
produced
Materials to Team Printed materials or digital files NONE Can go straight to "Finalisation" if NONE NONE
no Quantitative element
Team Reviews Compiled and produced test Agreed Test instrument Speci fy Sample (to RDS) Specification template Sample specification -
Draft the coding frame Mark scheme Draft Coding Framework Questionnaire writing (Design) Questionairre template where Project specific designed
appropriate questionairre
Print Authorisation Agreed Test instrument Print Authorisation update in Proj Questionnaire peer review including Project specific designed Amendments to be made to the
Mgmgnt system Focus groups , cognitive interviewing questionairre instrument
Confirmation of non-print Print Authorisation update in Proj NONE Questionnaire revised Project specific designed Project specific instrument amended
requirements, delivery and archiving Mgmgnt system questionairre
Requirements to Facilities (changes
from earlier process must have been
notified through Change Control
NONE NONE Questionnaire DTP’d Instrument specification DTP'd items in draft instrument
Print test materials Print specification, print authorisation Printed materials Questionnaire peer / RDS review, DTP'd items in draft instrument Amendments to be made to the
and DTP'd test instrument including Focus groups , cognitive instrument
interviewing
Accept print materials ready for Printed materials Despatch authorisation update in Questionnaire Revised DTP'd items in draft instrument Project specific instrument amended
despatch Proj Mgmnt system
Despatch tests List of schools involved in the tril - Despatch notes
addresses, contacts
Write Analysis Specification Specification Templates Draft specification Client agreement to Questionairre Project specific instrument amended Agreed Questionairre
Analysis Specification for Stats Analysis specification Agreed specification Produce final DTP’d Questionnaire for Draft Instrument and amendments Final CRC instrument
relevant media mechanisms
Approve Data entry Set-up Analysis specification Updated information Print Authorisation Agreed instrument Print Authorisation update in Proj
Mgmgnt system
Testing period NONE NONE Confirmation of non-print Print Authorisation update in Proj NONE
requirements, delivery and archiving Mgmgnt system
Requirements to Facilities (changes
from earlier process must have been
notified through Change Control
Tests returned NONE completed answer books Print questionnaire or produce Print specification, print authorisation Printed materials
relevant delivery mechanism and DTP'd test instrument
Marker training Batches of pre-test 1 answer Trained markers Despatch questionnaires or make live List of establishments involved - Despatch notes
booklets addresses, contacts
Mark & coding schemes Reviewed Draft Mark and coding schemes Revised mark and coding scheme Write Analysis Specification Specification Templates Draft specification
Marking Approved mark scheme - scripts Raw marks Analysis Specification agreed with Analysis specification Agreed specification
Stats
Data entry Raw marks Raw data Approve Data entry Set-up Analysis specification Updated i nformation
Data Preparation Raw data Prepared Data files Survey period NONE NONE
NONE NONE Questionnaires / data returned NONE NONE
Analysis Prepared Data files - Templates Draft analysis Coding Raw dat a files Coded data files
Write technical report - appendix Draft analysis - Templates Draft technical appendix Data entry or "Pulsetrain"! Raw marks Raw data
NONE NONE Data Preparation Raw data Prepared Data files
Consolidate the report Draft analysis - Templates Draft report - Level or standards Analysis Prepared Data files - Templates Draft analysis
setting report for National Curriculum
tests
Internal Review Draft report - Level or standards Final first pre test report Write technical report - appendix Draft analysis - Templates Draft technical appendix
setting report for National Curriculum
tests
Expert review Items & Performance report for client Draft Performance report for client Consolidate the Quantitative report, Draft analysis - Templates - draft Draft report
review review where appropriate Technical appendix
Teacher Panel Items & Performance report for client Performance report for client review Internal Review Draft report Agreed Draft report
review
(Clients') Review Meetings Draft Performance report for client Agreed items for inclusion in pre-test Draft Report to Client Agreed Draft report NONE
review 1 - with amendments
Report to Client Final report Final Client report NONE NONE
NONE NONE Clients Review Meeting Agreed Draft report Reviewed Client report - Actions for
the Actions log
Specify Sample (to RDS) Specification template Sample specification - Draft Review dissemination decision Proj Info System PHR detail on Enter updated PHR detail on
dissemination process decision* dissemination - auto inform
Communications
Sample Agreed Sample specification Agreed specification Prepare Final fully consolidated Draft reports incl client comments Final report
report
Specify Tests Specification template Test specification Write Research summary - includes Final Client Report Research summary and Key Points
"Key points"
DTP Produce Tests Test specification DTP'd test instruments Report to Client Report Template Final Client report
Other related materials - instructions Templates - instructions, manuals Finished "related materials"
Questionnaires, instruments etc etc
produced
Materials to Team Printed materials or digital files NONE
Team Reviews Compiled and produced test Agreed Test instrument
Print Authorisation Agreed Test instrument Print Authorisation update in Proj
Mgmgnt system
Confirmation of delivery and archiving Print Authorisation update in Proj NONE
Requirements to Facilities (changes Mgmgnt system
from earlier process must have been
notified through Change Control
NONE NONE
Print test materials Print specification, print authorisation Printed materials
and DTP'd test instrument
Accept print materials ready for Printed materials Despatch authorisation update in
despatch Proj Mgmnt system
Despatch tests List of schools involved in the tril - Despatch notes
addresses, contacts
Write Analysis Specification Specification Templates Draft specification
Analysis Specification for Stats Analysis specification Agreed specification
Approve Data entry Set-up Analysis specification Updated information
Testing period NONE NONE
Tests returned NONE completed answer books
Mark & coding schemes Reviewed Draft Mark and coding schemes Revised mark and coding scheme
Marker training Batches of pre-test 2 answer Trained markers
booklets
Marking Approved mark scheme - scripts Raw marks
Data entry Raw marks Raw data
Data Preparation Raw data Prepared Data files
NONE NONE
Analysis Prepared Data files - Templates Draft analysis
Write technical report - appendix Draft analysis - Templates Draft technical appendix
NONE NONE
Consolidate the report Draft analysis - Templates Draft report - Level or standards
setting report for National Curriculum
tests
Internal Review Draft report - Level or standards Final report
setting report for National Curriculum
tests
Report to Client Final report Final Client report
NONE NONE
Clients Review Meeting(s) - possibly Final Client report Reviewed Client report - Actions for
iterative the Actions log
Review dissemination decision Proj Info System PHR detail on Enter updated PHR detail on
dissemination process decision* dissemination - auto inform CMIS
Prepare Final Materials Reviewed Client report - Actions for Final assessment materials - to
the Actions log meet Client requirements
Finalise mark Schemes Pre-test 1&2 reviewed markschemes - Final Mark Scheme - to meet Client
Final assessment materials requirements