Вы находитесь на странице: 1из 5

Assessment – A conceptual framework for assessment / research

There are four main divisions in assessment / research:

Evaluation
Quantitative
Qualitative
Review

Evaluation
Evaluation can be of policy, initiatives or interventions and can be formative or
summative in nature. An evaluation may utilise both sub categories of assessment /
research, Qualitative and Quantitative. In terms of assessment interventions (or
instruments) evaluations allow you to assess the instruments themselves.

Quantitative Research, a definition –


The use of structured questions where the response options have been
predetermined and a large number of respondents is involved.

This approach can be sub categorised into the following discrete types of
assessment:
Descriptive (non-experimental)
Quantitative - Quasi-Experimental
(A quasi-experiment can allocate some treatment, to part of a population, but not
another, but without any kind of randomisation. It is obviously not as good as a real
experiment from a theoretical standpoint, since you can never be sure that the groups
have not been assigned according to some more or less systematic plan. The classic in
this was the Lanarkshire milk experiment, supposed to observe the effect of free milk on
children’s growth, where the teachers scuppered the project by giving the milk to the
children who needed it most)
Quantitative - Experimental
(An experiment compares two (or more) programmes of instruction, schools etc,
individuals are randomly assigned to one or other arm of a study, and the results
compared at the end.)
Nota Bene: Experiments and even quasi-experiments can be difficult to set up and
one may be tempted to use pre-existing groups and compare their performance. This
is an ex post facto design. However ex post facto designs have the problem that the
groups involved may be intrinsically different. So some pupils might attend grammar
schools and some comprehensives, and if on the whole pupils in the grammar
schools did better, than it could be suggested that this was due to differences in
attainment at entry. If we have information on such type of factor (e.g. achievement
at entry), and can make allowance for it statistically

Qualitative research, a definition -


Collecting, analyzing, and interpreting data by observing what people do and say –
useful at the beginning of a research project to establish the status quo - (can be needed to
explain findings in Quantitative research)

Quantitative or Qualitative

The distinctions between the two approaches are shown in the comparative table
below:
Quantitative Qualitative
Objective Subjective
Research questions: How many? Strength of Research questions: What? Why?
association?

"Hard" science "Soft" science


Literature review must be done early in study Literature review may be done as study
progresses or afterwards

Test theory Develops theory


One reality: focus is concise and narrow Multiple realities: focus is complex and broad

Facts are value-free and unbiased Facts are value-laden and biased
Reduction, control, precision Discovery, description, understanding, shared
interpretation

Measurable Interpretive
Mechanistic: parts equal the whole Organismic: whole is greater than the parts
Report statistical analysis. Report rich narrative, individual; interpretation.
Basic element of analysis is words/ideas.

Basic element of analysis is numbers


Researcher is separate Researcher is part of process
Subjects Participants
Context free Context dependent
Hypothesis Research questions
Reasoning is logistic and deductive Reasoning is dialectic and inductive
Establishes relationships, causation Describes meaning, discovery
Uses instruments Uses communications and observation
Strives for generalization Strives for uniqueness
Generalizations leading to prediction, explanation, and Patterns and theories developed for
understanding understanding

Highly controlled setting: experimental setting (outcome Flexible approach: natural setting (process
oriented) oriented)

Sample size: n Sample size is not a concern; seeks "informal


rich" sample

"Counts the beans" Provides information as to "which beans are worth


counting"

For each of these sub categories there are different approaches. It should be
remembered that these approaches to assessment are often used in association
with each other rather than in isolation. Each has its own data collection and data
analysis methods. It is, therefore, imperative to define at the outset what the final
outcome (or report) requires as proof and then determine, in reverse, what data
collection method will allow that analysis to take place. This in turn will determine the
approach (or combination of approaches). The data gathering instruments, or
“assessments” can then be constructed to best deliver the required data. Open the
embedded PDF spreadsheet (fig1) for greater detail on the different approaches

In educational terms the construct of the assessment instrument must be broad


enough in its coverage of the curriculum, have a range of difficulty in its questions,
have validity – face, construct etc, be reliable and comparable over time.

Figure 2, flow diagram ,shows the relationships between the assessment approach
and the analyses that follow from the data collected. The outcomes can be used
either in a formative or a summative manner depending on the construct of the test.

Figure 3 depicts the end-to-end process of developing and delivering both


quantitative and qualitative assessments.

Figure 1: Assessment approaches


Figure 2: Approaches and their analyses as used in the NFER
A comparison of the development of Quantitative and Qualitative assessments / research

Quantitative Assessment Quantitative Assessment


Top Level Processes Documentation / information "Documentation" or output from Top Level Processes - greyed Documentation / information "Documentation" or output from
dependencies - input into the the process out entries common with dependencies - input into the the process
process Quantitative Assessment process
“ITT” formulated and agreed or ITT or Draft ITT Final ITT “ITT” formulated and agreed or ITT or Draft ITT Final ITT
received from Client received from Client
Initial "appropriateness" review of ITT Final ITT Decision to proceed Initial "appropriateness" review of ITT Final ITT Decision to proceed
(is it worth considering) (is it worth considering)
Revise CRM information / Update Entry fields from CRM (BDG) project Entry into CCRD & Updated CRM & Revise CRM information / Update Entry fields from CRM (BDG) project Entry into CCRD & Updated CRM &
"Tenders" segment of Information requirements - TBA "Tenders" segment of Information "Tenders" segment of Information requirements - TBA "Tenders" segment of Information
D'base D'base D'base D'base
Assign ITT owner ITT Updated PIO Tenders database table Assign ITT owner ITT Updated PIO Tenders database table

Enter project data detail incl ITT Updated "Proj mgmnt system" with Enter project data detail incl ITT Updated "Proj mgmnt system" with
dissemination process decision* auto Dissemination spc for auto dissemination process decision* auto Dissemination spc for auto
costings from stndard print cost table costings from stndard print cost table

Review ITT ITT NONE Review ITT ITT NONE


Input “Checkpoint dates” into Project PHR Updated PHR & schedules Input “Checkpoint dates” into Project PHR Updated PHR & schedules
Highlight Report (Pre-delivery stage) Highlight Report (Pre-delivery stage)

Prepare document structure and Style guide Skeleton ITT response document Prepare document structure and Style guide Skeleton ITT response document
standard text templates standard text templates
Set up Document Management Doc Mgmnt guide Correctly structured Doc Mgmnt Set up Document Management Doc Mgmnt guide Correctly structured Doc Mgmnt
system files and assign rights system file with correct access system files and assign rights system file with correct access
permissions permissions
Health and safety review H&S Policy report Set up Document Management H&S Policy report
system files and assign rights
Project Risk assessment Risk Policy report Set up Document Management Risk Policy report
system files and assign rights
Compile standard text entries CVs & other standing texts including Current data included in updated ITT Set up Document Management CVs & other standing texts including Current data included in updated ITT
dissemination text response system files and assign rights dissemination text response
Write Technical response Text files supplied by contributory Updated ITT Set up Document Management Text files supplied by contributory Updated ITT
Specialist Services system files and assign rights Specialist Services
Prepare costings Departmental costings supplied in Costing calculations to match Clients Set up Document Management Departmental costings supplied in Costing calculations to match Clients
response to ITT owners request for requirements, an Updated ITT a system files and assign rights response to ITT owners request for requirements, an Updated ITT a
services profiled budget for Finance services profiled budget for Finance

Data Services costs (total includes O'head, staff costs Project specific profiled costings Set up Document Management O'head, staff costs Project specific profiled costings
O'head and staff costs) system files and assign rights
Statistical analysis costs given in O'head, staff costs Project specific profiled costings Set up Document Management O'head, staff costs Project specific profiled costings
"ave" statistician days) system files and assign rights
Dissemination Freelance and Dissemination methods & Project specific profiled costings Set up Document Management Dissemination methods & Project specific profiled costings
Print/publishing costs specification, Print cost matrix system files and assign rights specification, Print cost matrix
O'head costs including Fixed costs and overhead allocations NONE Set up Document Management Fixed costs and overhead allocations NONE
Communications costs - where system files and assign rights
appropriate
Staff costs Staff rates by Grade NONE Set up Document Management Staff rates by Grade NONE
system files and assign rights
Dept Staff time alloc Current resource allocations calenderised projected staff Set up Document Management Current resource allocations calenderised projected staff
deployment for resource planning and system files and assign rights deployment for resource planning and
management reporting management reporting
Produce schedules / Timetable ITT / Draft response to ITT, specialist Project schedule including Set up Document Management ITT / Draft response to ITT, specialist Project schedule including
services sub-process schedules Checkpoints & product delivery to system files and assign rights services sub-process schedules Checkpoints & product delivery to
clients as a minimum clients as a minimum
Produce response to ITT / Proposal ITT / Draft response to ITT, specialist Proposed Response to Client - incl Set up Document Management ITT / Draft response to ITT, specialist Proposed Response to Client - incl
services sub-process schedules system files and assign rights services sub-process schedules

Peer review Proposed Response to Client Annotated comments on Response Set up Document Management Proposed Response to Client Annotated comments on Response
to Tender system files and assign rights to Tender
Stakeholder review Proposed Response to Client Annotated comments on Response Set up Document Management Proposed Response to Client Annotated comments on Response
to Tender - impact analysis of system files and assign rights to Tender - impact analysis of
alternatives agreed & supplied alternatives agreed & supplied

NONE NONE NONE NONE


Decision to proceed Final proposed Response to Tender - Agreed response to tender Decision to proceed Final proposed Response to Tender - Agreed response to tender
including final agreement from including final agreement from
Stakeholders to amendments made Stakeholders to amendments made
where there is an impact where there is an impact
Authorisation to submit response to Agreed response to tender Authorisation Authorisation to submit response to Agreed response to tender Authorisation
Client Client
Submit response Agreed response to tender NONE Submit response Agreed response to tender NONE
Update CRM database Agreed response to tender Updated CRM database Update CRM database Agreed response to tender Updated CRM database
Produce project Board Paper / Agreed response to tender Project Board Paper (report on Produce project Board Paper / Agreed response to tender Project Board Paper (report on
Project Outline (Board Paper without the projects won) and Project Outline Project Outline (Board Paper without the projects won) and Project Outline
Finances) (Board Paper - financials) Finances) (Board Paper - financials)
Produce Board Paper (for Research Agreed response to tender, Board Board paper for RDF approval Produce Board Paper (for Research Agreed response to tender, Board Board paper for RDF approval
Development Fund approval paper template Development Fund approval paper template
Update Proj Info System Agreed response to tender Updated "Proj mgmnt system" and Update Proj Info System Agreed response to tender Updated "Proj mgmnt system" and
"DTP / P rint specifications "DTP / Print specifications
NONE NONE NONE NONE
Contract pitch & negotiations Original Invitation to Tender & Agreed Updated Response to tender Contract pitch & negotiations Original Invitation to Tender & Agreed Updated Response to tender
response to tender response to tender
Client agrees project – or Client criteria & Original Invitation to Letter of intent or Contract Client agrees project – or Client criteria & Original Invitation to Letter of intent or Contract
Tender & Agreed response to tender Tender & Agreed response to tender
- Proposal unsuccessful Client criteria - Decision on awarding Letter / email - Proposal unsuccessful Client criteria - Decision on awarding Letter / email
the contract the contract
If unsuccessful - Client debriefing to Agreed response to tender Updated CRM database - updated If unsuccessful - Client debriefing to Agreed response to tender Updated CRM database - updated
establish criteria Proj mgmnt system d'base - Lesson establish criteria Proj mgmnt system d'base - Lesson
learnt report - updated Actions Log learnt report - updated Actions Log
Update Proj Info System with Letter of intent or Contract - Letter / Updated "Proj mgmnt system" and Update Proj Info System with Letter of intent or Contract - Letter / Updated "Proj mgmnt system" and
reasons email "DTP / Print specifications reasons email "DTP / Print specifications
Inform stakeholders Decision on awarding the contract Notice of contract status - awarded / Inform stakeholders Decision on awarding the contract Notice of contract status - awarded /
lost lost
Client signs / Supplier receives contract Scanned parts of contract - or Client signs / Supplier receives contract Scanned parts of contract - or
Contract physical storage of contract Contract physical storage of contract
NONE NONE NONE NONE
Throughout the DELIVERY stage any Agreed response to tender Request for change - impact analysis Throughout the DELIVERY stage any Agreed response to tender Reques t for change - impact analysis
change to the specification of any change to the specification of any
element must have an Impact element must have an Impact
Analysis and must be agreed with Analysis and must be agreed with
the impacted parties - Project the impacted parties - Project
Governance - and will be reported on Governance - and will be reported on
the Project Highlight Report the Project Highlight Report
Notify Finance with Profiled Budget Contract Costing calculations to Profiled budget for accounts - auto Notify Finance with Profiled Budget Contract Costing calculations to Profiled budget for accounts - auto
match Clients requirements, a compiled from the input to the ITT match Clients requirements, a compiled from the input to the ITT
profiled budget for Finance profiled budget for Finance
Update Proj Info System Letter of intent or Contract - Letter / Updated "Proj mgmnt system" Update Proj Info System Letter of intent or Contract - Letter / Updated "Proj mgmnt system"
email email
Project Team meeting Agreed response to tender and NONE Project Team meeting Agreed response to tender and NONE
contract synopsis - Profiled budget contract synopsis - Profiled budget
Confirm Requirements to DTP, Final agreed requirements as Agreed specification Confirm Requirements to DTP, Final agreed requirements as Agreed specification
Statistical Analysis, Data Services, detailed in the final contract / Statistical Analysis, Data Services, detailed in the final contract /
Communications response Communications response
Notify non-print requirements, Final agreed requirements as Agreed specification Notify non-print requirements, Final agreed requirements as Agreed specification
delivery and archiving Requirements detailed in the final contract / delivery and archiving Requirements detailed in the final contract /
to Facilities (changes from must response to Facilities (changes from must response
have been notified through Change have been notified through Change
Control) Control)
Review / Complete Project Agreed response to tender and Updated documents where Review / Complete Project Agreed response to tender and Updated documents where
Origination Documentation (project contract synopsis - Profiled budget - necessary, Change Control with Origination Documentation (project contract synopsis - Profiled budget - necessary, Change Control with
form) Final agreed requirements as impact analysis where necessary form) Final agreed requirements as impact analysis where necessary
detailed in the final contract / (others impacted) detailed in the final contract / (others impacted)
response & agreed specifications response & agreed specifications

Tests Specification Specification template Project specific Specification Select/specify Sample Specification template Sample specification - Draft
Early development (Text Selection) Project specific Specification Item specification - Text maps (Eng) Write Interview Schedule (Prompts ) NONE Project specific instrument
Writers Meeting Item specification - Text maps (Eng) NONE Interview schedule (prompts) peer Project specific instrument Amendments to be made to the
review instrument
Item Writing Item specification - Text maps (Eng) Draft Items (Eng internal) (Maths / Interview schedule (prompts) revised Project specific instrument Project specific instrument amended
Science external)
Develop mark scheme Items Draft item level marking scheme Instrument produced Project specific instrument amended Digital version of the instrument

Items entered by DTP Item specification DTP'd items in dummy tests Interview schedule / prompts agreed Project specific instrument amended Agreed interview schedule / prompts
by client
Informal Trialing DTP'd items in dummy tests Item performance in trials report Inform RDS of subjects used Agreed Sample specification Notiification
Internal Review Meetings Item performance in trials report Draft Performance report for client Timetables (Schedules etc) produced contract timetable Planned range of dates for fieldwork
review /reviewed
Expert review Items & Performance report for client Draft Performance report for client Contact subjects & arrange visits Planned range of dates for fieldowork Updated timetable of fieldworker
review review visits
Teacher Panel Items & Performance report for client Performance report for client review Fieldworkers meeting DTP'd instrument NONE
review
(Clients') Review Meetings Draft Performance report for client Agreed items for inclusion in pre-test Fieldwork visits DTP'd instrument Draft interview / observation notes,
review 1 - with amendments recordings etc
Items Revised Agreed items for inclusion in pre-test Final items for inclusion in pre-test 1 Write up fieldwork notes Draft interview / observation notes, Fieldworker report
1 - with amendments recordings etc
NONE NONE Agree report structure Contractural requirements - Report structure
Fieldworker report
Specify Sample for Data Services Specification template Sample specification - Draft analysis Fieldworker's reports Technical analysis
Sample Agreed Sample specification Agreed specification Synthesis Fieldworker reports - and digital compiled materials
materials, literature reviews etc
Specify Tests Specification template Test specification Write report Synthesis, report Templates - where Draft report
appropriate
DTP Produce Tests Test specification DTP'd test instruments Internal Review Draft report Agreed Draft report
Other related materials - instructions Templates - instructions, manuals Finished "related materials" Draft Report to Client Agreed Draft report NONE
Questionnaires, instruments etc etc
produced
Materials to Team Printed materials or digital files NONE Can go straight to "Finalisation" if NONE NONE
no Quantitative element
Team Reviews Compiled and produced test Agreed Test instrument Speci fy Sample (to RDS) Specification template Sample specification -
Draft the coding frame Mark scheme Draft Coding Framework Questionnaire writing (Design) Questionairre template where Project specific designed
appropriate questionairre
Print Authorisation Agreed Test instrument Print Authorisation update in Proj Questionnaire peer review including Project specific designed Amendments to be made to the
Mgmgnt system Focus groups , cognitive interviewing questionairre instrument

Confirmation of non-print Print Authorisation update in Proj NONE Questionnaire revised Project specific designed Project specific instrument amended
requirements, delivery and archiving Mgmgnt system questionairre
Requirements to Facilities (changes
from earlier process must have been
notified through Change Control
NONE NONE Questionnaire DTP’d Instrument specification DTP'd items in draft instrument
Print test materials Print specification, print authorisation Printed materials Questionnaire peer / RDS review, DTP'd items in draft instrument Amendments to be made to the
and DTP'd test instrument including Focus groups , cognitive instrument
interviewing
Accept print materials ready for Printed materials Despatch authorisation update in Questionnaire Revised DTP'd items in draft instrument Project specific instrument amended
despatch Proj Mgmnt system
Despatch tests List of schools involved in the tril - Despatch notes
addresses, contacts
Write Analysis Specification Specification Templates Draft specification Client agreement to Questionairre Project specific instrument amended Agreed Questionairre

Analysis Specification for Stats Analysis specification Agreed specification Produce final DTP’d Questionnaire for Draft Instrument and amendments Final CRC instrument
relevant media mechanisms
Approve Data entry Set-up Analysis specification Updated information Print Authorisation Agreed instrument Print Authorisation update in Proj
Mgmgnt system
Testing period NONE NONE Confirmation of non-print Print Authorisation update in Proj NONE
requirements, delivery and archiving Mgmgnt system
Requirements to Facilities (changes
from earlier process must have been
notified through Change Control
Tests returned NONE completed answer books Print questionnaire or produce Print specification, print authorisation Printed materials
relevant delivery mechanism and DTP'd test instrument

Marker training Batches of pre-test 1 answer Trained markers Despatch questionnaires or make live List of establishments involved - Despatch notes
booklets addresses, contacts
Mark & coding schemes Reviewed Draft Mark and coding schemes Revised mark and coding scheme Write Analysis Specification Specification Templates Draft specification
Marking Approved mark scheme - scripts Raw marks Analysis Specification agreed with Analysis specification Agreed specification
Stats
Data entry Raw marks Raw data Approve Data entry Set-up Analysis specification Updated i nformation
Data Preparation Raw data Prepared Data files Survey period NONE NONE
NONE NONE Questionnaires / data returned NONE NONE
Analysis Prepared Data files - Templates Draft analysis Coding Raw dat a files Coded data files
Write technical report - appendix Draft analysis - Templates Draft technical appendix Data entry or "Pulsetrain"! Raw marks Raw data
NONE NONE Data Preparation Raw data Prepared Data files
Consolidate the report Draft analysis - Templates Draft report - Level or standards Analysis Prepared Data files - Templates Draft analysis
setting report for National Curriculum
tests
Internal Review Draft report - Level or standards Final first pre test report Write technical report - appendix Draft analysis - Templates Draft technical appendix
setting report for National Curriculum
tests
Expert review Items & Performance report for client Draft Performance report for client Consolidate the Quantitative report, Draft analysis - Templates - draft Draft report
review review where appropriate Technical appendix
Teacher Panel Items & Performance report for client Performance report for client review Internal Review Draft report Agreed Draft report
review
(Clients') Review Meetings Draft Performance report for client Agreed items for inclusion in pre-test Draft Report to Client Agreed Draft report NONE
review 1 - with amendments
Report to Client Final report Final Client report NONE NONE
NONE NONE Clients Review Meeting Agreed Draft report Reviewed Client report - Actions for
the Actions log
Specify Sample (to RDS) Specification template Sample specification - Draft Review dissemination decision Proj Info System PHR detail on Enter updated PHR detail on
dissemination process decision* dissemination - auto inform
Communications
Sample Agreed Sample specification Agreed specification Prepare Final fully consolidated Draft reports incl client comments Final report
report
Specify Tests Specification template Test specification Write Research summary - includes Final Client Report Research summary and Key Points
"Key points"
DTP Produce Tests Test specification DTP'd test instruments Report to Client Report Template Final Client report
Other related materials - instructions Templates - instructions, manuals Finished "related materials"
Questionnaires, instruments etc etc
produced
Materials to Team Printed materials or digital files NONE
Team Reviews Compiled and produced test Agreed Test instrument
Print Authorisation Agreed Test instrument Print Authorisation update in Proj
Mgmgnt system
Confirmation of delivery and archiving Print Authorisation update in Proj NONE
Requirements to Facilities (changes Mgmgnt system
from earlier process must have been
notified through Change Control

NONE NONE
Print test materials Print specification, print authorisation Printed materials
and DTP'd test instrument

Accept print materials ready for Printed materials Despatch authorisation update in
despatch Proj Mgmnt system
Despatch tests List of schools involved in the tril - Despatch notes
addresses, contacts
Write Analysis Specification Specification Templates Draft specification
Analysis Specification for Stats Analysis specification Agreed specification
Approve Data entry Set-up Analysis specification Updated information
Testing period NONE NONE
Tests returned NONE completed answer books
Mark & coding schemes Reviewed Draft Mark and coding schemes Revised mark and coding scheme
Marker training Batches of pre-test 2 answer Trained markers
booklets
Marking Approved mark scheme - scripts Raw marks
Data entry Raw marks Raw data
Data Preparation Raw data Prepared Data files
NONE NONE
Analysis Prepared Data files - Templates Draft analysis
Write technical report - appendix Draft analysis - Templates Draft technical appendix
NONE NONE
Consolidate the report Draft analysis - Templates Draft report - Level or standards
setting report for National Curriculum
tests
Internal Review Draft report - Level or standards Final report
setting report for National Curriculum
tests
Report to Client Final report Final Client report
NONE NONE
Clients Review Meeting(s) - possibly Final Client report Reviewed Client report - Actions for
iterative the Actions log
Review dissemination decision Proj Info System PHR detail on Enter updated PHR detail on
dissemination process decision* dissemination - auto inform CMIS
Prepare Final Materials Reviewed Client report - Actions for Final assessment materials - to
the Actions log meet Client requirements
Finalise mark Schemes Pre-test 1&2 reviewed markschemes - Final Mark Scheme - to meet Client
Final assessment materials requirements

Hand Over to Client Final materials, final mark scheme NONE


Write standards report Outcomes of Pre-test 2, Anchor test Draft Standards report
reports, Draft report - Level or
standards setting report for National
Curriculum tests & PREVIOUS
YEARS REPORTS
Standard Setting Meeting Draft Standards report Final Standards Report
Write Research summary - includes Final Client Report Research summary and Key Points
"Key points"
Report to Client Report Template Final Client report

Figure 3: A comparison of the development of Quantitative and Qualitative


assessments / research at the NFER

Вам также может понравиться