Pick&Mix mapping into MIT90s

By Paul Bacsich1 This is a short report which shows how the Pick&Mix core and supplementary criteria (Bacsich, 2006a) map into the MIT90s strategic framework (Scott Morton 1991), put in the context of groupings from two other methodologies for benchmarking e-learning: BENVIC (Sangr et al, 2002) and Quality on the Line (IHEP, 2000). There is some evidence that Pick&Mix covers what UK universities and university-level colleges currently think is important in e-learning benchmarking, in that it was developed using justification for the various criteria as a result of a 2005 literature search (Bacsich, 2005) and has been through a series of refinements (Bacsich, 2006c): including a pre-pilot with one UK university in early 2005, feedback at several consultative workshops in later 2005 prior to the Benchmarking Pilot, and challenges and refinement from three UK universities within the Benchmarking Pilot. This public paper is an updated version of an internal report for the Higher Education Academy of work done in May 2006. The version of Pick&Mix used is Pick&Mix version 1.2. Conclusions 1. The MIT90s strategic framework appears to be a valid set of containers for Pick&Mix criteria. There are some situations where it is not completely clear which (of two containers) the criterion needs to go in hopefully case law from the (rather few) earlier MIT90s documents related to e-learning can answer this. There is evidence from Wills (2006) that she faced similar issues. 2. In support, BENVIC (of European provenance, 2002, based on research in 1999-2001) also handles Pick&Mix criteria quite well considering its date of origin and its origins in distance teaching. The containers (criterion groups) of BENVIC are similar to those found elsewhere, with the exception of Accessibility this last covers not only what in the UK we call accessibility but also the area of Widening Participation. It is assumed that the use of Accessibility as a separate container has more to do with the overarching political situation in European e-learning at the time of BENVIC (Virtual Campuses trying to justify their existence) than because of any deeper reason. 3. In contrast, mapping Pick&Mix into the Quality on the Line framework (of US provenance, April 2000, based on research done mainly in 1999) was more patchy. This seems to be because Quality on the Line (QoL) takes a very minimalist (and US rhetorical?) view of the institution as a community of scholars management is largely absent. It is strange (and out of line with other systems such as BENVIC, even of that era) that IT is not given a separate category is this because of the origins of QoL in the strong US distance learning tradition? This mapping approach tends to confirm that there is little to learn from QoL now, except to draw an audit trail to justify descendant criteria in other systems. However, it should be noted that at least one European research team were until recently using QoL (Lepori and Succi, 2003). 4. This work has been informative. In fact it was as a result of this work that the decision was taken to tag Pick&Mix criteria with MIT90s categories in Pick&Mix 2.0 (Bacsich, 2006b). 5. Thus it would be useful if those interested in developing criteria for MIT90s were to look at mappings into MIT90s from other methodologies in particular ACODE (ACODE, 2006) and ELTI (Beetham et al, 2003).

I am indebted to the UK Higher Education Academy for partial support of this work under the Concordance Project. Grateful thanks are also due to Peter Chatterton, my co-worker on some Concordance Project studies. Paul Bacsich file PicknMix-MIT90s-20060925

Pick&Mix mapping into MIT90s

Pick&Mix criterion 01 Adoption phase overall (Rogers) VLE stage level 5 statement All taken it up except some laggards One VLE MIT90s Individuals staff and students Technology (trumps Management Processes) 03 04 Tools use HEI-wide use of several tools Technology Technical Capability n/a Technical Capability n/a BENVIC Teaching Capability (not Technical Capability) n/a QoL

02

Technical Capability n/a

IT underpinning All services usable, with Technology internal evidence to back usability this up Accessibility e-learning material and services are accessible, and key components validated by external agencies Technology (trumps Individuals students)

05

Accessibility

n/a

06

e-Learning Strategy (longer-term)

Regularly updated eStrategy Learning Strategy, integrated with Learning and Teaching Strategy and all related strategies Effective decision-making Management for e-learning across the processes whole institution, including variations when justified

Institutional Capability

Institutional support (by implication, by extension from the technology plan) Course development (by implication)

07

Decision-making (for projects & programmes)

Institutional Capability

08

Pedagogic guidelines for Individuals staff Instructional Design/ Pedagogy the whole HEI, and acted on Management Learning material HEI-wide standards for learning material, which processes are adhered to and embedded at any early stage, e.g. by style sheets Training All staff trained in VLE Individuals staff use, appropriate to job type and retrained when needed Work planning system Management which recognises the main processes differences that e-learning courses have from traditional Activity-Based Costing being used in part Management processes

Teaching Capability

Course development (Q04) Course development (Q04)

09

Learning Development

10

Learning Development?

Faculty support (Q20)

11

Academic workload

Institutional Capability

n/a

12 13

Costs Planning (annual)

Institutional Capability Institutional Capability

n/a n/a

Integrated annual planning Management process for e-learning processes within overall annual planning and some links to IT and space planning Regular evaluation of all Management courses using a variety of processes measurement techniques and involving outside agencies where appropriate

14

Evaluation

Evaluation

Evaluation and assessment

Paul Bacsich

25 September 2006

Pick&Mix mapping into MIT90s

Pick&Mix criterion 15 Organisation level 5 statement MIT90s BENVIC Institutional Capability QoL Institutional support

Central unit has Director- Structures level university manager in charge and links to support teams in faculties Individuals staff (trumps Technology??)

16

Technical support All staff engaged in eto academic staff learning process have nearby fast-response tech support Quality and Excellence

Technical Capability Faculty support (Q18)

17

Adoption of some Management appropriate quality processes methodology (EFQM, etc) integrated with course quality mechanisms derived from QAA precepts Staff engaged only in or supporting the teaching process can reach high levels of salary and responsibility 99.9% Individuals staff (trumps Management Processes?)

Evaluation

Evaluation and assessment (by implication)

18

Staff recognition for e-learning

Teaching Capability

n/a

53

Underpinning IT/comms reliability Underpinning IT/comms performance for major e-learning systems

Technology

Technical Capability Institutional support

54

All e-learning systems operate in all uptime within documented and accepted response guidelines

Technology

Technical Capability Institutional support

55

Foresight-informed Both look-ahead and lab, External environment Institutional working in concert; at Capability development least one of these should agenda for ebe a sector leader learning IPR in e-learning IPR embedded and enforced in staff, consultant and supplier contracts Management processes Institutional Capability

n/a

57

n/a

58

Market research

Market research done External environment Institutional centrally and in or on Capability behalf of all Schools, and aware of e-learning aspects; updated annually or prior to major programme planning Competitor research done External environment Institutional centrally and in or on Capability behalf of all Schools, and fully aware of e-learning aspects Seamless integration with Technology uniformity of data formats, interface and response time

n/a

59

Competitor research

n/a

62

Level of integration of elearning with admin systems

Technical Capability Institutional support (by implication)

Paul Bacsich

25 September 2006

Pick&Mix mapping into MIT90s

Pick&Mix criterion 63 Level and delicacy of exploitation of students preexisting IT and information skills level 5 statement MIT90s BENVIC QoL Student support (but whole tone is what the HEI does to students)

Annual student surveys Individuals students Learning Delivery? and focus groups are used (trumps Technology?) to determine skill levels and this is taken into account for programme proposals and e-learning support All Departments operate an electronic system for the management and detection of plagiarism Management processes (trumps Students) Teaching Capability (or Technical Capability?)

65

Level, skill and delicacy of management of plagiarism

Faculty support (by implication)

66

Level of integration of elearning with physical space management Effective management of risks associated with e-learning

Integration of e-learning strategy, plans and decisions with space management process

Management processes

Institutional capability

n/a

67

Programmes integrate Management proactive risk processes management processes at all levels

Institutional capability

n/a

68

Amount and level Essentially RAE 4* of research leveraged on elearning operations Level of integration of elearning with Widening Participation agenda

External environment?

Evaluation??

n/a

70 C

Integration of e-learning Management strategy, plans and processes decisions with Widening Participation processes, within a framework of cost- and incomeawareness

Accessibility (which n/a includes WP)

91

Students have good Individuals students Learner Services Overall student understanding of e- understanding of the rules governing assignment learning submission, feedback, implications for plagiarism, costs, them attendance, etc and act on them Competence and Help Desk is deemed as timeliness of Help best practice Desk for students Technology (trumps Students?) Learner Services (or Technical Capability?)

Student support (by implication, but whole tone is what the HEI does to students)

92

Student support

94

Annual Student Individuals students Learner Services? Overall student satisfaction with e- Satisfaction survey which explicitly addresses the learning main e-learning issues of relevance to students

Student support (by implication)

Paul Bacsich

25 September 2006

Pick&Mix mapping into MIT90s

References (All URLs are current as of 24 September 2006.) ACODE (2006) Benchmarks for the use of technology in learning and teaching in universities (draft), Australasian Council on Open Distance and E-learning, August 2006, available at http://www.acode.edu.au/projects/acodebenchmarksfinal3may06full.pdf. Bacsich, P. (2005) Theory of Benchmarking for e-Learning: A Top-Level Literature Review, April 2005, available at http://www.cs.mdx.ac.uk/news/Benchmark-theory.pdf. Bacsich, P. (2006a) Benchmarking e-learning in HEIs the Pick & Mix approach: Second Update (13 April 2006) Release 1.2, April 2006, available at http://elearning.heacademy.ac.uk/weblogs/benchmarking/wpcontent/uploads/2006/04/eLBPilotPaulBacsich-rev2-final.doc. Bacsich, P. (2006b) Pick & Mix release 2.0 beta 1 release notes, August 2006, available at http://elearning.heacademy.ac.uk/weblogs/consultant1/wp-content/uploads/2006/08/PnM2-summaryrel1-release-notes.doc. Bacsich, P. (2006c) Higher Education Academy e-Learning Benchmarking Project: Consultant Final Public Report (on Pick&Mix and eMM pilots), August 2006, available at http://elearning.heacademy.ac.uk/weblogs/benchmarking/wp-content/uploads/2006/09/bacsichreport-public20060901.doc. Bacsich, P. (2006d) The relevance of the MIT90s framework to benchmarking e-learning, September 2006, accessible via a 25 September 2006 posting on the academy e-benchmarking blog at http://elearning.heacademy.ac.uk/weblogs/benchmarking/. Beetham, H. et al (2003) The ELTI Audit Tools, Institute of Learning and Research Technology, University of Bristol, February 2003, available at http://www.jisc.ac.uk/uploaded_documents/Workshop_pack_audit_tools_v5.doc. IHEP (2000) Quality on the Line: Benchmarks for success in Internet-based distance education, Institute for Higher Education Policy report, April 2000, available at www.ihep.com/Pubs/PDF/Quality.pdf. Lepori, B. and Succi, C. (2003) eLearning in Higher Education: Prospects for Swiss Universities; 2nd Report of the Educational Management Mandate (EDUM) in the Swiss Virtual Campus, University of Lugano, August 2003, available at www.ti-edu.ch/servizi/ricerca/papers/edum2.pdf. Sangr, A. et al (2002) Evaluation Methodology report: Benchmarking of virtual campuses, BENVIC, February 2002, available at http://www.benvic.odl.org/16_02_tot.pdf. Scott Morton, M. (editor) (1991) The Corporation of the 1990s: Information Technology and Organizational Transformation, Oxford University Press, ISBN 0195063589. This is exhaustively discussed in Bacsich (2006d). Wills, S. (2006) Strategic Planning for Blended eLearning, paper presented to the IEE conference ITHET06, Sydney, July 2006, available at http://ro.uow.edu.au/asdpapers/36/.

Paul Bacsich

25 September 2006