Lessons learned from 21 mini-projects that used the e-learning guidelines

John Milne, Amanda Gilbert, and Andrea Barr

Table of contents
Lessons learned from 21 mini-projects that used the e-learning guidelines.................................................................................1 John Milne, Amanda Gilbert, and Andrea Barr......................................................................1 Executive summary.........................................................................................3 Introduction 5 Methodology...................................................................................................5 Lessons learned from the 21 projects...........................................................7 Why use the e-learning guidelines?...............................................................7
Move the organisation forward in e-learning.....................................................................7 Ensure and enhance quality in e-learning .......................................................................7 Understand students better..............................................................................................8 Increase staff capability and awareness...........................................................................8 Provide a framework for conversations about e-learning..................................................9

How can you use the e-learning guidelines?...............................................10

Use a set of guidelines to improve organisational process ............................................12 Use a few guidelines to improve organisational processes............................................13 Use a set of guidelines to improve courses ...................................................................13 Use a few guidelines to improve courses ......................................................................13

What can you use the e-learning guidelines for? ........................................13

Better processes and new tools.....................................................................................14 Staff development workshops .......................................................................................15 Knowledge resources.....................................................................................................15 Customised guidelines...................................................................................................16

What factors should you consider when you use the e-learning guidelines? ......................................................................................................................16
Project design and planning ..........................................................................................16 Staff availability and commitment...................................................................................17 Student availability and engagement..............................................................................17 Communication..............................................................................................................18 Effects of guidelines on organisations............................................................................18 Changing priorities within the organisation.....................................................................19

Conclusion....................................................................................................19 Appendix A: Descriptions of the 21 projects..............................................21

Mini-projects that implemented a few guidelines............................................................21 Mini-projects that implemented a set of guidelines.........................................................25

Appendix B: Survey results about the effect of the e-learning guideline projects...............................................28 Appendix C: Risks identified across the 21 mini-projects ....................................................................29

Executive summary
In 2004 the Tertiary Education Commission (TEC) funded development of the e-learning guidelines to help organisations improve their e-learning practice. In 2007 the Commission funded a follow-up project to pilot the guidelines in 21 mini-projects, across 18 organisations. This report considers how these mini-projects used the guidelines, and shares the lessons learned. The evaluation team used four strategies to capture the learning from the miniproject contacts. These were: 1. 2. 3. 4. email questions face-to-face interviews a workshop survey a risk section in the two milestone reports that were part of the project.

The lessons learned answered the following four questions: 1. 2. 3. 4. Why use the e-learning guidelines? How can you use the e-learning guidelines? What can you use the e-learning guidelines for? What factors should you consider when you use the e-learning guidelines?

1. Why use the e-learning guidelines?

Project participants identified five reasons for using the e-learning guidelines. They wanted to: 1. 2. 3. 4. 5. move the organisation forward in e-learning ensure and enhance quality in e-learning understand students better increase staff capability and awareness provide a framework for conversations about e-learning.

2. How can you use the e-learning guidelines?

The mini-projects all implemented the guidelines differently, but their practice could be described by the model of Select, Implement, Review, and Share. The project identified the following four ways of using the guidelines: 1. 2. 3. 4. using using using using a a a a set of guidelines to improve organisational processes few guidelines to improve organisational processes set of guidelines to improve courses few guidelines to improve courses.

3. What can you use the e-learning guidelines for?

The mini-projects identified four improvements the e-learning guidelines might produce in an organisation. They are: 1. 2. 3. 4. better processes and new tools staff development workshops knowledge resources customised guidelines.

4. What factors should you consider when you use the e-learning guidelines?
The 21 mini-projects identified six factors to consider when using the e-learning guidelines: 1. 2. 3. 4. 5. 6. project design and planning staff availability and commitment student availability and engagement communication effects of guidelines on organisations changing priorities within the organisation.

The 21 mini-projects showed that using the e-learning guidelines can add value for individuals and organisations involved in e-learning. We hope that, by learning from the successes and mistakes in the mini-projects, other practitioners will see how the guidelines might help their organisation, will be encouraged to use the guidelines, and will be able to use them effectively.

Introduction
In 2004 the Tertiary Education Commission (TEC) funded development of the e-learning guidelines to help organisations improve their e-learning practice. The guidelines, which were designed to provide up-to-date, practical information and direction on e-learning for teachers, support staff, and managers, became available on the web in 2005. Practitioners were aware of the guidelines, but only a handful of organisations actively used them. The Commission therefore funded a follow-up project in 2007 to pilot use of the guidelines in 21 miniprojects. This report considers how these mini-projects used the guidelines, and shares the lessons learned. We hope that, by learning from the successes and mistakes in the mini-projects, other practitioners will see how the guidelines might help their organisation, will be encouraged to use the guidelines, and will be able to use them effectively. The mini-projects used the guidelines across 18 organisations1 to: improve organisational policies and processes for e-learning inform decisions about specific e-learning courses use technology more effectively for learning improve staff development for e-learning.

The 21 mini-projects are briefly described in Appendix A. More details are available on the e-learning guidelines website. (See http://elg.massey.ac.nz/index.php?title=Case_Study)

Methodology
Initially, we (the evaluators) intended to use an action learning approach to discover how to use the guidelines more effectively. We recognised that the project teams in the 21 projects would be learning throughout the project. Our plan was to share our knowledge as we went, and then apply this knowledge over the 12 months of the funded project. We also hoped that this sharing of experience would continue after the end of the formal project. However, this approach was slow to produce results, possibly because project teams were busy with their projects and didnt want to be distracted. The passive nature of this approach may also have been a factor. We decided to take a more direct approach by using the processes outlined below. Four processes captured the learning shared in this report. 1. The project team emailed the 21 mini-project team leaders to ask about their experience in implementing the e-learning guidelines. We hoped that responses to this request would show how implementation can proceed successfully. Nine team leaders replied. 2. Two independent evaluators interviewed 12 of the 21 project contacts. These 12 were selected to include examples from each of the different ways the guidelines were used. The questions aimed to understand: a. each projects rationale and processes in more detail b. experiences that interviewees believed would be useful to others
1

Christchurch Polytechnic Institute of Technology, Waiariki Institute of Technology, and the University of Canterbury each ran two projects.

c. whether the interviewees believed the guidelines had been well disseminated or embedded within their organisation. After each interview, the evaluators analysed the transcripts and developed a diagram of the process the project had used. From the 12 process diagrams and transcripts, they developed process models and identified lessons learned. 3. Nineteen project contacts completed a survey at an end-of-project workshop. The survey asked respondents whether they believed their mini-project had achieved their overall objectives, and to give examples where relevant. Appendix B lists the survey questions and results. 4. Each project identified three risks in their two milestone reports (October 2007 and February 2008). Appendix C categorises the risks identified across the 21 projects.

Lessons learned from the 21 projects

Why use the e-learning guidelines?
Project teams identified five reasons for using the e-learning guidelines. They wanted to: 1. 2. 3. 4. 5. move the organisation forward in e-learning ensure and enhance quality in e-learning understand students better increase staff capability and awareness provide a framework for conversations about e-learning.

Move the organisation forward in e-learning

A number of project leaders talked about the importance of e-learning in their own organisations. The groups believe that e-learning is a key part of any tertiary education agenda. Participating in any project about e-learning is therefore seen to be beneficial to the organisation. Some project leaders feel that the e-learning guidelines are a useful way to drive change within their organisations and that they provide an appropriate structure, or framework, for change because they reinforce organisational visions for e-learning. Generally, we didnt expect that organisations would implement the guidelines as written, but that they would use the concepts, and tailor the guidelines to their particular needs. Comments:
[the guidelines] will drive change (technological, pedagogical and attitude) towards e-learning.

will provide a focus point for the college vision of e-learning

will give a framework to those using e-learning strategies

[are] a useful tool to use with staff as we move towards more flexible delivery

[are] a foundation for staff to work on

support(ing) the implementation of a formal curriculum development process

[will] provide a new set of Mori-based guidelines modelled on current e-learning experiences.

Ensure and enhance quality in e-learning

Improving the quality of teaching was one of the two most common advantages identified by project teams (the other is increasing staff capability and

awareness). The framework for e-learning provided by the guidelines is a sound basis for examining the quality of an organisations provision. A number of project teams felt that the guidelines would be useful for identifying quality issues. Comments:
[the guidelines] will help ensure consistent quality across the institution

[are] a useful tool with which to review our processes and policies

will be made visible to staff through our conceptual framework and Quality Management Systems, policies and processes

raise awareness about quality

The project work helped organisations to embed good practice in e-learning by raising awareness of the guidelines, encouraging the organisation to take ownership, reformatting them for their own teaching and learning context, and testing them through programme review and development processes to improve the quality of the student learning experience. Everyone who responded to the end-of-workshop survey said that the project helped their organisation to ensure higher quality e-learning (see Appendix B, Question 1). These improvements were the result of introducing or refining the quality processes, or because staff believed that high-quality e-learning relates to effective professional development that, in turn, results in better teaching and learning. The respondents recognised the value of the e-learning guidelines for professional development.

Understand students better

Some projects developed a better understanding of their students through the project evaluation. They gained information about student engagement with learning, how students use computers, and how they perceive the value of a course. In one case, this learning was used to develop a model for e-learning enhancement. This model is now a core process that can be applied to a range of courses.

Increase staff capability and awareness

Supporting staff in using e-learning was the other most common advantage seen by project teams. The guidelines were considered to be helpful in setting agendas for staff development and identifying best practice. The guidelines website was also useful as a repository for good practice examples. Project leaders mentioned the need for teachers to increase their understanding of e-learning, and to connect their existing knowledge with new ideas. They also spoke positively about the possibility of a learning community where they could share experiences and develop collaborative projects. Comments:
[the guidelines] will underpin future staff professional development in e-teaching based on best practice

 will inform the creation of an online e-teaching resource

[are a useful tool] to use with staff when looking at redevelopments

we will use [the guidelines] as a repository of good practice linked from our website

[are] valuable to colleagues in developing new ideas and practice, or in making connections between what they may be familiar with and what is new

develop [teachers] ideas to see existing resources in new ways

can be used by staff (and students) on our other programmes

The work done with tutors who taught the courses selected for the project was targeted professional development that increased their capability in learning design for greater student engagement. Through these projects, staff have engaged with the e-learning guidelines, and this has resulted in raising awareness of them.

Provide a framework for conversations about e-learning

By using the guidelines to select a dimension to focus on and then implementing research activities, the teams had opportunities to try interventions, test the guidelines, collect data, see if the guidelines were working, and make any necessary changes. This proved to be an effective framework for developing the model for enhancing specific guidelines. Comments:
The e-learning guidelines provide a framework for a conversation about e-learning quality or the way a course is run that is extremely useful for people at all levels of understanding of e-learning. (Interview 2)

The value of the guidelines is in the questions that they wouldnt have thought to ask themselves [and] which give them enough information [so] they can start to come up with a sensible answer in their own environment. (Interview 4)

With the speed of uptake of e-learning activities we need to have those guidelines in place, well understood to demonstrate to people that they can use whats there and they can add their own stuff. We need to get visibility of the guidelines up a lot higher. I think the guidelines provide a framework which is overarching to the eMaturity Model. The guidelines can be used for organisations as starting questions if they want to improve the pattern of their e-MM results. (Interview 5)

How can you use the e-learning guidelines?

The e-learning guidelines provide a framework for organisations to improve e-learning quality. They help organisations to inform tertiary staff of good pedagogical practice, help them to design e-learning, support quality e-learning activity, and help staff to reflect on their e-learning practice. To achieve these goals the mini-projects used the guidelines at a course level or for organisational processes. In each approach they used either a set of guidelines or selected just a few of them.

Select Develop a team Identify key priorities for the context Review guidelines and decide on the number of guidelines and approach

A few guidelines

A set of guidelines

A course level approach

An organisational level approach

Implement Obtain resources and a budget Review what others have done Develop a plan Implement the plan and act on findings

Review and share Evaluate the effect of the guideline(s) Keep your organisation informed of the project Share the information with others at the e-learning guidelines website

10

Figure 1: Summary of approaches taken by the mini-projects

11

The mini-projects all implemented the guidelines differently, but their practice could be described by the model of Select, Implement, Review and Share (Figure 1).

Select
The selection phase included planning. At this stage, the mini-projects considered the local context and reviewed how the guidelines would help to improve the quality of e-learning. All of the mini-projects had a group that included teaching staff, support staff, and senior managers (who sponsored the project). This group considered the organisation, its context, and key issues of elearning quality. In their initial review of the guidelines, mini-projects decided whether they would implement a set of guidelines or a few guidelines. Those that used a set of guidelines used them to review practices or processes. Those that selected a few guidelines targeted a defined area and worked on improving that. In both cases, the guidelines were a tool to guide improvements. It is an important feature of the guidelines that people take greater ownership when they localise them for a particular context. Mini-projects focused either on a course level or organisation level. Those that worked at course level had direct contact with students. Data was collected from students, showing the effect of the changes that resulted from implementing the guidelines. Those that worked at an organisational level put in processes to help staff provide high-quality e-learning.

Implement
In the implementation phase, teams formalised the plan and then carried it out. Because all mini-projects had external funding through the evaluation of the overall project, they had plans with milestones. This helped the projects progress.

Review and share

The mini-projects reviewed the process of quality improvement and shared their findings at the e-learning guidelines website through case studies and in exemplars associated with the guidelines. The mini-projects also shared findings within their organisations.

Use a set of guidelines to improve organisational process

Some mini-projects worked on improving organisational processes by using a set of guidelines. This work was often integrated into the organisations policies. Examples from the mini-projects include using the guidelines as a basis for developing tools for the course design process, and implementing a selection of guidelines to improve staff development practice. Although they differed in detail, they all used a three-step process. 1. Selecting relevant guidelines for the organisation: This selection involved discussion with key people at different levels of the organisation. The mini-projects sometimes localised the guidelines in light of the discussions, which included identifying criteria to determine the evidence needed to show that the guidelines were achieved. Senior managers sponsored the projects and helped to ensure the outcomes were integrated into organisational policy. 2. Implementing the set of guidelines: The mini-projects applied the

12

guidelines to identify any gaps between current practice and the guidelines. This resulted in literature reviews that informed the work and staff development to help support staff with the new processes. 3. Reviewing and evaluating the effect of the changes: Part of the process was to share the findings of the mini-project through a case study on the e-learning guidelines website.

Use a few guidelines to improve organisational processes

Some of the mini-projects focused on organisational level improvement by implementing a few guidelines. These projects worked on guidelines that were important to their organisation. Examples of the issues they worked on included staff development, and copyright and intellectual property policy. The implementation phase included completing a literature review, and developing a resource or some research into implementing the guideline. The final stages were reviews of the processes and the development of material to share on the guidelines website.

Use a set of guidelines to improve courses

Some mini-projects selected a set of guidelines to refine a particular course. The first step was to review the guidelines to identify those that were relevant to the course. This required discussion with key people, and then adapting the guidelines in light of the discussions. The discussions included identifying criteria to determine the evidence that would indicate that the guidelines have been met. The second step was to apply the guidelines to identify gaps between them and current practice. The third step was to review and evaluate the effect of the changes. Finally, they shared the information with the tertiary community.

Use a few guidelines to improve courses

Some mini-projects used a few guidelines to improve courses. Their first step was to select one or a few guidelines that reflected issues for a particular course. To do this, they identified key issues and then reviewed the guidelines to identify those that covered that particular area these included guidelines on student persistence and guidelines on student choice. Other mini-projects developed guidelines on using technology in courses examples included guidelines to support language social networking sites, guidelines for virtual guest speakers, and guidelines for using mobile technology. The project teams that designed these mini-projects chose the technology based on its potential for students. The projects were linked to overarching e-learning guidelines within the project. The implementation step for these projects started with a review of the information about the guideline on the e-learning guidelines website. Further general information was collected in literature reviews. The team then identified a course or courses that could be adapted and evaluated. Feedback from students by discussion or questionnaires provided a snapshot of the initial status. Data from students informed the process, highlighted successful instances of using technology, and indicated areas for change. Changes were made to the course based on this information. The mini-projects shared their experiences in case studies and updated the exemplar information on the guideline.

What can you use the e-learning guidelines for?

The project leaders report that the e-learning guidelines have been a useful starting point for staff at their organisations. Their projects have initiated new

13

quality processes or developed the knowledge base about the guidelines. Improvements to the guidelines themselves included refining them as a result of feedback generated when the guidelines were used in practice. Project leaders who worked on a few specific guidelines spoke in detail about developing the knowledge base around the guidelines. Project leaders also spoke of how others can learn from their experiences. The elearning guidelines provide a platform for staff development, are a source of information, and are also a tool to help develop processes. Only a few of the projects focused on staff development. However, others recognised that the guidelines provide a platform for staff development so staff involved in their projects can discuss quality in e-learning. Other project contacts spoke of how their projects and case studies demonstrate how to use the guidelines. Project contacts spoke of sharing processes or information with the tertiary community. Some projects used the e-learning guidelines to develop processes that they documented and shared. The mini-projects identified four improvements the e-learning guidelines might produce in an organisation. They are: 1. 2. 3. 4. better processes and new tools staff development workshops knowledge resources customised guidelines.

Better processes and new tools

The guidelines can be embedded in the organisation in a number of ways. They can be used as tools for reviewing e-learning policies and procedures that may, for example, be identified as needing improvement through an e-Learning Maturity Model evaluation. Some mini-projects resulted in processes and tools to help with the quality of elearning. They localised the guidelines, rewording and developing criteria that were relevant for that organisation. The organisation was reassured that their capability would be robust because the guidelines have been developed from a strong research base. Some mini-projects developed resources and guidelines on using the technology within a well-grounded teaching and learning context. The project teams focused on their particular guideline or guidelines, and the evaluation data collected from students identified the success of the considered use of these technologies. The projects also exposed those within the project to the wider guidelines and how they could be used. Comments:
The e-learning guidelines support and guide people. They can be built into processes where they can be very useful if they are used at the start of the process. The e-learning guidelines are written as questions so they can be used to drive and embed best practice. They can be helpful as ways of working with academic managers to get buy-in to processes. (Interview 3)

going through guideline by guideline saying, How is the course already doing this or how might we do it better?. So we did really use them as a benchmark for what is happening but also as a prompt for how could we do things better in terms

14

of the guidelines. (Interview 2)

the idea is to actually use the e-learning guidelines to propose that [that] is the framework [they will] use to evaluate the tool that they look at or actually implement. (Interview 2)

Staff development workshops

Workshops provided professional development to help staff to refocus on good teaching and learning practice. Extensive consultation with staff raised their awareness and understanding of the purpose of the guidelines and their usefulness. Consultation also promoted ownership of areas of responsibility. The staff development workshops provided a way for the organisation to share information about the guidelines and then to get feedback so they could be localised for the organisation. Participants spoke of the sense of ownership that developed from this consultation. The workshops provided further professional development to help staff to refocus on good teaching and learning practice. The project that took a more focused approach identified that online methods of staff development are more effective than a face-to-face approach when the staff have a good understanding of IT, work well in groups, and can work independently. Comments:
[the guidelines] should be recommended or required reading. For people just to hold that in front of them. I will be using them in one of the courses I teach. (Interview 1)

Knowledge resources
The tertiary community can use the e-learning guidelines website for help and information. They can explore the guidelines, find guidelines that relate to specific topics, and search a wide range of information about the guidelines. The website allows people to add or edit information, so the community can develop their expertise in using the guidelines. A number of projects implemented one or a few of the guidelines. These projects had an opportunity to work within their organisations and also to share ideas with the wider community through the e-learning guidelines website. They shared case studies about implementation, providing an overview of what they had done and explaining how to implement a few guidelines at any one time. It was important for these organisations that this work improved their capability to use e-learning. Projects have developed resources and examples for policy makers, teachers and e-learning specialists. Comments:
Use them, read them. The case studies add value too. (Interview 1)

I can see that having gone through this project it doesnt matter what level you are at its just a really useful framework for a conversation about the quality or the way

15

a course is being taught so I would say they are extremely useful. (Interview 2)

Customised guidelines
It was important for some mini-projects to customise the guidelines and create a local set. This included changing the language and, often, reducing the number of guidelines but also adding new ones. This process included consulting staff about what the guidelines should be. The wnanga wrote new guidelines from a Mori perspective. The new guidelines were based on analysis of current good practice in the wnanga and the established guidelines. These guidelines are supported by two case studies one about a course that is being taught using the Mori guidelines and the other about staff training and learning. The content of the guidelines is based on the organisations leadership in good practice in Mori learning and teaching. The project is the beginning of a process to embed the guidelines into the wnanga. Staff and student participation in the surveys has raised awareness of what constitutes good practice. Staff capability has been strengthened through the professional development provided by the project activities. The student learning experience has been enhanced by integrating tikanga Mori such as waiata, karakia, and whanaungatanga, and an understanding of how online learning can facilitate their learning. The approach has been supplemented with hybrid terms that mix English and Mori words.

What factors should you consider when you use the elearning guidelines?
The 21 mini-projects identified six factors to consider when you use the elearning guidelines: 1. 2. 3. 4. 5. 6. project design and planning staff availability and commitment student availability and engagement communication embedding changing priorities within the organisation.

Project design and planning

Nine projects mentioned project design issues as a risk. Specific issues related to the projects aims and objectives, the way the project was designed, and the plan to implement the project. Project team leaders mentioned the need to have flexible project designs and realistic timeframes. Concerns included a lack of clarity when projects were passed to people who did not write the original project plan. The new project contacts spent time identifying what they had to do and discussing the project with others. Some were concerned about the size of the project, keeping the project on track, and not having other tasks added into the project. Some of these design issues, such as legal risks or the risks of not having a model specific to the project, relate to specific projects. There was a reduction in design issues between the first and second milestone reports (October 2007 and February 2008). Those that remained were standard project concerns listed in the first milestone in October, or were design issues that were relevant only to that project.

16

Financial issues fell into two categories. The administration systems in some organisations had difficulty transferring and receiving the project funds. Others mentioned underestimating the budget. Lack of senior management involvement was mentioned by only one project. The mitigation was to meet with the senior managers to discuss how the project could be best integrated into the organisations processes. The way the risk was mitigated gives a good demonstration of the value of the risk process. Some projects mentioned technological risks such as the theft of hardware, or using software that is not well supported. Several talked about the risk of ethics not being approved by the organisation, and the work not being able to proceed. Comments:
delays in getting ethics approval getting ethics committee to respond quickly enough major problems have been around research processes [getting ethics approval] meant we could not capture staff before the X mas break

Staff availability and commitment

The most commonly listed risk for staff was workload. In most cases, project members were employed as permanent support staff. In some cases, project members were teaching staff, or the project relied on the involvement of teaching staff. Staff generally juggled the e-learning guidelines project with their core roles and, therefore, lacked time for the project. To be successful, the project teams had to recognise the changed workload, reallocate responsibilities, and carefully plan to complete the work. Other issues identified were staff resignations, the difficulty of employing temporary staff for the project, the time it took for staff to absorb the new information and learn the skills required, and some staff proving unsuitable. Comments:
competing priorities, retaining staff to undertake key tasks the time colleagues have to learn about new technology time available to colleagues to attend workshops/presentations

Fewer projects identified workload as an issue in the February 2008 milestone report, indicating that these issues were more significant earlier in the project. This early concern probably relates to a lack of project design and planning.

Student availability and engagement

Six projects had difficulty motivating students to provide feedback in surveys or focus groups. One project had problems getting access to courses this is an organisational issue. Seven projects mentioned timetabling challenges through the academic year these made it difficult to gain access to students at times that suited the project timeline, such as during student breaks. More mini-projects voiced concern in the February milestone report about accessing students and motivating them to provide good feedback. This difficulty

17

reflects the number of projects that were collecting feedback from students at this time. These two issues, and the issue in the section above of gaining ethical approval, show the importance of planning realistic timelines and being aware of other activities and policies across the organisation. They also demonstrate the challenge of gaining access to students and ensuring that they give the desired feedback.

Communication
Five projects identified the following communication issues: communication with other parts of the organisation about the project and what it will deliver communication and contribution by the people in the project team buy-in and understanding by staff.

Effects of guidelines on organisations

The 12 interview participants were asked for evidence of how the e-learning guidelines affected the organisation. Evidence was examined at two levels: the first was evidence of dissemination so that people within the organisation are at least aware of the guidelines. The second was embedding the guidelines.

Disseminating the guidelines

One project considered the project had had a significant effect throughout the organisation. This project had a very collaborative process that was designed to develop ownership. They consulted widely with staff about the project and asked for advice on the project activities. They organised a series of workshops to introduce staff to the guidelines and sought feedback on what participants found useful, who had responsibilities for areas, and thoughts about how to fit this work into their internal processes. Workshop participants and e-learning champions helped to spread the message about the project and how the guidelines should be used. Information was also sent to programme managers, the academic manager, quality services, ICT, flexible learning, the strategic planning team, and learning support. They gave a presentation to the senior management team and distributed relevant guidelines at this meeting. The participants described dissemination as ranging from being restricted to the project team, to creating wide awareness throughout the organisation. Generally, the projects run in smaller organisations had a greater effect across the organisation. At a lower level of dissemination, six projects had a local effect on the project team, which included some staff in the departments that were directly involved in the trials. These organisations outlined plans to raise awareness further. One participant said that they were not keen to promote the work until their project was complete and there was something substantial for those in the organisation to use. At the next level of dissemination, four participants described similar effects on the project team and others directly involved, but other staff members got information about the project. These staff members included other teaching staff, students, senior managers and other groups such as the ethics committee.

Embedding the guidelines

The 12 interviewees were also asked about evidence of a second level of effect; that is, embedding the e-learning guidelines in the organisations processes. This

18

is a more rigorous measure of effectiveness but, paradoxically, if the guidelines are embedded as usual practice, staff may not be aware of them. All interviewees described how the e-learning guidelines have been embedded in their organisations, and they also outlined how their project outcomes were embedded and localised. Some of the projects described a wider effect on their organisations. These outcomes were new or revised policies or processes such as e-learning policy, or using the guidelines to feed into the overall teaching and learning policy. One participant explained that the guidelines have been used for more than e-learning they have been used to underpin the structure of the teaching and learning strategy for the whole organisation. Another participants organisation is considering how to measure the success of the adoption and ownership of guidelines within each department, and evaluating their effect on successful learning. Professional development was used to encourage staff to use the guidelines in teaching and learning decisions, and for selecting appropriate tools for learning.

Changing priorities within the organisation

In some cases, organisational change directly affected the projects. In one example, the courses selected as the focus for the project were not offered for study. Comments:
the at risk courses we chose are not being offered this year or are at the wrong time [it] means more work than anticipated

In others, organisational priorities changed, and urgent activities became the priority. One organisation was being restructured it was difficult to work on processes that were under review and were likely to change.

Conclusion
The 21 mini-projects demonstrate that the e-learning guidelines can inform people and enable them to provide effective e-learning practice. The miniprojects used the guidelines in different ways, but all mini-projects supported staff to help them make good decisions about e-learning. The focus of using the guidelines is on student outcomes rather than management compliance and control. The overarching approach to the guidelines has been to focus on students or, more importantly, their learning. Implementing the guidelines is a complex process, and staff need careful management and support. It is one thing to have a set of e-learning guidelines, but it takes consideration and effort to know how to achieve the standards they describe. To ensure high quality, the guidelines are based on pedagogical principles. By applying the guidelines, the teams on the mini-projects actively considered quality in e-learning. The mini-projects report many benefits, including improving organisational policies and processes for e-learning, informing decisions about elearning courses, and using technology more effectively for learning and improving staff development. The ultimate aim is to for learners to be attracted to the courses and achieve the learning outcomes. The project teams consider

19

this is now more likely to occur.

20

Appendix A: Descriptions of the 21 projects

The 21 projects have implemented the e-learning guidelines in a number of ways. In this appendix the projects have been divided into two groups: those that implemented a few guidelines, and those that implemented a set.

Mini-projects that implemented a few guidelines

Auckland University of Technology (AUT)
Contact: Andrew Higgins Project title: The implementation of the e-learning guideline on staff development Guidelines: T07 What ICT support is in place specifically for e-learning, and is this support reviewed regularly? AUT studied how effectively staff retain information in an online setting in comparison with an interpersonal training programme. Staff training programmes were revised as a result of assessing the effectiveness of strategies, and staff members views. It was found that non-traditional online methods of staff development are more effective when staff have a positive view of e-learning technology, are competent at working in groups, and are confident independent learners.

Bay of Plenty Polytechnic

Contact: Amy Wilson Project title: Guidelines for effective activities with virtual guests Guidelines: SD3 Do students gain knowledge relevant to employment and/or current thinking in their field? ST9 Do the technologies employed help students to successfully meet the learning outcomes? The Bay of Plenty Polytechnic project examined whether activities that use virtual guest speakers are more, or less, effective than those that use speakers who are physically present. The project developed guidelines for virtual guests.

Christchurch Polytechnic Institute of Technology

Contact: Nicki Page Project title: Creativ8 Fostering and managing creative and innovative e-learning ideas across a tertiary institution: The implementation of an e-learning guideline Guideline: MO16 Does the institution support creativity and innovation in e-learning? This project developed an open-source web-based tool that can be set up through an organisations network to share innovative e-learning ideas. The Creative8 application was trialled and is freely available.

Christchurch Polytechnic Institute of Technology

Contacts: Nick Ford, Nicki Page Project title: Multimedia linguistic learning objects for science students: The implementation of two e-learning guidelines

21

Guidelines: ST5 Have activities been identified that allow individuals and groups learn through experience, including opportunities to demonstrate, practice skills, reinforce knowledge and develop understanding? ST9 Do the technologies help students successfully meet the learning outcomes? This project had four stages: design, production, implementation, and refinement of a number of interactive multimedia learning objects based on the lessons learned in the implementation. These learning objects provide science students with practical experience when using scientific language, so improving their knowledge and confidence.

Manukau Institute of Technology (MIT)

Contacts: Oriel Kelly, Helen Anderson Project title: The implementation of e-learning guidelines to support student persistence and success Guideline: SD8 Is there a mechanism in place to assess how students perceive the value of the course? The MIT project worked on courses in which students showed little persistence and had low retention and success rates. Plans for technological improvement were based on the results of student questionnaires. New materials were then designed and developed. A follow-up questionnaire showed students had better completion rates and were more satisfied. Teaching staff also benefited from enhanced knowledge and skills, and the new approach (using student surveys and developing new material) is available for staff to apply to other courses.

Massey University
Contacts: Cynthia White, Ute Walker Project title: Accessing global perspectives through eTandem exchanges online Guidelines: SD6 Do online discussions enable student-student collaboration? ST4 Does the course require students to engage themselves in analysis, synthesis, and evaluation as part of their course and program requirements? The school of language studies at Massey University undertook a project involving eTandem projects on the internet. The eTandem projects allow students to practise their target language with native speakers, exchange points of view, and obtain feedback. The project developed guidelines and links to eTandem learning opportunities.

The Open Polytechnic of New Zealand

Contact: Sue Dark Project title: The implementation of an e-learning guideline on effective collaborative approaches to online learning Guidelines: TD9 What strategies in online discussions will build rapport with students? TD10 Should students present work using online discussion tools? TD11 Should staff use a team approach to develop and teach the course? The Open Polytechnic of New Zealand used research-based findings to develop a staff training programme. A case study was developed to help staff development.

22

Otago Polytechnic
Contact: Bronwyn Hegarty Project title: The power of design on flexible learning and digital network literacy Guidelines: TD11 Should staff use a team approach to develop and teach the course? TD12 Is the design of learning informed by research on effective e-learning? TO9 Are staff encouraged to participate in networks and learning communities involved in reviewing, developing or sharing good practice in the use of elearning? The Otago Polytechnic project supported the development of a course called Designing for Flexible Learning Practice (DFLP), which was part of a graduate certificate programme. The intention was to provide a wide range of educational options, practices, and models for developing flexible learning. One major outcome of the project was three case studies of design these case studies were described against the e-learning guidelines. When interviewed, staff who taught on the course indicated that they have changed their teaching by using a wider range of technologies, making technology a greater part of everyday practice, and sharing ideas and expertise with peer networks. Their confidence in using educational technologies also increased after they took part in the course. The learners benefited from the modifications that improved the course. Practitioners in the wider e-learning community have benefited from the development of a model of course design that incorporates open educational resources, methods to enhance self-efficacy for online learning, and strategies to promote digital network literacy in learning and teaching.

University of Canterbury - Education

Contacts: Julie Mackey, Elaine Mayo Project title: Constructing and assessing relevant knowledge within an ecommunity: Experiences from the community that surrounds the qualifications in Applied e-Teaching and Support. Guidelines: SD1 Are students able to agree some or all of their learning goals in negotiation with teaching staff? SD2 Do students have any choice in terms of either: what they learn, the learning activities they will engage in, the particular resources they will study, and/or assessment? This project used the applied e-teaching and support qualifications developed by the Canterbury Tertiary Alliance. It evaluated and refined e-learning processes that increase the flexibility of learning options for students. It showed that students and staff endorse e-learning that allows students to negotiate learning goals, and allows student choice. To be effective, flexibility and choice need to be clearly defined and carefully introduced, and students must be well supported. Students need to have sufficient learning maturity to take responsibility for their own learning.

23

University of Canterbury
Contacts: Tim Bell, Bill Rosenberg Project title: Guidelines for using podcasting in a tertiary environment Guidelines: The project used a variety of guidelines The University of Canterbury study assessed the effectiveness of using podcasts as a teaching tool in a tertiary environment. The project designed guidelines for educational podcasting, developed a sustainable model for using the technology, and evaluated the flexibility of the medium in different courses. Two main groups for podcasts were assessed: students enrolled in specific courses, and high-school students preparing for university. For each group, the effectiveness of podcasting was assessed and guidelines developed.

Unitec
Contact: Thom Cochrane Project title: Innovation in e-learning: The potential of wireless mobile devices and Web2 Guidelines: TD12 Is the design of learning informed by research on effective e-learning? TD13 Do students get opportunities to practise meaningful and relevant online research, if appropriate for the course? SD7 Is there a defined process to follow that identifies the feasibility of and appropriate delivery modes for the course? The project developed innovative ways to integrate wireless technology into teaching and learning at Unitec. The core activity was to write and maintain a reflective blog. The main tool for posting on the blog was a smartphone, which was used with mobile-friendly web tools. The trials resulted in a move towards a social constructivist way of learning and positive improvements in student output and engagement. Both students and staff want to use more technology within their courses.

Victoria University of Wellington

Contact: Stephen Marshall Project title: Implementation of e-learning guidelines on intellectual property and copyright Guidelines: TD14 Are e-learning materials within the boundaries of copyright and licence agreements? MO19 Does the institution have policies and agreements on ownership of intellectual property relating to e-learning? MO22 Are processes in place to ensure legal requirements of copyright and intellectual property? Victoria University of Wellington developed a learning object on intellectual property rights within e-learning. This learning object supports managers and teaching staff who are involved in working through problems relating to intellectual property rights in e-learning.

24

Waiariki Institute of Technology

Contact: Jens Hansen, Leonie Nicholls Project title: The implementation of e-learning guidelines to ensure that students receive targeted guidance and study skills and support required for their e-learning experience Guidelines: SO8 Do students get guidance on study skills for the e-learning environment? SO9 Are there proactive and well-publicised strategies in place for student support? The Waiariki Institute of Technology project identified the study skills that Mori students need to learn effectively in an e-learning context. A literature review and interviews with students have been completed. The institute is to translate an already established and effective face-to-face mentoring programme into an effective online learning programme.

Mini-projects that implemented a set of guidelines

Career Force
Contact: Claire Derham-Cole Project title: What is an effective e-learning framework for Industry Training Organisations? Guidelines: The project used a variety of guidelines Career Force developed an e-learning framework for the organisation. The framework is based on an investigation into how workplaces can be supported with e-learning. The project looked at workplace attitudes to e-learning, what would be needed to enable e-learning, and how e-learning frameworks can be implemented. This information, together with the guidelines, was used to develop an e-learning plan for the organisation. The plan used a combination of aspects including managing trainees, on-the-job embedded learning and assessment, face-to-face sessions in the workplace, live webcasts and podcasts, performance support, collaboration and community, multimedia archives with CDRom, and web-based learning.

Eastern Institute of Technology

Contact: Joyce Seitzinger Project title: Adapting e-Learning Guidelines into Review and Evaluation Toolsets Guidelines: The project used a variety of guidelines The EIT project developed a tool to help managers select e-learning programmes. This was done by creating an e-learning prioritisation tool that helps to rate factors such as teacher readiness, course readiness, and project suitability. The tool uses five categories each rated from 1 to 5. Each category has criteria that are also rated from 1 to 5. This tool can help to allocate scarce resources, and encourage staff to talk about e-learning factors. It can also be used as a tool in conversation between e-learning advisors and managers. Because the tool is scalable, it can be used in courses or programmes.

25

New Zealand Tertiary College

Contact: Selena Fox, Julie Whittaker Project title: Staff Development Framework for e-Learning: An Early Childhood Teacher Education Perspective Guidelines: The project used a variety of guidelines Relevant e-learning guidelines were selected and then applied to operations. Staff were consulted on the e-learning guidelines and their implementation into academic staff development. A course was developed for the academic and support staff, and then it became part of the professional development programme. A new policy was formulated and added to the NZTC quality management system. There is college-wide endorsement of, and support for, the e-learning guidelines.

Northland Polytechnic (NorthTec)

Contact: Toni Horrell Project title: Implementation of e-learning guidelines Guidelines: The project used a variety of guidelines NorthTec localised the guidelines that were relevant to them. They provided staff training on the guidelines, and then introduced the guidelines when developing a new programme by making them part of the quality management system. As a result, the NorthTec e-learning guidelines support all staff involved in e-learning. The guidelines are also the basis for the criteria for the excellence in teaching awards.

The University of Auckland

Contacts: Cathy Gunn, Adam Blake Project title: Promoting quality e-learning practice in tertiary teaching Guidelines: The project used a variety of guidelines The project used student-based e-learning guidelines to evaluate and design an e-learning element for a new post-graduate certificate in academic practice. The e-learning component of the course recognises the importance of academics professional development in e-learning. The project set up an e-learning course using components such as threaded discussion for peer critique and reflection, and a new computer lab-based task to develop greater insight into how learners use technology. Students used the elearning guidelines to guide and justify a proposal to use e-learning to enhance teaching and learning in their own context. Measuring the student-focused e-learning guidelines against the courses generated ideas for enhancing quality and allowed benchmarking of pedagogical and organisational aspects of the existing courses. The project gave students greater direct experience with technology for teaching and learning, so they can confidently implement these in their own practice. Student learning was enhanced through increased online communication and collaboration between students and staff. The guidelines also created a framework for conversation for enhancing e-learning. Because of the large number of student-centered guidelines, the focus quickly moved to practical ideas that stemmed from the guidelines, rather than the guidelines themselves.

26

Te Whare Wnanga o Awanuirangi

Contact: Mark Laws Project title: Ng Kaupapa Mori Arataki mo te eWnanga / Mori Guidelines for the Student Online Learning and Management System. Guidelines: The project used a variety of guidelines Te Whare Wnanga o Awanuirangi wants to control the development of culturally specific e-learning guidelines by allowing Mori staff to develop e-learning guidelines for Mori students. This development used the current elearning guidelines as a starting point. The main outcome of this project was to develop Mori pedagogies (such as integrating traditional customary practices like whakapapa and waiata into the eWnanga learning management system) for the online environment. This project benefits not only learners and staff, but also other practitioners who see the advantages of using this new set of Mori elearning guidelines.

Waiariki Institute of Technology Whare Takiura

Contacts: Jens Hansen, Mandy Williams Project title: The implementation of an e-learning guideline to ensure that effective design of e-learning is underpinned by research and best practice Guidelines: TD12 Is the design of learning informed by research on effective e-learning? This project was based on the development of processes that help Waiariki staff who are involved in the design, development, and maintenance of e-learning resources. E-learning processes were developed in consultation with staff and students. Staff and student impressions were included in a case study.

Wellington Institute of Technology (WelTec)

Project title: Integrating e-learning guidelines into the course redesign process Contact: Mary Weddell, Rachael Kelleher Guidelines: The project used a variety of guidelines WelTec used a wide range of e-learning guidelines to help with course design and redevelopment. They designed a method to review learning outcomes and to measure how appropriate e-learning was for each of the outcomes. WelTec created exemplars of best practice to support curriculum development in the areas of designing online discussions, facilitating online discussions, designing assessments, and designing multiple forms of engagement. The project showed that a combination of a clear overall process and expert support in how to complete each stage of this process is the most effective way to get results. Its expected that the support needed will lessen as academic managers and teaching teams develop their own curriculum design skills.

27

Appendix B: Survey results about the effect of the e-learning guideline projects

Table 1: Questions and results from end-of-workshop survey Responses as percentages Yes No N/A 100 0 0 90 5 5 74 68 68 63 63 58 42 10 16 9 21 16 21 21 16 16 32 16 21 21 37

Questions 1. Did this project help to ensure quality of e-learning? 2. Did your institution localise the elg for your own situation and priorities? 3. Did this project enhance community engagement by allowing sharing and collaboration? 4. Did the elg help you and your organisation discuss e-learning quality? 5. Did the elg impact on your course / department / organisation? 6. Did the elg help your organisation develop strategic elearning capability? 7. At the teaching level did the elg enhance student learning? 8. Did the elg help your organisation provide the quality environment students need to achieve good learning outcomes? 9. Did the elg support managers by providing an overarching framework that individual institutions can localise for their own situation and priorities.

The mini-projects each chose to work on some of the project objectives; it was not therefore expected that any one project would achieve all of the objectives. Overall, the results showed that the project met the aims and objectives in the original project plan.

28

Appendix C: Risks identified across the 21 mini-projects

Table 1: Categories of risks identified across the mini-projects Risk category Workload Design Timetabling Students access Communication and contribution Staff resignations Staffing contractors Technological Ethics None Financial issues Senior management support Redevelopment of other processes Number of projects that identify risk 31 Oct 2007 28 Feb 2008 12 6 9 6 7 7 6 11 5 4 4 3 3 3 2 2 2 1 1 3 0 1 0 1 0 1

29