PROYECTOS EUROPEOS

 

MASSIVE: A EUROPEAN PROJECT TO ENHANCE ELEARNING IMPLEMENTATION IN HIGHER EDUCATION

 

ISABEL PÉREZ

BEGOÑA ARENAS

UNIVERSITY OF GRANADA

SCIENTER ESPAÑA

Abstract

This paper presents the development and results of the European eLearning project MASSIVE coordinated by the University of Granada . The main output has been a peer review model to enhance eLearning in traditional universities. This model has a holistic and integrative approach by taking into consideration the following six areas: university strategies, university libraries, management of IPR, support to teachers, support to students and design of online courses. In this context, the emphasis of MASSIVE has been to design a peer review methodology in order to develop a flexible and functional methodological handbook that could help with the systematisation of the peer review process. Finally, as a result of the project, there has been a proposal of possibilities to transform this model into a practical service.

 

1. Introduction

The MASSIVE project was part funded by the EC within the eLearning Programme of the Directorate-General for Education and Culture. It was coordinated by the University of Granada, the rest of the consortium included ten other universities and research centres; all of them had previous experience in the areas of interest of MASSIVE and perceived a strong need to improve quality and mutual support in the integration of ICT into teaching and learning (FIM-new learning, Scienter, University of Barcelona, Budapesti Mûszaki és Gazdaságtudományi Egyetem, School of Education - University of Edinburgh, University of Bergen, Spanish Digital Society of Authors and Publishers and Scienter España).

The project inspiration was the present impact of ICT in Higher Education and the necessity of research on how to implement these ICT components with a high standard of quality. This led the consortium to design a non commercial model of mutual support services for European traditional universities to successfully implement the virtual component of teaching, either from an initial starting point or to improve what they are doing already, but without re-inventing the wheel once again. To develop this model MASSIVE identified the following specific objectives:

• Defining a conceptual model for the integration of ICT into teaching and learning practice;
• Identifying and classifying good practices in the organisation of support services to the University community regarding University virtual components;
• Exploring and comparing the elements for transferability according to a mutual support non-commercial model;
• Validating the approaches to develop the support services;
• Ensuring the wide dissemination of the practices and the use of the model.

The conceptual model for integration of ICT was based on a holistic and integrative approach, as the project considers that setting up the eLearning offer in a traditional university concerns more than one sector or department in the university. The project identified six areas of interest which are inter-related; all of them critical for the implementation of quality eLearning education:

• University Strategies towards the integration of ICT in the Teaching/learning practice
• Evolution of University Libraries
• Management of IPR issues
• Support to Teaching staff
• Support to Students
• Design of online contents

Obviously, the strategic area has a major influence in all the others and there are also especial links between some of the areas: university libraries, IPR issues and design of online courses or support to teachers and to students.

As to the model of support services itself, for several reasons peer review seemed to be the appropriate methodology to use: it is widely utilised to introduce new techniques and to improve the educational offer by integrating correctives among colleagues; it allows the involved institutions to take part in a dialogue between both sides as colleagues do; and the institution goes through a process of reflection upon their weaknesses and strengths .

 

2. MASSIVE Project Development

To start with, there were some questions to be answered:

• What does quality mean in this field? And in relation to each of the six areas?
• How can higher education institutions ensure they are not re-inventing the wheel?
• How can they profit from the experiences of early pioneers in the field?

Part of these had been answered in previous projects, and so we first analysed their results and carried out twelve case studies, as examples of good practices of eLearning in traditional universities; this analysis gave us a framework to establish criteria and tools to recognise what quality means in each of the areas and to infer which strategies, services and other elements should be transferable to the peer review model of support we were going to define. The outcome was a desk research and a comparative report that served as the basis for preparing the peer review methodology.

 

3. MASSIVE Peer Review Methodology

The next step in the project was to define the peer review methodology that MASSIVE wanted to put it into practice in six universities, five of them partners in the project. The methodology was set out in terms of the following aspects:

 

3.1 Purposes

Summarising, apart from testing the model, the main purposes of our peer review model were to provide formative feedback on the current ‘state of the art' in the participating universities by identifying ‘gaps' and suggesting ways of addressing them, enabling the participating Universities to improve and further develop their ‘virtual capacity' and to improve their practices.

 

3.2 Participants

Our model of peer review involved all actors related to eLearning in the six areas we defined, that is: teaching staff involved in the delivery of courses; ‘specialist' staff, particularly those responsible for course design; platform design and maintenance; course administration and evaluation; other stakeholders (e.g. managers and administrators) and students. Apart from that, the peer review team proposed included 3 visiting experts and one expert from the reviewed university.

 

3.3 Approach and Procedures

The MASSIVE peer review approach aimed mainly to offer opportunities to learn in a collaborative environment between pairs, so that the methodology is more focused on developing educational opportunities than on evaluating or judging the procedures and actions. Besides, in the context of MASSIVE, we need to link the peer review practice to the particular features of ‘virtual' learning settings. So, taking into account the results of the state of the art review (especially those from the ‘BENVIC' project¹) we suggested that the key dimensions the peer review needs to focus on include: learner services, learning delivery, learning development, teaching capability, evaluation and assessment capacity, accessibility, technical capability and institutional capability.

As to the procedure we proposed it has the following characteristics:

• Collegial, that is, the process will begin with a ‘stakeholder analysis' intended to reflect the needs, positions and objectives of all actors involved.
• Open, assuming that peer review outputs will not be confidential documents but shared between reviewers and participating Universities.
• Multi-dimensionality and triangulation, so that the review process will incorporate data collection using a range of sources.
• Ethical focus, taking care that the review process will not compromise the integrity of the University provision or relationship with their students.
• Formative, so that peer evaluation will be a component of a continuous improvement process.

 

3.4 Data collection methods

To collect data we suggested a combination of methods including the followings: documents and archives; structured/semi structured interviews (‘face to face', by e-mail or by telephone) and focus groups; ‘direct' and ‘indirect' observation (e.g. viewing online courses).

 

3.5 Data collection Tools

A peer review tools kit was also developed in order to guide and structure the peer review experience; it was formed by:

• Overall introduction. Introduction to 6 issues/themes.
• General briefing paper for peer reviewers providing concrete and operational guidelines.
• General briefing paper for receiving/hosting Universities. Particular attention is paid in illustrating to universities the approach adopted in the review process (bench-learning).
• Interview grid for management team (targeted to the management staff).
• Interview grid for stakeholders groups (targeted to students, technical staff, librarians, teachers, etc.).
• Thematic interview grids (This tool is focused on supporting an in-depth analysis on each specific themes. The hosting University will choose on which of 6 themes they would like to activate the peer reviewing process).
• Guidelines for focus groups.
• Format for reporting the results of the peer review process.

4. Peer review in practice

The methodology handbook summarised above is a reference tool that helps the teams organise the process. However, in practice it has to be adapted to each case, according to the specific characteristics of the hosting university and to the number of areas researched, in this sense flexibility was one of the lesson we learnt through the application process.

Normally the Peer Review has a predominantly ‘analytical' purpose: to critically review the service models and approaches in participating universities. The process will take place in three phases:

A preparatory stage (countdown: one month before the visit) – this entails gathering background information about an institution, and its current and future planned use of eLearning.

A site visit stage (2 working days) – which involves a collaborative dialogue between the peer reviewers and a range of representatives of the University.

An analysis and reporting stage (one month after the visit)– on the basis of the data gathered from the preceding stages, this final part of the review process focuses on the production of recommendations through collaborative reflection between the peer review team and the hosting institution.

Data collection: in practice this involves a combination of a range of methods and instruments including:

• Documents and Archives – these encompass a range of ‘secondary' data, including: documents and reports (brochures; prospectus and course outlines; ‘mission' statements; records of committee meetings; evaluation reports). Another source of archival data will be log files (for example recording utilization rates and patterns of use of the technology platforms and services).
• Interviews – these include structured/semi structured interviews (administered ‘face to face', by e-mail or by telephone) and focus groups.
• Observation – this includes ‘direct' observation (which implies some form of engagement with participants, for example getting learners to ‘think aloud' when using instructional services) and ‘indirect' observation (for example participating in a classroom setting without directly engaging with learners).

The types of questions ask during the peer review are:

1. Profiling questions: to determine how the university is situated within the eLearning spectrum.
2. Diagnostic questions: which relate to the university's mission to conduct or use eLearning.
3. Strategic questions: to determine the strategies and resources available to the institution to enable it to carry out its stated missions with respect to eLearning.
4. Operational questions: to determine how eLearning strategies are put into practice.
5. Assessment questions: to benchmark the performance indicators related to the use of eLearning within the institution.

The actual implementation of the peer review model in MASSIVE revealed some of the difficulties we may encounter in future actions, for example difficulties in:

• deciding what approach to choose, descriptive, analytical or prescriptive;
• finding relevant documents;
• getting the right people for the interviews;
• following the processes as they are conceived in the Handbook, etc;
• language use; and in
• time management, one of the main problematic issues in the whole process, as it turned out that, in general, every step required more time than planned.

 

5. From the MASSIVE model to the service

 

Based on the methodology and practice described above, eventually the MASSIVE consortium developed a service that is now available for all the universities that might wish to use it. The MASSIVE service portfolio has been described in terms of 4 possible options included in the following diagram:

 

 

These 4 possibilities or support services form a complete matrix to supply the segmented European needs of the traditional universities.

The main idea is to provide a continuous advice service (provided by the University of Granada and Scienter España) to help university users define the appropriate segment and level of the service and support they may require.

The 4 options are built on each-other, so the MASSIVE set of services are based on communication with the users in order to upgrade the initial need and to go through the different steps of service levels or track the changes of needs of the user and support them dynamically.

Part of the service includes a Peer review tool kit that is available for free on the web site of MASSIVE and it contains:

• Overall introduction to resent the overall scheme/framework in terms of aims, expected results, services provided processes, requirements and tools.
• Introduction to 6 issues/themes (which aims at presenting the current state of the art on the key themes, the main problems and difficulties, set of criteria and best practice)
• General briefing paper for peer reviewers.
• General briefing paper for receiving/hosting universities.
• Interview grid for management team.
• Interview grid for stakeholders groups.
• Thematic interview grids.
• Guidelines for focus groups.
• Format for reporting.
• Promotional materials to attract the interest of university staff and to clarify the aim and dynamics of the peer-reviewing exercise.

In the following sections we explain the 4 possibilities of services identified by the partnership:

 

5.1 Online access to handbook and tools

This is the simplest service offered and consists of free downloading of the methodology handbook and tools from the MASSIVE web site ( http://www.massive-project.org/ ). To allow this we have set up a module using the platform ILIAS, University of Granada's LMS, to disseminate and valorise the results of the project, in particular the handbook will be offered for free to the Universities that aim to develop a peer review service themselves (after a simple registration).

The platform will allow for downloading documents and registration so that a database of experts and stakeholders involved in quality and ICT in HE will be created. All identified users will be posted upon changes of the documentation. Valorisation will be done by collecting comments and case studies of the use of the downloaded documentation on the know-how.

The Massive team will regularly review and update the know-how in the light of the real use of the technology. The user feedback will be an important element to close the feedback-loop and key to the continuous development of the platform.

This form of service will be delivered for free after the project eligibility period (2005-2006) by the University of Granada and Scienter España. Also specific advice and support will be provided on demand for free.

 

5.2 National and Institutional Seminars

This service consists of organising national seminars on the topic and also institutional seminars on demand by a specific university that may want to analyse the issue of ICT implementation in a specific context. The main objectives of this service are:

• to disseminate the peer review handbook and tools;
• to complement the peer-reviewing service with the availability of a vital learning community (the European Virtual Campus Academy – EVCA);
• to establish a diagnostic capacity that will allow the peer review activities and the related collaborative learning activities to integrate the emerging critical themes.
• to provide with expert advice on demand on the themes selected by the universities.

 

5.3 Peer review without accreditation

This service is similar to the experience put into practice in MASSIVE where a university asks for a peer review in the field of ICT. In this case there will be a contract between the university and the group of experts to carry out the peer review process. It could be a whole process of all the areas identified by Massive or a process focused on one or two areas chosen by the institution.

The service offers planning, guiding and evaluating the process. There is a need of close cooperation with the University so as to meet their specific needs. The service will be a non-profit one, but will cover the necessary costs for it to be delivered.

 

5.4 Institutional Accreditation through the UNIQUE project

Finally, the last service in this portfolio consists of a peer review process adapted and enriched by an accreditation system designed in the framework of the UNIQUe project (2007-2008). This project proposed by 4 European networks (MENON, EuroPACE, EFQUEL and EFMD) aims at creating, testing and launching an eLearning quality label for ICT use in higher education. The process of accreditation will be based on the peer review model designed in MASSIVE. Specifications of the UNIQUe quality label will be prepared and tested with a network of 12 universities what will help in the dissemination and sustainability of the accreditation service.

 

6. Conclusions

To summarise what has been exposed above, MASSIVE aimed to contribute to quality improvement in traditional higher education institutions by considering peers' assessment a rich and fruitful approach and designing a model of support based on that methodology.

To do that, apart from doing an extensive desk research in the selected areas and define quality criteria and tools, most of the emphasis of the project was placed on developing a peer review methodology and on reflecting on the implementation of the peer review process, being the goal to define a flexible model of support.

Finally, this peer review model will be sustainable in a future set of services that may vary according to the interests of the applicant institution.

 

¹http://www.benvic.odl.org/