Informatik

DIUS co-ordination meeting July 5, 2011: Quality assessment at the University of Koblenz-Landau
Klaus G. Troitzsch

Informatik

Evaluating universities
Education is perhaps the main field in which all kinds of evaluations are regularly applied. Example: Evaluating higher education institutions and their study programmes

14/04/2012

http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx

2/102

Informatik

Identify evaluation objects

The object of evaluation (and quality assurance) is the core product of a university:
the responsible
refers to responsibility towards students / disciplines / societies

warranty
content-related and organisational securing of processes, less than guarantee, more than just provision

on a functional study process

not necessarily successful, as success can only be judged ex post

for all enrolled students

14/04/2012 http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx 3/102

Informatik

Identify evaluation goals

quality aspects have to be defined this is the task of the departments / faculties of a university who define the desired contents of a study programme departments / faculties include both professors/lecturers and students

14/04/2012

http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx

4/102

Informatik

Guidelines for data collection

quality assurance is an empirical process empirical data about the courses offered and the progress of individual students have to be collected and analysed keep track of all offered courses with number of students participating, marks received by students and by teachers keep track of the progress of individual students

14/04/2012

http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx

5/102

Informatik

Indicators and instruments

warranty: workload questionnaire (standardised) responsiveness:
achievement questionnaire (standardised) narrative interviews and focus groups (nonstandardised) discussion among the faculty members of a study programme

study process: statistical data of the examination office students: entrance questionnaire
14/04/2012 http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx 6/102

Informatik

Questionnaires used
The questionnaire on workload and feasibility of studies (WLS;
is a three-part questionnaire that is completed at different times during a running semester. This is done online. The students are asked by email to participate .

The Trierer Inventory on lecture evaluation (TRIL)

is available in three versions (s. Appendix B): for evaluation of input events (usually lectures: TRIL2BM1), discussion events (usually seminars: TRIL2BM2) and exercises (TRIL2BM3). The questionnaire is completed on paper in the lesson, ideally approx. 4-6 weeks before the end of the lecture period. Thus, the lecturers receive feedback in time to talk about it with their students.

The questionnaire on achieving the learning objective

is the result of a multi-stage procedure. In a first step, the lecturers indicate the qualification objectives from the module description that are relevant for their lectures. In teaching-related courses of study, this also includes whether or not the curricular standards of the state of Rhineland-Palatinate are implemented. In a second step, the lecturers phrase teaching objectives assigned to the qualification objectives. Then these learning objectives are rendered as a questionnaire by the method centre. Finally, the questionnaire is completed by the students around the end of the semester. The appendix includes an example for a lecturer questionnaire (Appendix C) and the resulting student questionnaire including the teaching objectives phrased by the lecturer (Appendix D).

The initial studies survey (StEiB)

is used to ask freshmen on the transfer from school to university (selection of subject and location of studies, taking up studies and usefulness of information given by university facilities), the study requirements (financing, dealing with typical initial problems; previous performance, acquisition of key competences in school and their relevance for studies) and objectives in studies. This questionnaire is to be distributed in large freshmen lessons and either completed on site (with the consent of the lecturer) or at home and handed in at a central office. Time for completion is approx. 30 minutes. The departments are able to add subject-specific questions.

An alumni survey
is already regularly performed with all graduates of the state of Rhineland Palatinate by the Zentrum fr Qualittssicherung of the Mainz university (ZQ, also see Hochschulevaluierungsverbund Sdwest e.V.). Graduates of the university of Koblenz-Landau who completed university in 2005 were the first to take part in the survey. The survey includes not only information on subject, type and location of graduation, but also questions on a general final assessment of the studies (satisfaction with and success of studies, realisation of original objectives) and on the transfer from studies to working life, acquisition of key competences by the studies and their relevance in working life. Further graduates are to be asked in future. If applicable, surveys are to be performed by the universities directly in future. The departments are able to add subject-specific questions.

7/102

Informatik

Collect data
subjective evaluation of courses by students data about the progress of individual students distribution of
students per study programme and semester, marks given in each module, general marks at the final examination, numbers of students in each course, drop-out rates

14/04/2012

http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx

8/102

Informatik

Explore and analyse data

explore: simple statistics, mainly univariate analyse: find out dependencies between variables data mining: use different statistical methods for classifying students, teachers, courses

14/04/2012

http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx

9/102

Informatik

Feedback results to the stakeholders

discuss the student evaluation results among faculty members offer help to faculty members with negative best evaluated teacher .. worst evaluated teacher evaluation results find out which students need mentoring find out which modules are attractive / difficult / boring and which are not
Boxplots show the distribution of general marks given by students to teachers. The median of the best evaluated teacher is one standard deviation above the mean, the evaluation of the worst evaluated teacher is 2.5 standard deviations below the mean (-2 is good, +4 is bad).
14/04/2012 http://www.uni-koblenz.de/~kgt/lehre/ResMeth/ResearchMethods2011.ppsx 10/102