Source: Sager, J. C. (1989).

Quality and standards – the evaluation of

translations. In C. Picker (Ed.), The translator’s handbook, (pp. 91-102).
London: Aslib, the association for information management.

Chapter 8 Quality and standards - the

evaluation of translations
JUAN C. SAGER

There are no absolute standards of translation quality but only more or less
appropriate translations for the purpose for which they are intended.

Preliminary observations
In order to discuss meaningfully the quality of a translator's work we have to
consider the process of translation as an industrial process, subject to
considerations of time and effort by which any work is measured. Similarly,
the result of this process, the translation itself, is a commercial product of
the information market, to which a certain price can be attached. It follows
that the price and hence the quality of translations should be commensurate
with the value attached to an original which is used for the same purpose
and with the importance attributed to it in the process of communication.
Translations are dependent texts in two senses: they are dependent upon
an original which has a comparable form and serves the same function; in
special cases the translation serves a different function possibly with a dif-
ferent form in which case we speak of translation as derived texts, e.g. in
the gist translation of a letter or the summary translation of a contract. They
are also dependent upon someone who commissions the translation either
for their own language. The initiators of the translation implicitly or explicitly
specify the function the translation is to serve within a communication
process. They determine the time available for the work of the translator,
and, through the price they are willing to pay, the type and quality of the
translation required. The initiators thus play a crucial role in establishing
valid criteria for the assessment of translations, their cost-effectiveness,
their appropriateness and their quality. effectiveness, their appropriateness
and their quality.
Different types of texts require different methods of translation and lead to
different end products. Most translations are modelled on, if not identical
with, text forms of the target language but there are also translations which
 do not match known forms of target language texts and which can therefore

THE TRANSLATOR'S HANDBOOK

While there are established text types which we can identify and whose general
characteristics can be described, there is not an ideal model of letter, scientific
report or instruction; each organisation develops its own variants according to the
function any such document has in the communication processes which represent or
accompany its activities.
Consequently there is no ideal type of translation for any of these forms. Instead,
any organisation which regularly requires translations decides the function of
translations in the overall system of communication and develops patterns for types
of translated texts. In reaching such decisions, organisations rely on the advice of
translators who then have the role of information advisers.
While translation is essentially concerned with the problems of interlingual transfer
of messages, the practice of translation requires modifications of texts which are
independent of the languages involved. It is therefore convenient to distinguish
between the obligatory modifications of linguistic transfer with concomitant
pragmatic changes conditioned by cultural differences and the deliberate
modifications which are necessitated by a change in function of the translated text.
This latter type of modification (selection, reduction, expansion of text) could
theoretically be performed prior or subsequent to translation, but in practice these
two processes are performed simultaneously. Conference translators spend the
greater part of their time on this dual process. Other examples of such modifications
are the translation of a text with a directive purpose into an information document
and the selective or summarising oral translation.
Any evaluation involves both comparison and measurement on a relative or
absolute scale. Translations can be assessed in terms of completeness or accuracy
but in practice such measurements are insufficient or even irrelevant (in the case of
derived translations). In many instances a translation has to be assessed by the
same criteria as an original, that is in terms of the adequacy of a text for its intended
purpose and the cost-effectiveness of the method of production. With the increasing
availability of various automated means of assistance and partial or total
substitution of the human translator there is a greater diversification of products of
translation and considerable variation in production costs which have to be
considered. As translating is a diversified activity and only one phase in a complex
communication process, full translations have to be compared with selective or
summary translations, human translation with machine translation, wherever such
alternatives may be appropriate. Factors such as the reader's time, the cost of that
time, the maintenance cost of machine aids, time delays,
typing costs and even abstracting or summarising cost and efficiency must
therefore be taken into account and measured.
The time and effort of translation varies with the nature of the original,
the purpose of the dependent or derived text to be created and the skill of the
translator. While in general certain types of text and their translation can be
said to be more difficult than others and hence require more time, there is
considerable variation in this assessment according to individual trans

QUALITY AND STANDARDS

reading comprehension and writing skills appropriate to their work. They are
thus more or less familiar with the structures and styles of texts according to
frequency of use. Translators are usually required to have a wider range of these
skills than other professionals and a greater facility in using them; they may
also need a wider scope of subject knowledge. In practice individuals develop
propensities towards particular text types which together with solid subject
 knowledge make certain translation tasks easier to perform than others,
regardless of the theoretically perceived difficulty.

Parameters of translation which affect evaluation

In order to determine the criteria which are of significance in the evaluation of
translations we have to describe the considerable number of variables which
affect the translation process.
The form and type of the original
The basis of the translation can be a written text or an oral instruction, e.g. to
write a letter containing a certain amount of information.
The original may be a full text, a draft for a translation which loses its validity
as a message after it has been written in the foreign language or a note
containing the content of a message required in the foreign language. The
original text may be a complete text or part of a text, an excerpt or a
title only.
The form and type of the translation
Independent of the original which provides the stimulus for the translation, the
end-product may be a complete text or a part, e.g. a book, an article, an
abstract, an excerpt or a title and it may be reduced or expanded in relation to
the original, e.g. the minutes of a meeting in one language written from notes,
the gist translation of a letter or a summary translation of an article.
Translations can therefore be full substitutes of an original with the same range
of functions and readership or derived texts which co-exist with the original
involving two processes, e.g. a translation and a reduction.
The status of the translation
The status of a translation is determined by its communicative function in
relation to the original. W e can distinguish three basic types of translations:
A. The translation is an independent document, i.e. a full substitute for
the monolingual reader.
B. The translation is an alternative to the original and co-exists with it,
e.g. a multilingual brochure.
Either type may serve the same communicative functions as the original or be
derived texts serving a different function.
C. The translation is a full equal with the original in all respects and may
therefore serve as a basis for other translation. This type is frequent in
international organisations and some legal documents but is not as

THE TRANSLATOR'S HANDBOOK

out being exhaustive, gives a number of uses which may affect the nature of translation:
- scanning and discard
- reading for information
- detailed information and storage for future reference - draft for other texts
- publication - for prestige
- for public record - legal validity
In each case the readership may vary from one or a few to a very large number.

The textual factor

As originals are characterised by subject matter, general or specialised mode of expression and intention (directive,
informative, persuasive, evaluative, social, etc. ) which are reflected in appropriate text types and conventions of
presentation, translations must either conform to equivalent text types and conventions (type A and C), deliberately
choose others (type B), or indeed evolve their own types on the basis of user requirements (generally applied to type
B, but also conceivable for type C, or with even greater force to
machine translation output).
 Research factor
The amount and type of research required for a translation plays an important part in the quantitative aspects of
evaluation, but as this varies considerably with the individual's knowledge and the degree of specialisation permitted
by the job, it is practically impossible to quantify.

Revision factor
The amount and cost of revision can significantly affect the cost and quality of translation. In this context revision is
not understood as the checking for accuracy and completeness, but rather for consistency of style. The latter refers
to both the internal consistency of a long document translated by several individuals and the harmonisation, with a
house style, of an entire
translated text:

Summary
We can represent some of the factors which contribute to the diversification of translation in a table as below, from
which we exclude literary translation since it is subject different criteria and must therefore be evaluated

Criteria for evaluation

Evaluation of translation can be carried out with many different objectives and accordingly different criteria and
factors acquire varying importance. It is convenient to distinguish between macro-evaluation which aims at
assessing the value of the product and micro-evaluation aimed at improving the product. Our main concern is
macro-evaluation.

Objectives of evaluation
Evaluation can be carried out as partial assessment of translators; it can be concerned with assessing the faithfulness
of the translation with regard to content and intention - which is what most people understand when they speak of
quality. Alternatively or simultaneously evaluation can be aimed at assessing the cost of a translation in comparison
with other translations produced by the same or different means, e.g. human versus machine translation. Finally and
perhaps most importantly, translation can be assessed in terms of the appropriateness for its intended purpose,
which assumes that there are various alternatives possible for achieving the desired communicative effect.
This global approach includes such criteria as intelligibility, which must be measured against the intelligibility of
 the original, time of translation and time of reading by the end user.
Marginal criteria which come into play - especially with the alternative of machine translation in mind - are
disponibility of translation (i.e. the speed with which a translation can be completed and the availability of
translators for less common languages and highly specialised subjects), the requirements for single or multiple
translation and even the possibility of -storing translation versus repeating a translation when the need arises,

THE TRANSLATOR'S HANDBOOK

Preparation of translation
Translators spend a considerable part of their time preparing for the task of writing or dictating a
translation. This involves interpreting the specification and formulating strategies for implementing the
modifications required in each case. When routines have been developed this preparation time is
substantially reduced.
The type and degree of deliberate modification is decided by an analysis of the text in relation to its new
readership. It involves an analysis of the presuppositions on which the transfer of knowledge is based, the
assumptions made about the correct interpretation of the intention of the message as well as the
expectation the reader has about the message. The following table shows the scope of the analysis
required.

Situational factors can only be taken into account when the translation is
virtually concurrent with the transfer of the message; otherwise a new situation
is created by the intervention of the translation process. (Incidentally, this table
also shows the limitations of machine translation, which can only
act on the analysis of linguistic signals.)
Deliberate modification can be negligible if there is a large measure of
coincidence between writer intention and reader expectation. The British
Highway Code translated for French tourists would be such a case. A greater
degree of modification may be appropriate for the translation of a Trade Union
Rule Book as it is unlikely to be used for the same purpose, but only as a
comparison of procedures.
Presuppositions about differences in the knowledge between writer and
reader are only rarely taken into account.

The role of the initiator

The type of translation produced depends on the role of the initiator who can
 either be no act for. the writer of the original or the reader of the trans

QUALITY AND STANDARDS

translation is usually of type A, as in the case of a publisher who commissions

the translation of a book in order to open up another language market, or a
manufacturer who seeks a new export market. In both cases the translation has
to be evaluated in terms of the writer's intention and the extent to which it
achieves the desired effect, e.g. selling a product. The reader is then not
interested in the original, except when he or she finds fault with the new text.
If the initiator is the reader or his or her representative the type of translation
is usually of type B. The reader knows that there is an original which he or she
wants to read but is unable to do without a translation, and will specify what is
sought from the translation regardless of the writer's intention. The quality of
the translation is then judged on the extent to which the reader expectation is
satisfied without distorting the content of the original. When reader expectation
differs from writer intention the translator produces a derived text.
Translations of type C are neutral with respect to the initiator as translation
occurs simultaneously with writing, as for instance, in the work of the
jurist/linguist panels of the Council of Ministers of the EC. In production terms
these two roles can be interpreted as incoming (reader determined) and outgoing
(writer determined) texts. Standards applied to each type are then questions of
cost effectiveness and speed of information.

The awareness factor

Translations are often judged by the extent to which the reader is aware of
dealing with a translation. Such an awareness may be caused by linguistic
interference, which is generally deprecated (on the level of pragmatic or
cultural transfer and adjustment a translation may, however, deliberately retain
characteristics of the original for stylistic reasons, e.g. appeal to certain prestige
values). In reader initiated/oriented translation the knowledge of dealing with a
translation is always present and it is then a question of deliberate decisions to
what extent a translation should obliterate its origins. If the writer is aware of
translation, he or she can offer great assistance to the translator to produce a
translation of type A. The original may then be a draft to be replaced by the
translation. If readers are aware of the translation they can instruct the translator
as to their expectations, they can adjust their reading habits, decide whether to
have a translation revised, or choose raw or post-edited output from a machine
translation system. Reader awareness can lead to type B translation producing
full dependent, alternative, partial or selective translations.

The user factor

The nature of a translation can also be influenced by the use that is to be made
of it. While this is true of all texts, a translation can take into account
THE TRANSLATORS HANDBOOK w
QUALITY AND STANDARDS

Modes of evaluation
Evaluation has two aspects which usually complement each other in varying proportion according to the type of translation
involved.

(i) A new text can be compared to the original, via a metalinguistic representation, relatively to other translations, as in
examinations or absolutely to an ideal type for the particular type. In each case there are three categories to be compared as
in the diagram.

ideal _

translation

metaliguistic

I original } ---> representation ----> translation I¡

other _

translations

(ii) A translation can be described, analysed and evaluated as if it were the result of a writing or re-writing process, the
translator performing the role of a specialist writer. The greater the distance between contents form and intention between
the original and the translation (draft, advertising copy, gist, directives for information) the more important is the second
aspect. These two aspects also play an important role in the methods of evaluation. As translation is a mediation process
evaluation should involve the end user and possibly the initiator, if a different person. The end user can, however, assess
only under the second aspect as he or she would not require the translation if they were able to read the original. The first
aspect requires assessors who have an equal command of the languages involved, which in practice means other
translators.

Dimensions of evaluation
Evaluation can be qualitative or quantitative. Purely qualitative assessment ignores time, cost and other factors relevant to
industrial processes and products. It can only be applied relatively to the original or absolutely to the end product. It has its
value both for examinations of translators and for literary translation but in practice it is usually combined with aspects of
a quantitative assessment. The latter take into account the significance of quality in relation to the importance attributed to
the translation by the initiator in the wider industrial, commercial, political or social context of which it is a part. Quality is
then subsumed under the heading of appropriateness which consists of accuracy, intelligibility, functional adequacy and
formal presentation.

A second major aspect in quantitative assessment is composed of economic factors: time spent on translation both as a
production cost factor and as the time lag between initiation and delivery, ancillary costs of clerical and information
services, reading time by end user, etc.

Quantitative evaluation also has to consider alternatives to human translation, e.g. the partial or total use of machines
or even, where appropriate, an alternative linguistic regime; this could involve, e.g. adopting a single working language in
a multilingual organisation, introducing a controlled language or languages for certain text types and purposes, or even
reducing the overall volume of translation if the outlay does not appear to justify the benefit.

Methods of evaluation
A distinction must be made between the evaluation --------------------------------------------
- of a translator's work - of a single translation - of a translation
service - of a translation system

We exclude from consideration the evaluation of student exercises as belonging to the area of training which is marginal to
this handbook and concentrate on the evaluation of single translations.
 Evaluation of a translator's work
Unlike any other profession it is customary to select translators by competitive examination. This is particularly common
among large employers of translators and felt to be necessary because there is no agreed standard of training, no body
which controls admission to the profession and not even a protection for the title of translator. _
Employers must therefore feel that one or two test pieces produced in a two or three hour examination provide a more
reliable indication of the qualities they seek in a translator than previous examination results, references or work carried
out under unconstrained conditions. The ability to produce a certain amount of work in a given period of time is therefore
an important criterion for selection and confirms the importance of economic considerations.
There is considerable variation in the criteria used by translators' organisations for admission to membership. Some
admit only by examination which may contain translation itself, general and special subject knowledge and even
knowledge of the spoken language; some admit members on the basis of appropriate degree qualifications, professional
experience or submitted work; others require a combination of these skills and achievements. In all these cases we are
dealing with peer assessment and this is appropriate since the selection is made for membership of a professional group
whatever the collective quality of this group may be.

Evaluation of a single translation

This kind of assessment consists of examining a translation both as an independent piece of writing and a linguistic and
pragmatic transfer. Which of these two dimensions has greater importance is decided by the type of translation, the degree
of deliberate modification and derivation specified and the declared purpose of the new text.

A,

THE TRANSLATOR'S HANDBOOK

QUALITY AND STANDARDS

In order to be objective the evaluation procedure must be repeatable; a set of clearly

defined parameters is necessary and a scale of values with appropriate gradations. Since
every text is different and any one text may be translated to different specifications, the
parameters must be adjustable and susceptible to weighting so that an appropriate
assessment scheme can be designed for any one task based on the original and its full
specification.
The evaluation procedure must be economically feasible. A simple characterisation of
gravity of mistakes on a scale, e.g. distortion of meaning, omission and grammatical
incorrectness and minor stylistic infelicities and orthographic errors each on a point scale is
clearly insufficient for the evaluation of an industrial product; it may be, however, adequate
or indeed justified in large-scale examinations.
A basic distinction must be made between errors caused by inadequate knowledge of the
vocabulary, orthography, morphology or syntax of the target language and those resulting
from misinterpretation of the text or inadequate expression. Since the former are relatively
rare - the target language of a translation is generally the language of habitual use of the
translator - the latter are the main concern of evaluation.
Any detailed evaluation must assume the translation process to consist of selection
procedures with many predetermined and some free choices; many of these are mutually
exclusive.
The following elements must be combined in a grid for assessment and interdependences
must be accounted for both at the lexical and the syntagmatic level.
Types of error:
- inversion of meaning
- omission
- addition unless justified by the specification. - deviation
- modification
Effect of error on the whole text; e.g. a typographic error can change the meaning of a word
and thereby distort the whole text:
- linguistic effect: does the error affect a main or secondary part of
the sentence, e.g. the subject or a modifier?
- semantic effect: does the error affect a major or minor element, e.g.
the main argument, or an example?
- pragmatic effect: does the error affect the intention in a significant
or a negligible way, e.g. the general persuasive purpose or the tone
of voice?
It is obvious that with such a diverse matrix of criteria there will be a considerable variation
 in the results of assessment, which can only be balanced by averaging scores obtained by
different assessors.
A different method involves the end user as principal assessor after the accuracy of the
translation has been established (after a simple check for

intelligible (this may, of course, reflect on the original) and whether it is acceptable in terms of
comparable texts in their own language given a certain cost for translation. Both these rather
subjective assessments can be objectified to some extent by comparing the time required for
the reading of a translation and an original.
Other factors, such as presentation, layout, cost of production are important but are
essentially the same for the production of any text, whether translated or not. "'

Evaluation of a translation service

The quality of translation is only a part of the complex set of criteria that enter into the
assessment of a translation service which requires operation and management methods.
Factors which are important from a translator's
point of view are
- contact with and feedback from initiators and end users - distribution of work
among a team - support by and cost of revision - reference tools and library
assistance - filing and access to previous translation - secretarial assistance

Evaluation of a translation system

As machine translation is still in a stage of rapid development there are no established
methods of assessment, though considerable experimentation has taken place regarding the
evaluation of quality of output and the comparative cost of human versus machine translation.
Cost-effectiveness of machine translation cannot sensibly be calculated partly because there
is no clear philosophy how development costs should be reflected, partly because there is
insufficient data on maintenance including continuous dictionary-making costs and partly
because it is as yet impossible to calculate the cost of translator involvement in pre- or post-
editing.
The development of criteria for evaluation of machine translated texts is important not only
for its own sake; it is also an integral part of the feedback necessary for the improvement of
the machine translation system.
Only in the very few instances where a machine translation system is fully operational and
regularly processes large quantities of texts is a genuine comparison with human alternatives
possible.

Further Reading
GAMBIER, Y. (ed.) Trans, University of Turku School of Translation Studies, Turku, 1986.
GOUADEC, D. Paramètres de l'évaluation des traductions, in: META 2$, 2, 1981, 99-117.
HOUSE, J. A model of translation quality assessment, Tübingen: Narr,

THE TRANSLATOR'S HANDBOOK

NEUBERT, A. Text and Translation, VEB Verlag Enzyklopaedie, Leipzig, 1985.

REISS, K. Möglichkeiten und Grenzen der Übersetzungskritik, München: Huber, 1982 (4th ed.).
van SLYPE, G. Conception d'une méthodologie générale d'évaluation de la traduction automatique, in: Multilingua 1,
4, 1982, 221-223. SNELL-HORNBY, M. (ed. ) Übersetzungswissenschaft - EineNeuorientierung, Zur Integrierung von
Theorie und Praxis, Francke Verlag, Tübingen, 1987.