Guidelines for the evaluation of machine translation

This web page shows the guidelines that we gave to the translators who evaluated the machine translations of 'Clear English gives good machine translation'.

First, the translators evaluated fluency. Then, the translators evaluated accuracy.

A note about technical terms

Sometimes, a translation contains an English technical term, such as 'machine translation'. For the Norwegian evaluation and for the Spanish evaluation, we did not tell translators how to evaluate text that was not translated. Therefore, some evaluations were very different.

For the Welsh evaluation and for the Russian evaluation, the translators were told to assume that a reader knows the technical term. Similarly, a fluent English sentence can contain a French term. However, the assumption is that the reader knows the French term.

Fluency

For this part of the evaluation, only fluency is important. Possibly, translated text is not accurate when compared with the source text.

Each translator evaluated each translated sentence for fluency. The translators did not look at the source text. The translators chose the fluency from four options. We gave the translators the following guidelines:

Accuracy of meaning

Each translator compared each translation with the English source text, and evaluated the translation for accuracy of meaning compared with the source text. The translators chose the accuracy from four options. We gave the translators the following guidelines:

For this part of the evaluation, the fluency is not important. For example, if the translation contains incorrect spelling or if a subject and a verb do not agree, ignore the mistakes if the meaning is clear.

For this part of the evaluation, the exact words and their order are not important. For practical purposes, all of the following sentences have the same meaning:

Compare the translation with the source text, and evaluate the translation for accuracy of meaning compared with the source text. Choose one of the following quality levels: