Guidelines for the evaluation of machine translation
This web page shows the guidelines that we gave to the translators who evaluated the machine translations of 'Clear English gives good machine translation'.
First, the translators evaluated fluency. Then, the translators evaluated accuracy.
A note about technical terms
Sometimes, a translation contains an English technical term, such as 'machine translation'. For the Norwegian evaluation and for the Spanish evaluation, we did not tell translators how to evaluate text that was not translated. Therefore, some evaluations were very different.
For the Welsh evaluation and for the Russian evaluation, the translators were told to assume that a reader knows the technical term. Similarly, a fluent English sentence can contain a French term. However, the assumption is that the reader knows the French term.
Fluency
For this part of the evaluation, only fluency is important. Possibly, translated text is not accurate when compared with the source text.
Each translator evaluated each translated sentence for fluency. The translators did not look at the source text. The translators chose the fluency from four options. We gave the translators the following guidelines:
- Excellent fluency. Note. Possibly, text is fluent but has no real-world meaning. For example, in poetry, possible text is "sleeping green clouds dream furiously."
- Satisfactory fluency. Note. 'Satisfactory' does not mean 'good'. Examples:
- There are small mistakes in grammar and spelling, but the meaning is clear.
- Collocations are unusual, but understandable.
- The translation is not correct for your locale, but the meaning is clear.
- Bad fluency.
- Incomprehensible.
Accuracy of meaning
Each translator compared each translation with the English source text, and evaluated the translation for accuracy of meaning compared with the source text. The translators chose the accuracy from four options. We gave the translators the following guidelines:
For this part of the evaluation, the fluency is not important. For example, if the translation contains incorrect spelling or if a subject and a verb do not agree, ignore the mistakes if the meaning is clear.
For this part of the evaluation, the exact words and their order are not important. For practical purposes, all of the following sentences have the same meaning:
- Tomorrow, begin the project.
- Start the project tomorrow.
- Commence the project tomorrow.
Compare the translation with the source text, and evaluate the translation for accuracy of meaning compared with the source text. Choose one of the following quality levels:
- Correct meaning.
- Partly correct meaning. Examples:
- A sentence has two clauses. One clause is correct, and one clause is not correct.
- Ambiguous meaning. The translation has more than one meaning. One meaning is correct. Other meanings are not correct.
- Incorrect meaning. The translation has one meaning, and the meaning is not correct.
- Nonsense.
RSS feed