Evaluation of English-Welsh machine translation

An article about machine translation was translated into Welsh by Google Translate (http://translate.google.com). In January 2010, professional translators evaluated the translation for fluency and for accuracy.

The results for fluency are in Figure 1. The results for accuracy are in Figure 2. Full details of the evaluation are in mt-evaluation-en-cy.xls.

The fluency of machine translation from international English to Welsh is not satisfactory

Figure 1. The fluency of Welsh machine translation of international English

The accuracy of machine translation from international English to Welsh is not satisfactory

Figure 2. The accuracy of Welsh machine translation of international English

The following translators evaluated the machine translations:

The project management was by Cymen (www.cymen.co.uk).

Discussion of the results

Comments from the translators show that the primary problems with the machine translation are as follows:

Sentence 38 is an example of what not to write. The translation is low quality.

Some of the English text contains terms and examples that we do not want to translate. For example, sentence 41 is, "For example, instead of 'carry out', use 'complete', 'do', or 'perform'." Because of the method that we used, we could not prevent Google Translate from translating 'carry out', 'complete', 'do', or 'perform'.

Sometimes, translators gave different evaluations. For example, for accuracy, sentence 23 has the following evaluations:

In previous evaluations, we had a similar problem. To prevent this problem, we gave additional guidelines for the Welsh evaluation.

The additional guidelines were not sufficient to prevent large differences for some evaluations. Unfortunately, the project did not include a review stage, in which translators can discuss their different evaluations. In future projects, we will include a review stage.

For translations of English to Spanish and English to Norwegian, Google Translate gives satisfactory translations. Therefore, the low quality of the Welsh machine translation is for one or both of the following reasons:

An expert at the Language Technologies Unit (www.bangor.ac.uk/canolfanbedwyr/index.php.en), Bangor University, says that Google Translate gives better translations from Welsh to English than it does from English to Welsh. He thinks that some of the reasons for the difference are as follows:

Some comments from the translators

"Although I can generally understand the article, many of the sentences might not be easily understood by people who are not linguists. For example I instantly recognise the mistranslation of 'have/has' and can substitute the correct words, but this would probably be totally misleading for the general reader. The same is true of the lack of verbs, wrong word order etc." T4.

"There is a general problem with the translation of 'machine translation'. It is sometimes translated accurately, at other times translated as 'translation machine'. However, this does not really cloud the meaning in the context." T4.

"I wouldn't publish any phrases in this document apart from the sentences marked with 'Correct meaning'. However, if this tool was used so that a reader can understand the text on a website for example, it would be sufficient." T5.