Personal Translator User Test
The well-reputed Fraunhofer Institute for Industrial Engineering IAO extensively tested Personal Translator with regard to efficiency and usability. Forty candidates, each with different levels of knowledge of the English language, translated technical texts and business letters using Personal Translator and, also as an alternative, using only a conventional dictionary.
According to tests carried out by relevant specialist magazines the translation program Personal Translator (PT) has become the best automatic translation software on the market within only a few years. To test the extent to which Personal Translator helps increase the quality and efficiency of a translation as compared with the use of a conventional dictionary, the Usability Engineering marketing strategy team of the Fraunhofer IAO carried out a usability test with potential users.
The test focused on the extent to which Personal Translator simplifies the completion of typical user tasks as compared with the use of a conventional dictionary (on paper). These typical tasks included:
In detail, the following points were tested (each compared with the use of a conventional dictionary):
In this way, the three most important usability aspects of an interactive software product were analyzed in accordance with ISO 9241-11. The test evaluated the extent to which PT can be used by specified users to translate and write texts effectively, efficiently and with satisfaction (ISO 9241-11, ISO, 1996). The test also evaluated whether the use of PT saved time and increased productivity. Another interesting aspect was the extent to which PT helped users with a specific level of knowledge to achieve better results with different types of text. In addition, optimization recommendations for the design of the translation software were derived from the test.
2. Operationalization of the measured variables
Effectiveness characterizes the quality of the performed task. To operationalize effectiveness, the results of the participants were assessed separately by four teachers of the English language and graded from 0-15. To avoid biased evaluations due to the origin of the results, the assessment was carried out blind, i.e. the teachers did not know that the test was an evaluation of PT. They did not know the number of test subjects or that they were given two texts from each person. Once the test was completed, the texts were all numbered so that the test managers were able to link them to the test subjects. The teachers, however, had no idea who had written the texts and where they came from. They were only told that different translation aids were to be compared.
As the second criterion of usability, efficiency measures the resources expended in relation to the effectiveness: efficiency decreases with growing expenditure of resources. The expenditure of resources was measured using both objective and subjective characteristic variables. The time required for completing the task was an objectively measurable value used in the evaluation. Subjective variables were the ratings of the test subjects in terms of effort and time required on two scales from 0-100.
The satisfaction with the application was taken into account via a subjective scale: the test participants specified how much fun they had had in completing the task.
Click here for full details of the test (in German).