The new frontier of computer assisted translation technology is the effective integration of statistical MT within the translation workflow. In this respect, the SMT ability of incrementally learning from the translations produced by users plays a central role.
A still open problem is the evaluation of SMT systems that evolve over time. In this paper, the authors propose a new metric for assessing the quality of an adaptive MT component that is derived from the theory of learning curves: the percentage slope.
For assessing its effectiveness, it was developed a simple but effective adapting SMT system suitable to work in the context of a CAT tool supported by MT. The authors have compared several ways to plot the change in error rate over time for different systems and identified the most suitable for computing the percentage slope. Finally, it was shown that the percentage slope well exposes the paradigmatic behaviors of evolving SMT systems.
Proceedings of the 7th Workshop on Statistical Machine Translation, pages 433–441, Montr ́ al, Canada, June 7-8, 2012.