(alphabetical order of presenter)
Hanna Bechara |
Intelligent Translation Memory Matching and Retrieval Metric Exploiting Linguistic TechnologyTranslation Memories (TM) help translators in their task by retrieving previously translated matches and editing fuzzy matches when no exact match is found in the system. Current TM systems use simple edit distance or some variation of it, which largely rely on the surface form of the sentences and does not necessarily reflect the semantic similarity of segments as judged by humans. In this paper, we propose an intelligent metric to compute the fuzzy match score which is inspired by similarity and entailment techniques developed in Natural Language Processing. Translation Memories (TM) help translators in their task by retrieving previously translated matches and editing fuzzy matches when no exact match is found in the system. Current TM systems use simple edit distance or some variation of it, which largely rely on the surface form of the sentences and does not necessarily reflect the semantic similarity of segments as judged by humans. We have adapted our SemEval textual similarity and entailment system (Gupta et al., 2014) to measure similarity for TM segments. Given the amount of calculation involved in the task, we kept only those features which can be quickly calculated and have the most impact from the original system. The system uses features based on surface form, parts of speech information, lemma, typed dependency parsing, named entities, paraphrasing, machine translation evaluation, and corpus pattern analysis (Hanks, 2013). Stanford CoreNLP3 toolkit (Manning et al., 2014) provides lemma, parts of speech (POS), named entities, and dependencies relations of words in each sentence. We used the PPDB paraphrase database (Ganitkevitch et al., 2013) to get the paraphrases. After getting all features, we employed a support vector machine in order to build a regression model to predict semantic similarity using 5000 parallel sentences of the SICK dataset (Marelli et al., 2014), which contains human similarity scores as a training corpus. We carried out a preliminary evaluation on 500 sentences as input and 5000 sentences as a TM from DGT-TM corpora(Steinberger et al., 2012), using English as source and French as target. A target side (French) of 500 sentences input was considered as reference for evaluation. We used the edit-distance measure implemented by OmegaT as a baseline. The Meteor (Denkowski and Lavie, 2014) and BLEU (Papineni et al., 2002) scores were used in our evaluation. For each of the 500 input sentences, we retrieved the most similar sentence (and their proposed translation into French) as indicated by the baseline and our similarity metric. The proposed method yields better results for both BLEU and Meteor scores. The BLEU score improved from 7.20 to 7.80 and Meteor score improved from 0.175 to 0.193. Furthermore, when restricting the retrieval to 70% and above threshold, the Meteor score improved from 0.915 to 0.926 and BLEU score improved from 77.32 to 81.61. We are in a stage of improving and speeding up our system to create a larger-scale evaluation. The initial results, as stated earlier, show improvements in spite of training the system on a different genre of corpus. The SICK dataset consists of mostly simple sentences extracted from image captions while DGT-TM corpus has much larger and complex sentences. ————- Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation. |
||||||||||||||||||||||||||||||||||||||||||||
Hernan Costa |
iCompileCorpora: A Web-based Application to Semi-automatically Compile Multilingual Comparable CorporaCo-authors: Hernani Costa, Gloria Corpas Pastor and Miriam Sighiri The manual layer represents the option of compile monolingual or multilingual corpora. It will allow to manually upload documents from a local or remote directory into the platform. As nowadays there is an increasing demand for systems that can somehow cross the language boundaries by retrieving information of various languages with just one query, the third layer aims to answer this demand by taking advantage of CLIR techniques to find relevant information written in a language different from the one semi-automatically retrieved by the methodology used in the previous layer. The benefits of using corpora in translation have been demonstrated by various authors (cf. Zanettinet al., 2003; Corpas Pastor and Seghiri, 2009). The main advantages of its usage are their objectivity, reusability, multiplicity and applicability of uses and easy handling and quickly access to large volume of data. Moreover, they are a suitable tool for translators, as they can easily determine how specific words and their synonyms collocate and vary in practical use. Furthermore, in the last decade, a growing interest in bilingual and multilingual corpora have been shown by researchers working in other fields, such as terminology and specialised language, automatic and assisted translation, language teaching, Natural Language Processing, amongst others. Although corpora can be manually compiled, nowadays specialised tools can be used to automate this tedious process. Even though they were designed for other purposes, Terminus <http://terminus.iula.upf.edu//cgi-bin/terminus2.0/terminus.pl> and Corpogr´afo <http://www.linguateca.pt/corpografo/> should also be mentioned as examples of web-based compilation tools. As we can see, several semi-automatic compilation tools have been proposed so far. Nevertheless, their simplicity, lack of features, performance issues and usability problems result in a pressing need to design compilation tools tailored to fulfil not only translators and interpreters needs, but also professionals and ordinary people. Departing from a careful analysis of their weaknesses and strengths, we started by designing and developing a robust and agile web-based application to semi-automatically compile monolingual and multilingual comparable corpora, which we named iCompileCorpora. The dimensions that comprise iCompileCorpora can be represented in a layered model comprising a manual, a semi-automatic web-based and a semi-automatic Cross-Language Information Retrieval (CLIR) layer. This design option will permit not only to agile the compilation process, but also to hierarchically extend the manual layer features to the semi-automatic web-based layer and then to the semi-automatic CLIR layer. In detail, the manual layer represents the option of compile monolingual and multilingual corpora. It will allow to manually upload documents from a local or remote directory into the platform. The second layer will permit to exploit both monolingual and multilingual corpora mined from the Internet. Although this layer can be considered similar to the approaches used by BootCaT and WebBootCat, it has been designed to address some of their limitations (e.g. admit the use of more than one boolean operator when creating search query strings), as well as to improve the User Experience (UX) with this type of software. As nowadays there is an increasing demand for systems that can somehow cross the language boundaries by retrieving information of various languages with just one query, the third layer aims to answer this demand by taking advantage of CLIR techniques to find relevant information written in a language different from the one semi-automatically retrieved by the methodology used in the previous layer. To conclude, this article presents an ongoing project that aims to agile the compilation of monolingual and multilingual comparable corpora. iCompileCorpora intends to fulfil not only translators and interpreters needs, but also professionals and ordinary people, either by breaking some of the usability problems found in the current compilation tools available on the market or by improving their limitations and performance issues. Baroni, M. and Bernardini, S. (2004). BootCaT: Bootstrapping Corpora and Terms from theWeb. In 4th Int. Conf. on Language Resources and Evaluation, LREC’04, pages 1313–1316. |
||||||||||||||||||||||||||||||||||||||||||||
Sabine Hunsicker |
Machine Translation Quality Estimation Adapted to the Translation Workflow
No short abstract available. We invite you to read the full text below. 1 Introduction The varying quality of MT poses a problem in the translation workflow as it requires different levels of post-editing effort. As one sentence may be translated well, the next one may turn out to be completely unusable. There might be cases where discarding the MT suggestions and trans-lating from scratch is going to be faster. This de-cision time also increases the post-editing time. In order to be able to exploit the full potential of MT suggestions, they should be annotated with a score that is indicative for the translation quality. The translators are already familiar with such scores from the usage of translation memory (TM). MT does not assign a score to its output. The decoder calculates internal scores to find the best hypothesis from the translation options, but these decoder scores cannot be used to estimate a level of quality. For an LSP, a predictive score for MT quality would be very useful, as this would be in line with the way TMs are used in the workflow. An-other important advantage is that it provides an upfront estimation of the cost for a given translation. 2 Confidence Score The translation workflow at euroscript involves several MT-related stages. One of these stages contains the quality estimation component which we call confidence score, e.g. a component that would answer the question on ‘how confident is the MT that a particular sentence was well-translated? ‘ In order to reduce the annotation effort, we developed this score starting from the automatic scores. The approach can be easily automated so that it can be run immediately after training a new MT system. Another advantage is that there is no time lost in finding human annotators and for data creation before the new MT system is deployed in production. The prediction model makes use of a combina-tion of system-independent and sys-tem-dependent features. For example, the sentence lengths of the source, the reference and the MT candidate are taken into account. The sys-tem-dependent features vary on the MT system that should be evaluated. SMT systems usually provide different scores calculated during decod-ing. Each training instance includes the source sentence, the target sentence, the MT candidate, the feature vector and the automatic score. The training algorithm automatically chooses a well-distributed sample of training instances to train the prediction model. As the confidence score is integrated into the MT workflow, each MT request is automatically annotated with the confidence score. The confidence score is optimized to predict which of the following levels of quality the cur-rent translation belongs to:
3 Experiments For exemplification, we present our confidence score experiments on the language direction Eng-lishDanish. The texts used in our experiments come from the public domain. The training data was created by translating these texts with MT and then post-editing the results. As such, the translations are usually very close to the MT candidates, ex-cept where MT was of such a bad quality that it was discarded. The test set contains 1074 sentences. During translation with MT, the automatic scores for the classifier were collected and a con-
fidence model was trained on the resulting data. After integration into the translation workflow, the model was evaluated on unseen texts from the same domain. We trained prediction models for the follow-ing four automatic scores:
To compare the different metrics, we calculate three types of measures: the mean absolute error (MAE), the root mean squared error (RMSE) and Pearson’s correlation coefficient (r).
Table 1. Evaluation statistics for ENDA confidence score correlation.
Table 1 shows the evaluation results for all four prediction models. We see that the error rates differ considerably between the evaluation met-rics and the error rates. The prediction model based on the editing distances performs quite well: it achieves the lowest error rates and corre-lates moderately with the score it tries to predict. As predicting the full range of scores is a very complex task, we decided to scale down and only predict the three quality levels described in Section 2. To determine the thresholds of these levels, we ran two experiments, one with a very high level (95% for good, 75% for usable) and the other with a moderate level (75% for good, 50% for usable). From these experiments, we can tell that the moderate quality levels are easier to predict, as we achieve higher correlation values with them. The editing distance model performs well here as well: choosing the minimum thresholds, we achieve a correlation of 0.2588 of the confidence score to the actual editing distance score. In a human evaluation, we used a random sample of the evaluation data to be judged by professional translators. Of 221 sentences, 102 scores were judged to be appropriate and 119 to be inappropriate, indicating that the scoring mechanism needs more fine-tuning, but that it is still usable. 4 Conclusion We presented a practical example on how to in-corporate the confidence score into the tradition-al translation workflow. The first prototype of our confidence score works well in our production set-up. To fine-tuning the predictions require more evaluation and data, both of which are cre-ated automatically during the translation post-editing in production. |
||||||||||||||||||||||||||||||||||||||||||||
Emmanuel Rayner
Emmanuel (Manny) Rayner is a Collaborateur Scientifique (senior non-teaching research post) at the University of Geneva’s Multilingual Processing Group, and has previously held positions at SRI International (1991-1999) and NASA Ames Research Center (1999-2005).
Over the last 20 years, his research has focused on construction of speech-enabled dialogue systems.He has played key roles in several major projects in this area, including SRI’s Spoken Language Translator, NASA’s Clarissa and the Open Source Regulus platform At the University of Geneva, he has worked primarily with speech translation and CALL. He has more than 100 refereed publications in speech technology, computational linguistics, machine translation and artificial intelligence.
 Alejandro Armando is a certified translator graduated from the School of Languages, Universidad Nacional de Córdoba, Argentina.He works as a freelance translator for a number of international organizations and private sector clients.He is currently finishing a Master of Arts in Translation at the University of Geneva and conducting research on automatic translation and speech recognition. Nikos Tsourakis is also a Collaborateur Scientifique at UNIGE’s Multilingual Processing Group. He has played important roles in several UNIGE projects, where his work has primarily been concerned with deployment and evaluation of speech-enabled systems on mobile platforms. He has over 30 refereed publications in speech technology.
|
A Tool for Building Multilingual Voice Questionnaires
We describe an easy-to-use architecture which can be used to generate multilingual voice questionnaires, deployed on mobile platforms linked over a 3G connection to a remote server. The person administering the questionnaire chooses the next question by speaking it in their own language. The application uses speech recognition to identify it, speaks a pre-recorded translation in the respondent language, and displays a set of icons on the touch-screen. The respondent answers by pressing an icon. The questionnaire designer specifies the questionnaire using a simple format, in which there are three types of unit: Groups, Questions, and Answers; this is compiled into an accurate limited-vocabulary speech recogniser and a set of tables. The value of the tool, as compared to an application which mechanically asks a set of questions in a fixed order, is that it allows human judgement to be used to select appropriate questions within each Group; the situation may make some choices far more appropriate than others. The paper describes the tool in detail, taking as the main example an initial case study using a questionnaire designed for gathering information about information related to availability of anti-malaria measures in sub-Saharan Africa. There are many circumstances where it is useful to be able to administer multilingual voice questionnaires. A familiar example in Western society is admission to the accident and emergency room of a hospital: the nurse on duty will most likely start by asking for personal details, the nature of the patient’s immediate problem, previous medical history, and so on. If the nurse and the patient do not share a common language, difficulties arise. Another example, which will occupy us more in this paper, is information gathering for demographic and health topics (DHS Program 2014). We describe an easy-to-use architecture, inspired by the RAMP framework (Salihu 2013) which can be used to generate voice questionnaires of this type. Our questionnaires are deployed on mobile platforms — smartphones, tablets or laptops — linked over a 3G connection to a remote server. The person administering the questionnaire chooses the next question by speaking it in their own language. The application uses speech recognition (performed on the server) to identify it, speaks a pre-recorded translation in the respondent language, and displays a set of icons on the touch-screen. The respondent answers by pressing on the icons; each icon has an associated voice recording, in the respondent language, identifying its function. The questionnaire designer specifies the questionnaire using a simple format, in which there are three types of unit: Groups, Questions, and Answers. A Group specifies a top-level item on the questionnaire, a list of permitted Fillers, and a pointer to the next Group. A Question specifies one possible way to attempt to assign a Filler to a Group; it defines the Group which the Question belongs to, a list of surface realization of the question in the questionnaire administrator’s language (specified compactly in a minimal regular expression notation), a translation in each target language, and a list of permitted Answers, each one optionally associated with a Filler. An Answer defines a visual icon and an associated translation for each target language. The questionnaire description is compiled into an accurate limited-vocabulary speech recogniser and a set of tables. It is deployed over the web using methods developed at Geneva University and elsewhere in the context of various speech translation and CALL projects (Rayner et al 2006, 2014; Fuchs et al 2012). The value of the tool, as compared to an application which mechanically asks a set of questions in a fixed order, is that it allows human judgement to be used to select appropriate questions within each Group. For example, four of the Questions associated with a Group “Religion” whose possible Fillers are “Christian”, “Muslim” and “No religion”. might be realized as “What is your religion?” “Do you have a religion?”, “Are you a Muslim?” and “Are you a Christian?” Evidently, the situation may make some choices more appropriate than others; in certain cases, the difference may be very important. In the full paper, we will describe the tool in more detail, taking as our main example an initial case study using a questionnaire, consisting of 45 Groups and about 300 Questions, designed for gathering information about information related to availability of anti-malaria measures in sub-Saharan Africa. The overall goal is to measure nine internationally recognized vector control indicators that can be generated through household surveys. Some of the questions are about preventive measures — namely, the use of mosquito nets and indoor residual spraying — and bear a more immediate relevance to such indicators; others are about the characteristics of the household’s dwelling unit and are important to establish a connection between the socioeconomic status of the population and public health issues. All questions were extracted from model questionnaires recommended by the cluster of humanitarian actors that developed the indicators, which include the World Health Organization and the Roll Back Malaria Partnership. Our evaluation focusses in particular on questions of usability, habitability, rapid development of content, and quality of speech recognition performance. DHS Program (2014). http://dhsprogram.com/
|
Pierrette Bouillon is a full professor at Geneva University’s Faculty of Interpretation and Translation, where she also holds the position of Vice-Dean. She has published extensively in the fields of machine translation, lexical semantics and speech technology, and has acted as PI on numerous projects, most recently ACCEPT, a Seventh Framework project whose focus is machine translation for user generated text, and CALL-SLT, a Swiss National Science Foundation project centered on development of methods for speech-enabled CALL systems.