Task & Data

Task definition

The objective of the task is to indirectly generate translations for three language pairs, based on already known translations among eight languages altogether, in 14 bilingual dictionaries, involving four possible paths – all from German to Brazilian Portuguese – that feature between 1 to 4 pivot languages.

The test dataset consists of 100 randomly-selected German dictionary entries with their translations into a second language, and recursively exploring further translations in chained-up dictionaries – including up to 817 entries with 1,532 translation equivalents in the largest language pair that is provided. Besides the headwords and translations, the data includes information about parts of speech, subject domains and synonyms, as well as examples of usage and their translations.

The following language pairs are provided for the four different paths:

  1. German>English | English>Portuguese
  2. German>Japanese | Japanese>Spanish | Spanish>Portuguese
  3. German>Danish | Danish>French | French>Spanish | Spanish>Portuguese
  4. German>Dutch | Dutch>Spanish | Spanish>Danish | Danish>French | French>Portuguese

Also included are four Portuguese>German datasets, for closing the loop in each path, to help with the validation of the results.

The three new language pairs that should be generated are:

  • German>Portuguese
  • Danish>Spanish
  • Dutch>French

NOTE: The Portuguese>German sets are provided for the sole purpose of closing the loop as an aid for validating the results and MAY NOT BE reversed to improve results. However, The Spanish>Danish dataset – that is provided as part of path (4) – MAY BE reversed to help with improving the results.

Evaluation of the results of each system will be carried out against KD’s manually compiled dictionaries for these pairs from the Global Series and other resources, as well as by human translators.

Participants can contribute on either or both of the following tracks:

  • Systems that use only the KD data released for the task
  • Systems that exploit, in addition to the KD data, other freely available sources of background knowledge (e.g., lexical linked open data and parallel corpora) to improve performance

Beyond performance, participants are encouraged to consider the following issues in particular:

  • The role of the language family with respect to the newly generated pairs
  • The asymmetry of pairs, and how translation direction affects the results
  • The behavior of different parts of speech among different languages
  • The role the number of pivots plays in the process


The test dataset is available here.

You can also download the publicly-available baseline algorithm for a simple example of an inference algorithm which gives a baseline for results (see results here).