The Conference and Labs of the Evaluation Forum, or CLEF, is an organization promoting research in multilingual information access. Its specific functions are to maintain an underlying framework for testing information retrieval systems and to create repositories of data for researchers to use in developing comparable standards. The organization holds a conference every September in Europe since a first constituting workshop in 2000. From 1997 to 1999, TREC, the similar evaluation conference organised annually in the USA, included a track for the evaluation of Cross-Language IR for European languages. This track was coordinated jointly by NIST and by a group of European volunteers that grew over the years. At the end of 1999, a decision by some of the participants was made to transfer the activity to Europe and set it up independently. The aim was to expand coverage to a larger number of languages and to focus on a wider range of issues, including monolingual system evaluation for languages other than English. Over the years, CLEF has been supported by a number of various EU funded projects and initiatives.
Structure of CLEF
Before 2010, CLEF was organised as a workshop co-located with the European Conference on Digital Libraries, consisting of a number of evaluation labs or tracks, similarly to TREC. In 2010, CLEF moved to become a self-sufficiently organised conference with evaluation labs, laboratory workshops, and a main conference track. In 2012, INEX, a workshop on retrieval and access to structured text, previously organised annually at Schloß Dagstuhl, merged with CLEF to become one of its evaluation labs. Prior to each CLEF conference, participants in evaluation labs receive a set of challenge tasks. The tasks are designed to test various aspects of information retrieval systems and encourage their development. Groups of researchers propose and organize campaigns to satisfy those tasks and the results are used as benchmarks for the state of the art in the specific areas., In the beginning, CLEF focussed mainly on fairly typical information retrieval tasks, but has moved to more specific tasks. For example, the 2005 interactive image search task worked with illustrating non-fiction texts using images from Flickr and the 2010 medical retrieval task focused on retrieval of computed tomography, MRI, and radiographic images. In 2017, CLEF accommodated a number of tasks e.g. on identifying biological species from photographs or video clips, on stylistic analysis of authorship, and on health related information access.
