A Generic Evaluation Framework for Knowledge-Based Infrastructures: Design and Applications

Abstract

In this article, the authors introduce a generic framework for the evaluation of knowledge-based infrastructures, and present its applicability based on technologies developed within the Core Technology Cluster (CTC) of one of the most research-intensive programs of the German government toward new technologies for acquisition, processing, presentation and delivery of information on the Internet, the THESEUS program: (http://theseus-programm.de). The main components of the framework are presented, and the evaluation process is described, including also the identified steps that have to be completed in advance, such as the selection of appropriate evaluation criteria, metrics, and references. The flexibility of the presented framework enables its application for a wide range of technologies. This article reflects the application of the proposed framework for evaluation of software components that deal with key aspects toward the realization of the semantic web, namely: ontology reasoning, ontology mapping, machine learning algorithms used for learning semantic annotation from unstructured data, and document structure recognition.