Aim and Motivation

The successful development of industrial-strength technical solutions requires continuous testing and quality assessment on the basis of sophisticated and commonly accepted evaluation metrics. For this, real-world use case scenarios, scientific methodologies for testing, demonstrating, benchmarking, and comparing appear to be desirable.

The STI Test Beds & Challenges Service assembles, develops, and disseminates such services for semantic technologies. In particular, we:

• Provide open and publicly accessible methodologies and use cases for testing, demonstrating, comparing, and assessing semantic technologies
• Assemble use cases and related scientific results from research projects
• Organize events and competitions for demonstrating technologies, mostly co-located with major events (e.g. conferences, annual meetings)
• Disseminate the results and related initiatives to larger audiences
• Define a legal framework for reuse and adaptation of the test beds and challenges.

Therewith, we aim at facilitating a high quality scientific evaluation and comparison of technology developments, and also at providing support for demonstrating the benefits of semantic technologies to interested parties.

What We Offer

The STI Test Beds & Challenges Service (1) develops scientific methodologies for evaluating and assessing semantic technologies, (2) collects and integrates existing use cases, test beds, and challenges and continuously develops new ones, and (3) organizes respective workshops and events for presentation and dissemination of the results.

We distinguish 2 types of test environments that appear to be relevant for evaluating semantic technology developments:

(1) Test Beds are concerned with the technical evaluation of tools. This is usually done in the form of benchmarks. For this, a test bed provides a data set and defines specific criteria in order to test and compare technologies with respect to relevant measurements like interoperability, scalability, stability, etc.

(2) Challenges are concerned with the problem-solving capabilities of semantic technologies. The participants can demonstrate their technologies on the basis of a given use case scenario along with a problem description and a test data set.

The collected test beds and challenges cover all fields of semantic technologies that are addressed in STI research efforts. The test beds and challenges are provided in an open, publicly accessible manner. Apart from proving a suitable basis for evaluating and comparing technology developments, this appears to be promising means for demonstrating the capabilities and benefits of new semantic technology developments to industry and other interested parties. To strengthen the outreach, the results are continuously presented in major events for dissemination to larger audiences. The service is run by renowned researchers and practioneers with proved experience in running benchmarking activities and challenges.

Benefits

The following lists the benefits of the Test Beds & Challenges Service for the different stakeholders of STI International.