Test Beds & Challenges Service
 
 
Semantic Services Selection Contest

This annual contest is dedicated to the evaluation of the retrieval performance of Semantic Web service matchmakers. It takes place since 2007, and provides a comprehensive test bed of Web services and user queries (currently only OWL-S descriptions; WSMO and SAWSDL are planned for the future).

Website www-ags.dfki.uni-sb.de/~klusch/s3/
Provider S3 Contest Organisation Committee
Problem SWS matchmaking (primarily for discovery)
Data Set OWL-S Web service & user query descriptions + expected results
Description the annual competitions are usually co-located with ISWC conference

 
Semantic Web Challenge

The Semantic Web Challenge is decidated to allow participants to demonstrate the best of the Semantic Web - i.e. new applications and techniques that showcase the use and benefits of semantic technologies in comparison to conventional Web technology. It has established itself as a central part of the annual Semantic Web conferences and has received outstanding recognition.

Website challenge.semanticweb.org
Provider Semantic Web Challenge Organization Committee
Problem no specific problem (but: special tracks)
Data Set expected to be defined by participants
Description

the goals of the Semantic Web Challenge are

  • Facilitate the demonstration of what the Semantic Web can provide to society
  • Give researchers an opportunity to showcase their work and compare it to others
  • Stimulates current research to a higher final goal by showing the state-of-the-art every year

 
Semantic Web Service Challenge

The SWS Challenge is a widely recognized initiative for demonstrating, evaluating, and comparing technology developments for Semantic Web Services - i.e. the automated discovery, composition, usage, and mediation of Web services. Commenced in 2006, several workshops have been held where the technical solutions of the participants have been presented, discussed, and compared. The challenge will be continued in the future under the umbrella of STI International.

Website sws-challenge.org
Provider SWS Challenge Committee
Problem semantic WS discovery, composition, mediation, and execution
Data Set 5 test beds with WS descriptions, user goals, and expected results
Description The workshops are usually co-located with one of the major Semantic Web conferences. The initiative has received high attention in the last years, and the scientific results and best practices yield the W3C SWS Challenge Testbed Incubator activity.

 
Web Services Challenge

This is an annual competition held in conjunction with major conferences since 2005. Although dedicated to conventional, i.e. syntactical Web service technologies, the recent problem settings start to consider semantic aspects of XML data as well.

Website www.ws-challenge.org
Provider WS Challenge Consortium
Problem WS discovery, composition, execution (with syntactic technologies)
Data Set test repositories of syntactic Web Service descriptions
Description annual competitions in major software technology conferences with steadily increasing number of participants