|
Framework Programme 7 (2007-2013) Research infrastructures projects Summary: The SEALS project aims to address two key challenges: the creation of a lasting reference infrastructure for semantic technology evaluation (the SEALS Platform) and the continuous evaluation of semantic technologies at a large scale via public world-wide evaluation campaigns.
The SEALS Platform will be an independent, open, scalable, extensible and sustainable infrastructure that will allow the remote evaluation of semantic technologies by providing an integrated set of evaluation services and test suites. The SEALS Platform will be used in two public world-wide evaluation campaigns and the results of these evaluation campaigns will be employed in creating semantic technology roadmaps, identifying sets of efficient and compatible tools for developing large-scale semantic applications.
The semantic technology evaluation services will initially be available for five different types of technologies (ontology engineering tools, storage and reasoning systems, matching tools, semantic search tools, and semantic web service tools) and for different evaluation criteria (interoperability, scalability, etc.). The platform will provide easy and free access to the evaluation services and to the results of the evaluations performed, allowing researchers and users to effectively compare the available technologies, helping them to select appropriate technologies and advancing the state of the art through continuous evaluation. Objectives: To accomplish the vision of the SEALS project, the following scientific and technological objectives have to be achieved: Action plan: During a period of three years, the SEALS project will follow a two-phase spiral lifecycle model, being each phase delimited by one of the two planned world-wide evaluation campaigns that will provide valuable feedback to the work done.
In the first phase, the first public SEALS Evaluation Campaign will be defined and executed with the first prototype of the SEALS Platform and the first version of the evaluations, data sets, and software components for automatic evaluation. The results of this evaluation campaign will allow identifying semantic technology roadmaps and new requirements.
In the second phase of the project, the second public SEALS Evaluation Campaign will be carried out taking into account the lessons learnt in the first phase and the results of the evaluations performed in the SEALS Platform and its components. This process will lead to the development of a second prototype of the SEALS Platform and a second set of evaluations, data sets and components for automatic evaluation. The results of the second evaluation campaign will produce new semantic technology roadmaps and a basis for future evaluation campaigns. Networking activities: SEALS Networking Activities are devoted to foster a culture of co-operation between the different partners of SEALS, as well as between the SEALS project and the entire semantic technology community.
The dissemination and community building activities will establish a foundational network of technology users, providers, and researchers supported by the SEALS Platform. A foundational network is essential to obtaining sustainable research solutions, with promising potential to mature into marketable products. Besides, the SEALS evaluation campaigns will establish a strong link between the Networking, Service and Joint Research activities and will allow the validation of the evaluations and the SEALS Platform. Service activities: SEALS Service Activities are concerned with the development of the SEALS Platform for the automatic evaluation of a wide collection of semantic tools.
The SEALS Platform is built around six main components: the SEALS Service Manager, which provides access to the computational and data resources of the SEALS Platform and coordinates the other five components of the SEALS Platform; the Runtime Evaluation Service, which is designed to process and monitor a requested evaluation; and the Test Data Repository Service, the Tools Repository Service, the Results Repository Service, and the Evaluations Repository Service, which provide persistent storage for test data, tools, evaluation results, and evaluations, respectively. Joint Research activities: SEALS Joint Research Activities are driven towards the provision of well thought evaluations for different types of semantic technologies using different evaluation criteria, and towards the implementation of a collection of components, which will be integrated as part of the SEALS Platform, for automating the evaluation of such semantic technologies. The semantic technologies selected for evaluation have been grouped in five main categories, each led by a renowned research group in the Semantic Web community. These categories are the following: - Ontology engineering tools
- Storage and reasoning systems
- Matching tools
- Semantic search tools
- Semantic web service tools
User communities: SEALS will work towards the creation of an open and sustainable worldwide community focused on the evaluation and progressive development of semantic technology, which will survive the SEALS project. Our long-term goal is that the SEALS Platform be actively used and managed by the semantic community.
Therefore, SEALS is addressed to the entire semantic technology community, namely, researchers in semantic technologies, tool developers, and users. Participation will enable the establishment and diffusion of best practices in evaluation within the whole community. The semantic technology community will benefit by having access to a set of services that support the whole life cycle of the evaluation of the technologies. Therefore, this community will be able not just to evaluate tools by reusing the evaluations provided by the platform, but to define their own evaluations or to access the content stored in the platform (test data, results, etc.). International aspects:
The SEALS project answers an urgent need felt by the semantic technology community for evaluation of semantic technologies and their mutual compatibility. SEALS is expected to impact at a world-wide level, leading to a faster maturation of semantic technologies, which in turn should lead to an increased rate of adoption of research results by industry.
SEALS will innovate the way in which semantic technology is evaluated. The infrastructure developed within SEALS is, therefore, expected to provide future measuring sticks for both industry and academy when they evaluate their applications/innovations. Moreover, we plan to maintain the infrastructure up and running beyond the end of the project. Indirectly, this is expected to help accelerate innovation in all those fields in which evaluation mechanisms are provided, in the same way as the TREC benchmarks in information retrieval or the TPC benchmarks in database research do.
|