Benchmark construction and experimental evaluations for incoherent ontologies

As the core building blocks of the Semantic Web, ontologies provide shared vocabularies and conceptual knowledge for specific application fields. At the same time, ontologies can restrict their individuals and relationships through their semantic schema. However, logical conflicts of an ontology are often inevitably unavoidable in actual application scenarios when the ontology contains any form of disjointness or negation. Generally, logical conflicts can be divided into incoherence and inconsistency. Reasoning with incoherent ontologies may obtain many redundant relationships, and incoherence is a potential cause of inconsistency which seriously affects the correctness of semantic reasoning Therefore, handling incoherence is imperative, which mainly involves the fields of conflicts detection, ontology repair and justification computation. Various incoherent ontologies are indispensable to evaluate the proposed methods for handling incoherence. We provide a survey of relevant research works to study the proposed construction methods of incoherent ontologies. It is observed that incoherent ontologies in existing works still have the following limitations: (1) Lots of web pages used to download ontologies were temporarily constructed are now inaccessible; (2) Most of the existing incoherent ontologies were constructed in a simple way such as randomly adding disjointness axioms or merging ontologies through their alignments without considering all possible incoherence cases. To address the limitations, we propose a general framework to construct incoherent ontologies and design a hybrid algorithm to instantiate this framework. With the implemented construction methods, a comprehensive benchmark containing 116 ontologies is constructed. In our evaluations, incoherent ontologies in the benchmark are measured with 11 classic metrics. We then compare 5 representative ontology debugging systems and 3 repair methods. The evaluation results reveal that these ontologies could reflect different characteristics of each ontology system and repair method. All observations could guide researchers to select incoherent ontologies. The availability of our benchmark makes a contribution to the community of ontology debugging and repair fields. (c) 2022 Elsevier B.V. All rights reserved.

Automated summary

Ontologies, as the core building blocks of the Semantic Web, provide shared vocabularies and conceptual knowledge for specific application fields. However, logical conflicts often arise in actual application scenarios due to disjointness or negation in the ontologies. Incoherence and inconsistency are two types of logical conflicts. Handling incoherence is important as it can lead to inconsistency and affect the correctness of semantic reasoning. Various incoherent ontologies are essential for evaluating methods to handle incoherence.