MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graph

Entity Alignment (EA) is a crucial task in knowledge fusion, which aims to link entities with the same real-world identity from different Knowledge Graphs (KGs). Existing methods have achieved satisfactory performance, however, they mainly focus on single modal KG, which is difficult to be effectively applied to multi-modal scenes. In this paper, we propose a Multi-modal Joint entity Alignment Framework (MultijAF), which can effectively utilize the knowledge of various modalities. Concretely, we first learn the embeddings of different modalities, i.e., structure, attribute and image modalities. Next, we adopt an attention-based multi-modal fusion network to integrate these embeddings and use obtained joint embeddings to compute a joint embedding-based similarity matrix S-J. Moreover, we design a Numerical Process Module (NPM) to infer a similarity matrix S-N according to the numerical information of entities. In the end, we utilize a simple late fusion method to ensemble two similarity matrices for the final alignment. In addition, to reduce the cost of labeling data, we propose a novel NPM-based unsupervised multi-modal EA method. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed MultijAF. (C) 2022 Elsevier B.V. All rights reserved.

Automated summary

In this paper, a Multi-modal Joint entity Alignment Framework (MultijAF) is proposed to effectively utilize the knowledge of various modalities for entity alignment. By learning embeddings of different modalities, using a multi-modal fusion network, and designing a Numerical Process Module (NPM), the framework achieves satisfactory alignment performance. Additionally, an unsupervised multi-modal EA method is introduced to reduce the cost of labeling data.