Enabling Secure Cross-Modal Retrieval Over Encrypted Heterogeneous IoT Databases With Collective Matrix Factorization

Significant volume of information of a broad variety (or modalities, such as image, audio, video, and text) is sensed and collected [such as those by the Internet of Things (IoT) devices] regularly (e.g., hourly). Such information is then analyzed to inform decision making, such as clinical diagnosis and product recommendation. Data with different representations may have the same semantic information, and there have been considerable efforts devoted to designing efficient searching approaches on objects with different modalities. However, multimodal data carry sensitive information, and maintaining privacy is crucial in our privacy-aware and interconnected society. In this article, we combine both the collective matrix factorization (CMF) and homomorphic encryption (HE) to construct an efficient and accurate scheme to facilitate cross-modal retrieval, without the loss of any sensitive information. Our scheme identifies the unified feature vectors for every object in the training set with different modalities and obtains the mapping matrices for out-of-sample objects. After the encryption process, these matrices are stored on the remote cloud server (CS). Hence, the server can calculate the secure, unified features for any query. In this article, we also built a privacy-preserving index structure using locality-sensitive hashing (LSH), which provides both security and efficiency. Performance evaluations demonstrate the potential for our proposed scheme in the real-world IoT applications.