Stability evaluation for text localization systems via metamorphic testing

The success of learning techniques in solving a variety of hard AI problems promotes the flourish of recognition-based applications. Many state-of-the-art text localization systems, which can detect and report the positions of text segments in an image, are mainly implemented with learning-based techniques. Data-driven learning raises a series of questions on how to verify, validate and evaluate such learning-based systems. In this paper, we propose a methodology to automatically evaluate the stability of text localization systems via metamorphic relations, where a stable system should output consistent results for similar inputs with the same text segments. We introduce six metamorphic relations that should be preserved in a stable text localization system and define the corresponding metrics for stability evaluation. With the defined metamorphic relations, we apply metamorphic testing techniques to compare the inputs and outputs to evaluate system stability, and further diagnose the causes of inconsistency. The extensive experimentation on both academic and commercial text localization systems demonstrates the effectiveness of our method on stability evaluation for such systems. (C) 2021 Elsevier Inc. All rights reserved.

Automated summary

The paper proposes a methodology to automatically evaluate the stability of text localization systems using metamorphic relations, with six metamorphic relations defined for stable systems and corresponding metrics for evaluation. By applying metamorphic testing techniques based on the defined relations, system stability can be evaluated and causes of inconsistency diagnosed effectively. Experimental results on academic and commercial text localization systems demonstrate the effectiveness of the proposed method in stability evaluation.