4.6 Article

Describing UI Screenshots in Natural Language

出版社

ASSOC COMPUTING MACHINERY
DOI: 10.1145/3564702

关键词

Captioning; visual saliency; deep learning; natural language processing

向作者/读者索取更多资源

XUI is a novel method that uses computational models to provide informative descriptions of user interfaces, including an overview and detailed descriptions, enhancing understanding of the UI's purpose.
Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose of the UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by the global precedence effect to create informative descriptions of UIs, starting with an overview and then providing fine-grained descriptions about the most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, and natural language generation (NLG). XUI provides descriptions with up to three different granularity levels that, together, describe what is in the interface and what the user can do with it. We found that XUI descriptions are highly readable, are perceived to accurately describe the UI, and score similarly to human-generated UI descriptions. XUI is available as open-source software.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

4.6
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据