Syn

ACM International Conference on Intelligent User Interfaces (IUI), pp. 79–80

Abstract

User Interface design is an iterative process that progresses through low-, medium-, and high-fidelity prototypes. A few research projects use deep learning to automate this process by transforming low fidelity (lo-fi) sketches into front-end code. However, these research projects lack a large scale dataset of lo-fi sketches to train detection models. As a solution, we created Syn, a synthetic dataset containing 125,000 lo-fi sketches. These lo-fi sketches were synthetically generated using our UISketch dataset containing 5,917 sketches of 19 UI elements drawn by 350 participants. To realize the usage of Syn, we used it to train a UI element detector, Meta-Morph. It detects UI elements from a lo-fi sketch with 84.9% mAP and 72.7% AR. This work aims to support future research on UI element sketch detection and automating prototype fidelity transformation.

Authors 3

1. Vinoth Pandian Sermuga Pandian

   Fraunhofer Institute for Applied Information Technology

   Affiliation as printed

   Fraunhofer FIT, Sankt Augustin, Germany

2. Sarah Suleri Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

3. Matthias Jarke Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany