Automatic code generation from low fidelity graphical user interface sketches using deep learning

Abstract

In software development, the user interface(UI) design process involves the part from an imagination that begins on a piece of paper where designers share thoughts to the final prototype that is ready to implement. UI designers use a technique of paper prototyping in low-fidelity sketches to implement the conceptual design. Once a design is drawn, it is usually captured within a photograph, then converted to a high-level mockup, and manually translated into a working client-side code by developers. The process consumes considerable effort and also delays the design process. Recently, researchers attempted to recreate UI from their sketches using code generation techniques. In this paper, we present a system for automatic code generation that combines recent advances in computer vision and deep learning for translating a sketched wireframe into a UI skeleton code. The proposed system has two main components: component recognition and code generation. The component recognition learns to extract visual features and their spatial layouts from sketched UI images. The code generator creates the equivalent UI skeleton codes based on the information from the detection model. A layout algorithm has been developed to resolve the overlapping of components and structure the layout. The evaluation result of the component recognition model indicates that the Faster R-CNN with Inception ResNet v2 backbone model achieved a mean average precision (mAP) of 95.5 %, among the other trained models. To evaluate the generator, we conducted a user study based on visual similarities between input image and the rendered UI.

Tasks

Reproductions