ControlRoom3D: Room Generation Using Semantic Proxy Rooms

IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, pp. 6201–6210

Abstract

Manually creating 3D environments for AR/VR applications is a complex process requiring expert knowledge in 3D modeling software. Pioneering works facilitate this process by generating room meshes conditioned on textual style descriptions. Yet, many of these automatically generated 3D meshes do not adhere to typical room layouts, compromising their plausibility, e.g., by placing several beds in one bedroom. To address these challenges, we present ControlRoom3D, a novel method to generate high-quality room meshes. Central to our approach is a user-defined 3D semantic proxy room that outlines a rough room layout based on semantic bounding boxes and a textual description of the overall room style. Our key insight is that when rendered to 2D, this 3D representation provides valuable geometric and semantic information to control powerful 2D models to generate 3D consistent textures and geometry that aligns well with the proxy room. Backed up by an extensive study including quantitative metrics and qualitative user evaluations, our method generates diverse and globally plausible 3D room meshes, thus empowering users to design 3D rooms effortlessly without specialized knowledge.

Authors 13

1. Jonas Schult
   Affiliation as printed

   Meta GenAI

2. Lukas Höllein
   Affiliation as printed

   Meta GenAI

3. Bichen Wu
   Affiliation as printed

   Meta GenAI

4. Jialiang Wang
   Affiliation as printed

   Meta GenAI

5. Chih‐Yao Ma
   Affiliation as printed

   Meta GenAI

6. Kunpeng Li
   Affiliation as printed

   Meta GenAI

7. Xiaofang Wang
   Affiliation as printed

   Meta GenAI

8. Felix Wimbauer
   Affiliation as printed

   Meta GenAI

9. Zijian He
   Affiliation as printed

   Meta GenAI

10. Peizhao Zhang
    Affiliation as printed

    Meta GenAI

11. Bastian Leibe Aachen

    RWTH Aachen University

    Affiliation as printed

    RWTH Aachen University

12. Péter Vajda
    Affiliation as printed

    Meta GenAI

13. Ji Hou
    Affiliation as printed

    Meta GenAI