CGB-DM: Content and Graphic Balance Layout Generation with Transformer-based Diffusion Model

Yu Li*, Yifan Chen*, Gongye Liu, Wu Jie, Yujiu Yang
Tsinghua University
*Indicates Equal Contribution
Paper Code arXiv
Teaser Image

Visual comparisons of recent state-of-the-art layout generation methods. Previous methods appear content and graphic imbalance problems, including block, overlap and misalignment from underlay to text, while our method properly improves space of layouts and presents coherent arrangement.

Abstract

Layout generation is the foundation task of intelligent design, which requires the integration of visual aesthetics and harmonious expression of content delivery. However, existing methods still face challenges in generating precise and visually appealing layouts, including blocking, overlap, or spatial misalignment between layouts, which are closely related to the spatial structure of graphic layouts. We find that these methods overly focus on content information and lack constraints on layout spatial structure, resulting in an imbalance of learning content-aware and graphic-aware features. To tackle this issue, we propose Content and Graphic Balance Layout Generation with Transformer-based Diffusion Model (CGB-DM). Specifically, we first design a regulator that balances the predicted content and graphic weight, overcoming the tendency of paying more attention to the content on canvas. Secondly, we introduce a graphic constraint of saliency bounding box to further enhance the alignment of geometric features between layout representations and images. In addition, we adapt a transformer-based diffusion model as the backbone, whose powerful generation capability ensures the quality in layout generation. Extensive experimental results indicate that our method has achieved state-of-the-art performance in both quantitative and qualitative evaluations. Our model framework can also be expanded to other graphic design fields.

Method

Architecture Image

Our framework of Content and Graphic Balance Layout Generation with Transformer-based Diffusion Model (CGB-DM), as well as (a) Content and Graphic Balance Weight Predictor (CGBWP), and the (b)Layout Decoder Transformer block of Layout Decoder.

Experiments

We present the comparison results of our method with other baselines, including:

  • GAN-Based models: CGL-GAN[1], DS-GAN[2]
  • VAE-Based models: ICVT[3]
  • Diffusion-Based models: LayoutDM[4] (Equipped with an identical Image Encoder to handle input images.)
  • RAG-Based models: RALF[5]

More comparison results can be found in our paper.

Unconstrained Generation
Visual-Comparison Image
Constrained Generation(By CGB-DM)
Constrain Image

More Comparisons

References

[1] Min Zhou, Chenchen Xu, Ye Ma, Tiezheng Ge, Yuning Jiang, and Weiwei Xu. Composition-aware graphic layout gan for visual-textual presentation designs. arXiv preprint arXiv:2205.00303, 2022.12.

[2] Hsiao Yuan Hsu, Xiangteng He, Yuxin Peng, Hao Kong, and Qing Zhang. Posterlayout: A new benchmark and approach for content-aware visual-textual presentation layout. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.

[3] Yunning Cao, Ye Ma, Min Zhou, Chuanbin Liu, Hongtao Xie, Tiezheng Ge, and Yuning Jiang. Geometry aligned variational transformer for image-conditioned layout generation. In ACM Int. Conf. Multimedia, 2022.

[4] Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Int. Conf. Comput. Vis., 2023.

[5] Daichi Horita, Naoto Inoue, Kotaro Kikuchi, Kota Yamaguchi, and Kiyoharu Aizawa. Retrieval-augmented layout transformer for content-aware layout generation. 2023.

BibTeX


      @misc{li2024cgbdmcontentgraphicbalance,
            title={CGB-DM: Content and Graphic Balance Layout Generation with Transformer-based Diffusion Model}, 
            author={Yu Li and Yifan Chen and Gongye Liu and Jie Wu and Yujiu Yang},
            year={2024},
            eprint={2407.15233},
            archivePrefix={arXiv},
            primaryClass={cs.CV},
            url={https://arxiv.org/abs/2407.15233}, 
           }