Panoramic Image Indoor Layout Estimation Using Vision Transformer (PanoViT)

The presentation focuses on a panoramic‑image based indoor layout estimation method called PanoViT, which aims to predict wall, ceiling, and floor lines from a single 2D image and reconstruct a 3D room model for applications such as VR house tours.

Research Background – Indoor layout estimation traditionally splits the task into detecting 2D structural lines and post‑processing them into a 3D model. While perspective images are limited, panoramic images provide a wider field of view and richer information, making them attractive for this task despite challenges such as distortion.

Method Overview and Results – PanoViT consists of four modules: a CNN backbone that extracts multi‑scale feature maps from the panorama, a vision‑transformer encoder that learns global relationships in the feature space, a layout prediction head that outputs wall, ceiling, and floor line maps, and a boundary‑enhancement module that emphasizes line features using frequency‑domain masking. The model also employs a recurrent position embedding to handle horizontal shifts in panoramas and a 3D loss computed directly in world coordinates to mitigate distortion effects. Experiments on the Matterport3D and PanoContext datasets show that PanoViT achieves state‑of‑the‑art performance on 2DIoU and 3DIoU metrics, surpassing previous methods such as LayoutNet, HorizonNet, HohoNet, and LED2‑Net in most cases. Ablation studies confirm the effectiveness of the recurrent position embedding, boundary‑enhancement, and 3D loss components.

How to Use PanoViT in ModelScope – Users can access the model on the ModelScope platform (https://modelscope.cn/home), search for “panoramic indoor layout estimation”, open the provided notebook, upload a 1024×512 panoramic image, adjust the image path in the example code, and run the notebook to obtain wall‑line predictions.

The talk concludes with acknowledgments of the speaker, Shen Weichao, a senior algorithm engineer at Alibaba, and thanks the audience.