In recent times, there has been a growing interest in the development of a 3D language field to facilitate open-ended language queries in three-dimensional space. This approach holds great promise for advancing human-computer interaction and comprehension, offering applications in areas such as robotic navigation, 3D semantic understanding, autonomous driving, and augmented/virtual reality.
Despite the prominence of existing methods like LERF, their practical applicability is hindered by significant limitations in both speed and accuracy. To overcome these challenges, in a new paper LangSplat: 3D Language Gaussian Splattin, a research team from Tsinghua University and Harvard University introduces LangSplat, a groundbreaking 3D Gaussian Splatting-based method designed for 3D language fields. Notably, LangSplat surpasses the state-of-the-art LERF method while boasting a remarkable speed improvement of 199 times.
LangSplat stands out as the pioneering 3D Gaussian Splatting-based method tailored for 3D language fields. Departing from the conventional use of NeRF for constructing 3D representations, the researchers employ 3D Gaussian Splatting. This innovative approach represents a 3D scene as an amalgamation of 3D Gaussians and utilizes tile-based splatting for efficient rendering at high resolutions. The method involves defining a set of 3D language Gaussians, each enriched by a language embedding. These language-enhanced Gaussians undergo supervision through CLIP embeddings, extracted from image patches obtained from multiple training views, ensuring multi-view consistency.
In a bid to minimize memory costs and enhance rendering efficiency, the research team proposes the adoption of a scene-wise language autoencoder. This autoencoder maps CLIP embeddings in a scene to a low-dimensional latent space, ensuring that each language Gaussian contains only low-dimensional latent language features. The final language embeddings are then obtained through the decoding of the rendered features.
To address the issue of point ambiguity, the researchers advocate the use of the semantic hierarchy outlined by the Segment Anything Model (SAM). Learning with SAM-based masks not only imparts precise CLIP embeddings to each point, enhancing model accuracy, but also facilitates direct querying at predefined semantic scales. This eliminates the need for extensive searches across multiple absolute scales and auxiliary DINO features, thereby significantly improving efficiency.
Experimental results unequivocally showcase LangSplat’s superiority over existing state-of-the-art methods like LERF. This is particularly evident in its remarkable 199-fold speed improvement and enhanced performance in open-ended 3D language query tasks, underscoring the method’s potential for transformative impact in the field.
The paper LangSplat: 3D Language Gaussian Splatting on arXiv.
Author: Hecate He | Editor: Chain Zhang
We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.
Osh University and national state university are committed to shaping the future of medical professionals in Kyrgyzstan. Their partnership provides an excellent platform for students to excel in the healthcare field.
Shalamar Hospital is your destination for top neurological care in Lahore, with the best neurologist in Lahore providing expert consultations and treatments.
Presenting Tempo Garments: an elegant blend of sophistication and flair. Discover our exquisite selection of shirt, each one created with care and accuracy. Tempo Garments superior craftsmanship and timeless elegance will boost your wardrobe with everything from basic essentials to modern trends.