[ACL 2024] Official PyTorch code for extracting features and training downstream models with emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation.
VoxFormer: a Cutting-edge Baseline for 3D Semantic Occupancy Prediction **VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion**, CVPR 2023. Yiming Li, Zhiding Yu, Chris Choy, Chaowei Xiao, Jose M. Alvarez, Sanja Fidler, Chen Feng, Anima Anandkumar [[PDF]]( [[Project]]( [[Intro Video]]( News [2023/07]: We release the code of voxformer with 3D deformable attention module, achieving slightly better performance. [2023/06]: 🔥 We release SSCBench, a large-scale semantic scene completion benchmark derived from KITTI-360, nuScenes, and
Waymo. [2023/06]: Welcome to our CVPR poster session on 21 June (**WED-AM-082**), and check our online video. [2023/03]: 🔥 VoxFormer is accepted by CVPR 2023 as a highlight paper **(235/9155, 2. [2022/11]: VoxFormer achieve the SOTA on SemanticKITTI 3D SSC (Semantic Scene Completion) Task with **13. Abstract Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI
systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the visual features on 2D images correspond only to the visible scene structures
rather than the occluded or empty spaces. Therefore, starting with the featurization and prediction of the visible structures is more reliable. Once we obtain the set of sparse queries, we apply a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20. 1% in semantics and reduces GPU memory during training by ~45% to less than 16GB. Given
RGB images, 2D features are extracted by ResNet50 and the depth is estimated by an off-the-shelf depth predictor. The estimated depth after correction enables the class-agnostic query proposal stage: the query located at an occupied position will be selected to carry out deformable cross-attention with image features. Afterwards, mask tokens will be added for completing voxel features by deformable self-attention. The refined voxel features will be upsampled and projected to the output space for per
-voxel semantic segmentation. Note that our framework supports the input of single or multiple images. * | Getting Started Installation Prepare Dataset Run and Eval Model Zoo The query proposal network (QPN) for stage-1 is available here. For stage-2, please download the trained models based on the following table. 35|config |model | | R50 | VoxFormer-S | 20ep | 44. 35|config |model| | R50 | VoxFormer-T-3D | 20ep | 44. 69|config |model| | R50 | VoxFormer-S-3D | 20ep | 44. 86|config |model| Dataset [x
] SemanticKITTI [ ] KITTI-360 [ ] nuScenes Bibtex If this work is helpful for your research, please cite the following BibTeX entry. License
Frequently asked questions
How many stars and forks are recorded?
NVlabs/VoxFormer has 1,212 stars and 102 forks.
What is the repository rank?
NVlabs/VoxFormer is ranked 11,124.
What language and category are recorded?
NVlabs/VoxFormer has 1,212 stars, uses Python and belongs to Models.
Which repository is closest in rank?
NVlabs/VoxFormer is compared with hgneng/ekho, ranked 11,121 with 1,213 stars.
README
VoxFormer: a Cutting-edge Baseline for 3D Semantic Occupancy Prediction
VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion, CVPR 2023.
Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the visual features on 2D images correspond only to the visible scene structures rather than the occluded or empty spaces. Therefore, starting with the featurization and prediction of the visible structures is more reliable. Once we obtain the set of sparse queries, we apply a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics and reduces GPU memory during training by ~45% to less than 16GB.
Method
Figure 1. Overall framework of VoxFormer. Given RGB images, 2D features are extracted by ResNet50 and the depth is estimated by an off-the-shelf depth predictor. The estimated depth after correction enables the class-agnostic query proposal stage: the query located at an occupied position will be selected to carry out deformable cross-attention with image features. Afterwards, mask tokens will be added for completing voxel features by deformable self-attention. The refined voxel features will be upsampled and projected to the output space for per-voxel semantic segmentation. Note that our framework supports the input of single or multiple images.
If this work is helpful for your research, please cite the following BibTeX entry.
@InProceedings{li2023voxformer,
title={VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion},
author={Li, Yiming and Yu, Zhiding and Choy, Christopher and Xiao, Chaowei and Alvarez, Jose M and Fidler, Sanja and Feng, Chen and Anandkumar, Anima},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2023}
}
This work is made available under the Nvidia Source Code License-NC. Click here to view a copy of this license.
The pre-trained models are shared under CC-BY-NC-SA-4.0. If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
