OSTRA

OSTRA is a novel segmentation-then-reconstruction method for segmenting complex open objects in 3D point clouds. This method uses a Segment-Anything Model (SAM) to segment target objects and video object segmentation (VOS) technology to continuously track video frame segmentation targets. Our pipeline enables a complete segmentation process from videos to 3D cloud points and meshes in different level(semantic segmentation, instance segmentation and part segmentation).

You can check our detailed tutorials here!

💻Getting Started

Clone OSTRA

This project is tested under python3.9, cuda11.5 and pytorch1.11.0. An equivalent or higher version is recommended.

#Clone OSTRA
git clone --recursive https://github.com/ganlab/OSTRA.git
cd OSTRA

#Install dependencies:
pip install -r requirements.txt

Install Colmap

Our reconstruction process is based on Colmap. Please follow the instruction and install Colmap first.

Prepare Model

All these models are required for OSTRA: SAM: the default model is sam_vit_b_01ec64.pth DeAOT:the default model is R50_DeAOTL_PRE_YTB_DAV.pth XMem：the default model is XMem-s012.pth Grounding-DINO:the default model is groundingdino_swint_ogc.pth

WebUI

We developed WebUI that user can easily access.

python app.py --device cuda:0

📽️ Demo

Two samples of complex object segmentation:

📖Citation

Please considering cite our paper if you find this work useful!

@misc{xu2023stop,
      title={A One Stop 3D Target Reconstruction and multilevel Segmentation Method}, 
      author={Jiexiong Xu and Weikun Zhao and Zhiyan Tang and Xiangchao Gan},
      year={2023},
      eprint={2308.06974},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License

Free for non-profit research purposes. Please contact authors otherwise. The program itself may not be modified in any way and no redistribution is allowed. No condition is made or to be implied, nor is any warranty given or to be implied, as to the accuracy of PROPA, or that it will be suitable for any particular purpose or for use under any specific conditions, or that the content or use of PROPA will not constitute or result in infringement of third-party rights.

👏 Acknowledgements

This work is based on Segment Anything, Track Anything, Segment and Track Anything, Colmap and Open3D. The software is developed by following author(s) and supervised by Prof. Xiangchao Gan([email protected])

Authors:

Jiexiong Xu [email protected] work: framework and reconstruction module

Weikun Zhao [email protected] work: VOS module