Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CUDA out of memory issue. #119

Open
kunalkathare opened this issue Oct 29, 2024 · 3 comments
Open

CUDA out of memory issue. #119

kunalkathare opened this issue Oct 29, 2024 · 3 comments

Comments

@kunalkathare
Copy link

Hi I'm trying to run demo.py of 3D match on my own point clouds (there are around 100K points in each after downsampling to 0.025m), but I get,
RuntimeError: CUDA out of memory. Tried to allocate 8.38 GiB (GPU 0; 47.43 GiB total capacity; 42.66 GiB already allocated; 2.91 GiB free; 43.22 GiB reserved in total by PyTorch)

I am running it on a A6000 GPU.
Thanks in advance.

@cduch9
Copy link

cduch9 commented Nov 2, 2024

Exact same problem here, my point clouds have 60k points each, less than the typical point cloud in the 3DMatch database. It cannot be a simple question of "too large cloud". The end of the error stack says :

File "...\modules\ops\pairwise_distance.py", line 29, in pairwise_distance
   sq_distances = x2 - 2 * xy + y2
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 9.04 GiB. GPU 0 has a total capacity of 48.00 GiB of which 16.39 GiB is free. Of the allocated memory 18.28 GiB is allocated by PyTorch, and 11.93 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

I will try the proposed solution above but I suspect something else since everything works for the larger clouds in the provided datasets.

@cduch9
Copy link

cduch9 commented Nov 2, 2024

This solution seems better, I will try it first :
#96 (comment)

@cduch9
Copy link

cduch9 commented Nov 2, 2024

Well, instead of adjusting the parameter, I scaled down the 2 clouds in order to have about 5 units in size. It has the same effect of reducing the number of super points.

The memory problem disapeared but the allignment is way way off. The objects do not even touch each others! I will continue to investigate. Any hint will be welcomed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants