Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

24000 images, memory leak caused crashes during construction. #41

Open
lichangy opened this issue May 23, 2024 · 2 comments
Open

24000 images, memory leak caused crashes during construction. #41

lichangy opened this issue May 23, 2024 · 2 comments

Comments

@lichangy
Copy link

24000 images, memory leak caused crashes during construction.
The "prediction" method of MLModel leads to a continuous increase in memory.

@mazzzystar
Copy link
Owner

You're right. I tried to write a version using Claude 3.5 Sonnet that controls memory, for example, checking the current memory after each batch and then updating the batch size. This seems to be able to control memory usage. However, the problem is that the indexing speed becomes significantly slower.

If you don't mind the indexing speed slowing down, you can give it a try.

@antgiant
Copy link

antgiant commented Sep 8, 2024

I would be willing to give it a try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants