-
Notifications
You must be signed in to change notification settings - Fork 25
Big memory consumption when tasks are slow to be consumed #17
Comments
@kpanic have you tried using Simplest way would be something like:
But you can also connect to a remote node and then use the observer from there. |
@joakimk yes, I tried and it seems that the memory consumption is coming from My use case is to fire up a lot of tasks while streaming from a file btw, I am not using phoenix at the moment |
I'll keep this issue open in case anyone (including myself) feels like trying to fix it :) One good first step could be to create a script, somewhat like https://github.com/joakimk/toniq/blob/master/test/benchmark.exs that can be used to see how toniq handles lots of waiting jobs. |
Just running into this, this line https://github.com/joakimk/toniq/blob/master/lib/toniq/job_concurrency_limiter.ex#L86 really isn't a viable option. As you say at the top of the file "if you enqueue 10000 jobs, this Not only does this add avoidable overhead, it produces a lot of garbage for the GC as well. Please use http://erlang.org/doc/man/queue.html. The theoretical analysis can be found in https://www.cs.cmu.edu/~rwh/theses/okasaki.pdf but the gist of it is that you maintain 2 lists. When you "append" to the queue you really just prepend to one of the lists, getting nice O(1) properties. Then when you actually need items, you pop O(1) off the second list. If there's no items, you reverse the first list and it becomes the second list. You pay an O(n) penalty just once and then it's good for another O(n) operations, so you get a nice amortized O(1) for all operations. |
@benwilson512 any chance you can open a PR to solve this ? |
I'll look into it this weekend. |
@benwilson512 Thanks a lot in advance and thank you for all the hard work on GraphQL 👍 |
Nice to see that people using Toniq more actively that me is working on this problem. I'll certainly accept a pull request that fixes this. |
@benwilson512 have you been able to look into this yet ? |
Can someone confirm if the above merged pull request in 1.0.6 fixes this issue? |
@joakimk I will try it (hopefully) during the next weekend! 📗 |
@joakimk I tried toniq 1.0.6, and I saw not considerable differences. |
Toniq is not meant to handle every kind of asynchronous task, after all, we do have OTP itself, but I expect this problem could be common enough that Toniq should handle it. What we should do is probably to add some kind of stress-test suite for Toniq since that's one important aspect of a job-queue. I might get around to this myself, but I have a few other projects I want to finish first so it could be quite a while before I do. Please do link to a branch that shows the issue if it's easy to do so someone can try to fix this. |
This might be related https://youtu.be/6yoJ8sWRiyg?list=PLE7tQUdRKcyYoiEKWny0Jj72iu564bVFD&t=1517 (erlangs handling of big binaries). |
If it turns out that holding many jobs in memory is impractical, maybe they can be flushed out to redis when the list is too long. Possibly as incoming jobs if we also add a limit on importing. Adding jobs to that list is a bit like I mention here: https://github.com/joakimk/toniq#load-balancing. We have to be careful to ensure takeover works well with that though. |
Does not work, but saving it for futher prototyping. If this turns out to be a useful idea it should be reimplemented cleanly with tests. Related to #17
We have to consider takeover when there are many jobs as well. The way it's written now it assumes that moving jobs over inside redis will be fast. That won't be the case if there are lots of jobs jcelliott@b32df75#commitcomment-19167803 |
@kpanic you could try the import_limits branch. It will ensure there are only 500 jobs in memory at a time. It's untested (other than simple manual verification in iex) but it would be interesting to know if it fixes your issues :) If it does, we can reimplement it propertly. |
This should ensure there are never more jobs loaded than the limit. Still only a prototype. #17
Hi @joakimk thanks for pushing it ahead! I will try this weekend hopefully (fingers crossed!) |
First of all, thanks for toniq! It got me started on a project of mine.
I was wondering if this line
toniq/lib/toniq/job_concurrency_limiter.ex
Line 86 in 930f35f
What do you think @joakimk ?
Thanks in advance!
The text was updated successfully, but these errors were encountered: