Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve the performance of imported_keylist deconstruction #239

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

DarrenJiang13
Copy link

Problem

When I run memtier_benchmark with --data-import (including 500000 keys), it took a long time to quit after benchmark finished.
With pstack, I found the process was waiting in ~imported_keylist()

...
#6  std::vector<imported_keylist::key*, std::allocator<imported_keylist::key*> >::erase (__position=..., this=0x1e685e8) at /usr/include/c++/9/bits/stl_vector.h:1428
#7  imported_keylist::~imported_keylist (this=0x1e685e0, __in_chrg=<optimized out>) at obj_gen.cpp:534
#8  0x0000000000407a1a in main (argc=<optimized out>, argv=<optimized out>) at memtier_benchmark.cpp:1641

Solution

erase() one by one for a dead vector wastes some time. So I replaced erase() with clear(), which makes it much faster.

obj_gen.cpp Outdated
while (!m_keys.empty()) {
free(m_keys.front());
m_keys.erase(m_keys.begin());
for (key* k : m_keys) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's stick to the old style for old compiler compatibility.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I just changed the loop style.

obj_gen.cpp Outdated
m_keys.erase(m_keys.begin());
}
free(m_keys.front());
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it will work... we are not removing any element from the array... so m_keys.empty() always return false.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for that mistake. There must be something wrong with my brain yesterday.
I just change the code style with a for loop.

@DarrenJiang13 DarrenJiang13 force-pushed the fast-free-keylist branch 3 times, most recently from 17956d6 to ff46b67 Compare November 3, 2023 07:07
@DarrenJiang13
Copy link
Author

DarrenJiang13 commented Nov 3, 2023

Test command

./memtier_benchmark -p 6379 -t 2 -c 8 -n 500000 --ratio 1:0 --data-import ./dump100.csv

Test result:

key numbers time cost to quit after finishing benchmark (second)
before commit 500000 175
after commit 500000 1

Supplement

dump100.csv.zip

@DarrenJiang13
Copy link
Author

@filipecosta90 hi could you please check this PR? I think it is ready to be merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants