Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] NDS-H data generator stalls after scale factor 150 #16837

Open
JayjeetAtGithub opened this issue Sep 19, 2024 · 0 comments
Open

[BUG] NDS-H data generator stalls after scale factor 150 #16837

JayjeetAtGithub opened this issue Sep 19, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@JayjeetAtGithub
Copy link
Contributor

Describe the bug
Currently, libcudf supports upto 32-bit integers. Hence, the NDS-H data generator which is built using libcudf can only generate upto 2,147,483,647 rows before it hits the row limit. For NDS-H, this translates to a scale factor of ~350 (The lineitem table, which is the largest NDS-H table, has ~6M rows, so the largest lineitem table that can be generated is of scale factor 350). But, when we tried running the NDS-H data generator with scale factor > 150, it stalled indefinitely. The highest scale factor that it was able to successfully generate was 150. This was observed both with un-managed and managed memory, and the execution stalled even though there was enough memory left.

Steps/Code to reproduce bug
Run the data generator with scale factor > 150

Expected behavior
The NDS-H data generator should be able to generate upto scale factor ~350.

Environment overview (please complete the following information)

  • Environment location: NVIDIA H100 and NVIDIA GH200
  • Method of cuDF install: Built from source, branch-24.10
@JayjeetAtGithub JayjeetAtGithub added the bug Something isn't working label Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant