Check failed: (best_split_info.left_count) > (0)

[chixujohnny]

Hi, I found a bug when training with large X_train.

lgb-gpu version: 3.3.2
CUDA=11.1
CentOS
ram=2TB
GPU=A100-40G

when the X_train is more than (1800w, 1000), lgb-gpu will has a bug like this:
[LightGBM] [Fatal] Check failed: (best_split_info.left_count) > (0) at LightGBM/src/treelearner/serial_tree_learner.app, line 686

When I use LGB==3.2.1 , I have the same problem as #4480 : when the GPU memory more than 8.3G will Memory Object Allocation Failure

in this LGB version==3.3.2 , LGB can't load more than 17G GPU memory (GPU has 40G memory), it seems like something problem occur in the tree split step and only happened when GPU memory loaded more than 17G.

Another colleague has this same problem, his lgb version is 3.3.1

[jameslamb]

Thanks very much for using LightGBM!

I suspect that this is the same as other issues that have been reported (e.g. #4739, #3679), but it's hard to say without more details.

Are you able to provide any (and hopefully all) of the following?

• specific commands you used to install LightGBM
• the most minimal possible complete version of the code you're running which can reproduce this issue

That would be very helpful. Without such information and with only an error message, significant investigation will probably be required to figure out why you encountered this error.

[chixujohnny]

Thanks for your reply!

I build the source code in this command:
mkdir build ; cd build ; cmake -DUSE_GPU=1 .. && make -j

Sorry I can't paste my code because the code is on my comany machine. I can't screenshots or copy the code, but I can descript my code:
X = np.random.rand(18000000, 1000).astype('float32')
y = np.random.rand(18000000).astype('float32')
model = LGBMRegressor(**params)
model.fit(X, y)

[jameslamb]

That's ok, completely understand that the code might be sensitive.

Are you able to share the values for params? Configuration you're using might give us some clues to help narrow this down.

[chixujohnny]

params = {
‘n_estimators’ : 500,
'learning_rate' : 0.05,
'subsample_freq' : 6 ,
'subsample' : 0.91,
'colsample_bytree' : 0.83,
'colsample_bynode' : 0.78,
'num_leaves': 64,
'max_depth' : 8,
'reg_alpha' : 9,
'reg_lambda' : 3.5,
'min_child_samples' : 200,
'min_child_weight' : 88,
'max_bin': 71,
'enable_sparse': false,
'device_type': "gpu",
'gpu_use_dp': false
}

[chixujohnny]

That's ok, completely understand that the code might be sensitive.

Are you able to share the values for params? Configuration you're using might give us some clues to help narrow this down.

Hello, can you find a GPU which memory more than 17G (like V100-32G or A100-40G), and generate a random dataset to reappearance my code?

[jameslamb]

I personally don't have easy access to hardware like that. I might try at some point to get a VM from a cloud provider and work on some of the open GPU-specific issues in this project, but can't commit to that. Maybe some other maintainer or contributor will be able to help you.

[lironle6]

That's ok, completely understand that the code might be sensitive.
Are you able to share the values for params? Configuration you're using might give us some clues to help narrow this down.

Hello, can you find a GPU which memory more than 17G (like V100-32G or A100-40G), and generate a random dataset to reappearance my code?

I have A100-40G and the above code fails.
interestingly, when I run it on 9M rows instead of 18M it doesn't fail. a quick calculation shows that 9M rows is just below the 40GB mark so might be related, might not.
In my own code I get this error unless I lower num_leaves and have to tweak it again if I change max_bin, if this helps.
This is also company property so won't be able to share the data or code.
Other than that, will help any way I can.