[Core][GPU fraction][1/n] Unify node feasibility and availability checking #59278

Yicheng-Lu-llll · 2025-12-09T00:10:19Z

Description

Problem Statement

Currently, the scheduler’s node feasibility and availability checks are inconsistent with the actual resource allocation logic. The scheduler reasons only about aggregated GPU capacity per node, while the allocator(local) enforces constraints based on the per-GPU topology.

For example, consider a node with two GPUs, each with 0.2 GPU remaining. The scheduler observes 0.4 GPU available in total and concludes that a actor requesting 0.4 GPU can be placed on this node. However, the allocator(local) rejects the request because no single GPU has 0.4 GPU available.

what this PR does

The high-level goal of this PR is to make node feasibility and availability checks consistent between the scheduler and the resource allocator.

Although the detailed design is still a work in progress and need big refactor, the first step is to make the scheduler’s node feasibility and availability checks itself consistent and centralized.

Right now, Ray has three scheduling paths:

Normal task scheduling
Normal actor scheduling
Placement group
- Placement Group reservation(scheduling bundle)
- Task/Actor with Placement Group

Tasks and actors essentially share the same scheduling path and use the same node feasibility and availability check function. Placement group scheduling, however, implements its own logic in certain path, even though it is conceptually the same.

Since we may override or extend the node feasibility and availability checks in later PRs, it is better to first ensure that all scheduling paths use a single, shared implementation of this logic.

This PR addresses that problem.

Related issues

Related to #52133 #54729

Additional information

Here I list all the cases that make sure we are relying on the same node feasibility and availability checking func. Later we can just focusing on changing the func and underlying data structure:

Normal task/actor scheduling：

HybridSchedulingPolicy：
- ray/src/ray/raylet/scheduling/policy/hybrid_scheduling_policy.cc
  
  Line 41 in 555fab3
  
  return node_resources.IsFeasible(resource_request);
- ray/src/ray/raylet/scheduling/policy/hybrid_scheduling_policy.cc
  
  Line 137 in 555fab3
  
  node_resources.IsAvailable(resource_request, ignore_pull_manager_at_capacity);
SpreadSchedulingPolicy:
- ray/src/ray/raylet/scheduling/policy/spread_scheduling_policy.cc
  
  Line 49 in 555fab3
  
  if (!is_node_alive_(node_id) || !node.GetLocalView().IsFeasible(resource_request)) {
- ray/src/ray/raylet/scheduling/policy/spread_scheduling_policy.cc
  
  Line 54 in 555fab3
  
  !node.GetLocalView().IsAvailable(resource_request,

RandomSchedulingPolicy

ray/src/ray/raylet/scheduling/policy/random_scheduling_policy.cc

Lines 47 to 48 in 456d190

    
           if (is_node_available_(node_id) && node.GetLocalView().IsFeasible(resource_request) && 
        
               node.GetLocalView().IsAvailable(resource_request,

NodeAffinitySchedulingPolicy

Don't care, just schedule to user specified node by default

Having fallback option that checks:

ray/src/ray/raylet/scheduling/policy/node_affinity_scheduling_policy.cc

Lines 26 to 30 in 1180868

    
             nodes_.at(target_node_id).GetLocalView().IsFeasible(resource_request)) { 
        
           if (!options.node_affinity_spill_on_unavailable_ && 
        
               !options.node_affinity_fail_on_unavailable_) { 
        
             return target_node_id; 
        
           } else if (nodes_.at(target_node_id).GetLocalView().IsAvailable(resource_request)) {

NodeLabelSchedulingPolicy
- ray/src/ray/raylet/scheduling/policy/node_label_scheduling_policy.cc
  
  Line 171 in 1180868
  
  if (is_node_alive_(node_id) && node_resources.IsFeasible(resource_request)) {
- ray/src/ray/raylet/scheduling/policy/node_label_scheduling_policy.cc
  
  Line 186 in 1180868
  
  if (node_resources.IsAvailable(resource_request)) {

Placement Group reservation(scheduling bundle)：

PACK/SPREAD/STRICT_SPREAD
- ray/src/ray/raylet/scheduling/policy/scorer.cc
  
  Line 58 in 1180868
  
  if (requested > available) {
  - Note, after this PR, it will also be IsAvailable
STRICT_SPREAD
- ray/src/ray/raylet/scheduling/policy/scorer.cc
  
  Line 58 in 1180868
  
  if (requested > available) {
  - Note, after this PR, it will also be IsAvailable
- ray/src/ray/raylet/scheduling/policy/bundle_scheduling_policy.cc
  
  Line 185 in 1180868
  
  if (node_resources.IsAvailable(*iter->second)) {

Task/Actor with Placement Group：

AffinityWithBundleSchedulingPolicy

ray/src/ray/raylet/scheduling/policy/affinity_with_bundle_scheduling_policy.cc

Lines 25 to 26 in 1180868

    
           nodes_.at(node_id).GetLocalView().IsFeasible(resource_request) && 
        
           nodes_.at(node_id).GetLocalView().IsAvailable(resource_request))) {

Signed-off-by: yicheng <[email protected]>

Yicheng-Lu-llll · 2025-12-09T00:11:10Z

src/ray/raylet/scheduling/policy/scorer.cc

-  // Check if the node has required labels before scoring on the resources.
-  const auto &label_selector = required_resources.GetLabelSelector();
-  if (!node_resources.HasRequiredLabels(label_selector)) {
+  if (!node_resources.IsAvailable(required_resources)) {


IsAvailable already checks the required labels.

gemini-code-assist

Code Review

This pull request aims to unify node feasibility and availability checking in the scheduler. The changes in scorer.cc centralize the availability check by using node_resources.IsAvailable() and removing redundant checks within the LeastResourceScorer. This is a good refactoring that simplifies the code. My feedback includes a suggestion to add an assertion to enforce a critical precondition, which will make the code more robust.

gemini-code-assist · 2025-12-09T00:11:54Z

src/ray/raylet/scheduling/policy/scorer.cc

 double LeastResourceScorer::Calculate(const FixedPoint &requested,
                                      const FixedPoint &available) {
  RAY_CHECK(available >= 0) << "Available resource " << available.Double()
                            << " should be nonnegative.";


The removal of the requested > available check relies on node_resources.IsAvailable() being correct. To make the code more robust against potential bugs in the availability check, it's safer to add a RAY_CHECK(requested <= available). This enforces the assumption that Calculate is only called for feasible requests and will help catch issues early.

Suggested change

<< " should be nonnegative.";

<< " should be nonnegative.";

RAY_CHECK(requested <= available)

<< "Requested resource " << requested.Double()

<< " should not be greater than available resource " << available.Double()

<< ". This indicates a bug in the feasibility check.";

No need. From a design perspective, the logic for determining whether a node is schedulable and the logic for choosing the best node should be completely separate and follow a strict order.

edoakes · 2025-12-09T16:00:48Z

@ZacAttack @Sparks0219 PTAL too

MengjinYan · 2025-12-09T09:39:09Z

src/ray/raylet/scheduling/policy/scorer.cc

  return node_score;
 }

+// This function assumes the resource request has already passed the availability check


nit: Can you add the this also to the comment of the function in the .h file?

Added, Thank you!

MengjinYan · 2025-12-09T19:38:40Z

src/ray/raylet/scheduling/policy/scorer.cc

-      return -1.;
-    }
-    node_score += score;
+    node_score += Calculate(request_resource, node_available_resource);


From my understanding of the code:

Previously: we first checks the labels, then potentially update the available resources, the checks whether there are available resources in the Calculate function

Now: we first checks both the labels and the the available resources, then potentially update the available resources, the calculate the score assuming the node has enough resources for the request

I might miss something but looks like with the new implementation, there could be case where the node will have available resources before the resource update but not after which will make the new implementation not consistent with the previous implementation, is that right?

Sorry about that — I should have mentioned this!

IsAvailable already handles the normal_task_resources subtraction internally:

ray/src/ray/common/scheduling/cluster_resource_data.cc

Lines 98 to 103 in de7ac7d

if (!this->normal_task_resources.IsEmpty()) {

auto available_resources = this->available;

available_resources -= this->normal_task_resources;

return available_resources >= resource_request.GetResourceSet();

}

return this->available >= resource_request.GetResourceSet();

Got it. I missed that. Thanks for the context!

The current implementation is a bit confusion, as part of the project of improving the resource tracking effort, we should make it cleaner.

Sure, I will look deep into it and draft an follow up pr if needed！

Signed-off-by: yicheng <[email protected]>

MengjinYan

Thanks!

Unify node feasibility and availability checking

7900ecd

Signed-off-by: yicheng <[email protected]>

Yicheng-Lu-llll requested a review from a team as a code owner December 9, 2025 00:10

Yicheng-Lu-llll commented Dec 9, 2025

View reviewed changes

gemini-code-assist bot reviewed Dec 9, 2025

View reviewed changes

MengjinYan self-assigned this Dec 9, 2025

ray-gardener bot added the core Issues that should be addressed in Ray Core label Dec 9, 2025

Yicheng-Lu-llll added the go add ONLY when ready to merge, run all tests label Dec 9, 2025

MengjinYan reviewed Dec 9, 2025

View reviewed changes

add comment

f71ddad

Signed-off-by: yicheng <[email protected]>

MengjinYan approved these changes Dec 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Core][GPU fraction][1/n] Unify node feasibility and availability checking #59278

[Core][GPU fraction][1/n] Unify node feasibility and availability checking #59278

Yicheng-Lu-llll commented Dec 9, 2025 •

edited

Loading

Uh oh!

Yicheng-Lu-llll Dec 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Dec 9, 2025

Uh oh!

Yicheng-Lu-llll Dec 9, 2025

Uh oh!

edoakes commented Dec 9, 2025 •

edited

Loading

Uh oh!

MengjinYan Dec 9, 2025

Uh oh!

Yicheng-Lu-llll Dec 10, 2025

Uh oh!

MengjinYan Dec 9, 2025

Uh oh!

Yicheng-Lu-llll Dec 10, 2025 •

edited

Loading

Uh oh!

MengjinYan Dec 10, 2025

Uh oh!

Yicheng-Lu-llll Dec 10, 2025

Uh oh!

MengjinYan left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	if (is_node_available_(node_id) && node.GetLocalView().IsFeasible(resource_request) &&
	node.GetLocalView().IsAvailable(resource_request,

	nodes_.at(target_node_id).GetLocalView().IsFeasible(resource_request)) {
	if (!options.node_affinity_spill_on_unavailable_ &&
	!options.node_affinity_fail_on_unavailable_) {
	return target_node_id;
	} else if (nodes_.at(target_node_id).GetLocalView().IsAvailable(resource_request)) {

	nodes_.at(node_id).GetLocalView().IsFeasible(resource_request) &&
	nodes_.at(node_id).GetLocalView().IsAvailable(resource_request))) {

-                            << " should be nonnegative.";
+                            << " should be nonnegative.";
+  RAY_CHECK(requested <= available)
+      << "Requested resource " << requested.Double()
+      << " should not be greater than available resource " << available.Double()
+      << ". This indicates a bug in the feasibility check.";

	if (!this->normal_task_resources.IsEmpty()) {
	auto available_resources = this->available;
	available_resources -= this->normal_task_resources;
	return available_resources >= resource_request.GetResourceSet();
	}
	return this->available >= resource_request.GetResourceSet();

[Core][GPU fraction][1/n] Unify node feasibility and availability checking #59278

Are you sure you want to change the base?

[Core][GPU fraction][1/n] Unify node feasibility and availability checking #59278

Conversation

Yicheng-Lu-llll commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Problem Statement

what this PR does

Related issues

Additional information

Uh oh!

Yicheng-Lu-llll Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Yicheng-Lu-llll Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

edoakes commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MengjinYan Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Yicheng-Lu-llll Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

MengjinYan Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Yicheng-Lu-llll Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MengjinYan Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Yicheng-Lu-llll Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

MengjinYan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Yicheng-Lu-llll commented Dec 9, 2025 •

edited

Loading

edoakes commented Dec 9, 2025 •

edited

Loading

Yicheng-Lu-llll Dec 10, 2025 •

edited

Loading