feat: cluster-level resource scheduling suspend and resume capabilities #5937

Monokaix · 2024-12-11T10:20:41Z

What type of PR is this?
/kind api-change
/kind feature

What this PR does / why we need it:
Add rb suspension capability, background: #5690

Which issue(s) this PR fixes:
Fixes #5690

Special notes for your reviewer:
It extends the Suspension fileds and support rb susupend and resume, mostly like the previous issue #5217 and PRs implemented in it.

Does this PR introduce a user-facing change?:

Introduce cluster-level resource scheduling pause and resume capabilities

karmada-bot · 2024-12-11T10:20:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rainbowmango for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS
pkg/apis/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot · 2024-12-11T10:20:52Z

Welcome @Monokaix! It looks like this is your first PR to karmada-io/karmada 🎉

Monokaix · 2024-12-12T06:57:48Z

cc @RainbowMango @XiShanYongYe-Chang @chaosi-zju

codecov-commenter · 2024-12-12T07:13:56Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 62.13592% with 39 lines in your changes missing coverage. Please review.

Project coverage is 48.27%. Comparing base (471d850) to head (553e498).
Report is 14 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/util/helper/patch.go	0.00%	18 Missing ⚠️
pkg/controllers/binding/common.go	77.19%	10 Missing and 3 partials ⚠️
pkg/controllers/binding/binding_controller.go	0.00%	3 Missing and 1 partial ⚠️
...ers/binding/cluster_resource_binding_controller.go	0.00%	3 Missing and 1 partial ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5937      +/-   ##
==========================================
+ Coverage   48.15%   48.27%   +0.11%     
==========================================
  Files         664      664              
  Lines       54803    54848      +45     
==========================================
+ Hits        26393    26477      +84     
+ Misses      26693    26652      -41     
- Partials     1717     1719       +2

Flag	Coverage Δ
unittests	`48.27% <62.13%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pkg/scheduler/scheduler_test.go

RainbowMango

Don't we need to introduce a field to ResourceBinding?

RainbowMango · 2024-12-18T07:25:27Z

pkg/apis/policy/v1alpha1/propagation_types.go

+
+	// Scheduling controls whether scheduling should be suspended.
+	// nil means not suspend, no default value, only accepts 'true'.
+	// Karmada scheduler will pause scheduling when value is true and resume scheduling when it's nil.
+	Scheduling *bool `json:"scheduling,omitempty"`


Can you explain a little bit why PropagationPolicy needs this?

The Suspension is common struct which indicates a suspension semantics, so based on the existing Suspension, I added scheduling suspension other than dispatcher. If you think it is inappropriate, I can add a new suspension structure just for scheduling, but this seems a bit redundant.

Sorry, I didn't make it clear.
In my understanding, what you need is introducing scheduling suspension to ResourceBinding, my question is do you need to do that to PropgationPolicy as well?

PS:
I know both ResourceBinding and PropgationPolicy share the same typed structure (type Suspension struct). They may no longer be able to share the existing structure if we don't want to introduce scheduling suspension to PropgationPolicy.

Yeah you are right, currenly I don't need the filed in PropgationPolicy, unless we just want to debug this feature easily.

`Karmada scheduler``

Do we not need to limit it to karmada-scheduler? Whether all schedulers should care about this field.

@XiShanYongYe-Chang Yeah you are right, has modified: )

RainbowMango · 2024-12-18T07:27:52Z

pkg/apis/work/v1alpha2/binding_types.go

+	// Suspended represents the condition that the ResourceBinding or ClusterResourceBinding is suspended to schedule.
+	Suspended string = "Suspended"


This is ambiguous as people can not tell what's suspended.

As far as I know, the Rb resource is an abstract of scheduling, so maybe it's enough? And other conditions are Scheduled string = "Scheduled", which are also a semantic related to scheduling, or could you give some advice?

Directly use ScheduleSuspended?

pkg/controllers/binding/common.go

pkg/events/events.go

pkg/scheduler/scheduler.go

kevin-wangzefeng · 2024-12-23T09:18:18Z

pkg/apis/policy/v1alpha1/propagation_types.go

+
+	// Scheduling controls whether scheduling should be suspended.
+	// nil means not suspend, no default value, only accepts 'true'.
+	// scheduler will pause scheduling when value is true and resume scheduling when it's nil.
+	Scheduling *bool `json:"scheduling,omitempty"`


It's not clear why we need flag field here, also the field "scheduling" looks confusing what does the admin really want to stop system from doing.
Whether the resourcebinding shall be created?

The resourcebinding will be created but scheduler will not scheduler, because we need some admission check such as quota check, multi-tenancy prioritization, etc before schduler process resourcebinding.
Will add more comments: )

pkg/apis/work/v1alpha2/binding_types.go

Signed-off-by: Monokaix <[email protected]>

karmada-bot added the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label Dec 11, 2024

karmada-bot requested review from chaosi-zju, chaunceyjiang, Garrybest and pigletfly December 11, 2024 10:20

karmada-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Dec 11, 2024

karmada-bot requested review from Vacant2333, whitewindmills and XiShanYongYe-Chang December 11, 2024 10:20

karmada-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 11, 2024

Monokaix force-pushed the rb-suspend branch from 2362a7b to 9631963 Compare December 11, 2024 10:26

karmada-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 11, 2024

Monokaix force-pushed the rb-suspend branch from 9631963 to c553802 Compare December 11, 2024 10:26

karmada-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 11, 2024

Monokaix force-pushed the rb-suspend branch 2 times, most recently from c4d5f12 to 08407d6 Compare December 12, 2024 06:56

karmada-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 12, 2024

Monokaix force-pushed the rb-suspend branch from 08407d6 to 6dc8c58 Compare December 12, 2024 06:59

karmada-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 12, 2024

XiShanYongYe-Chang reviewed Dec 12, 2024

View reviewed changes

pkg/scheduler/scheduler_test.go Outdated Show resolved Hide resolved

Monokaix force-pushed the rb-suspend branch from 6dc8c58 to 03a2ca6 Compare December 12, 2024 08:10

karmada-bot removed the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 12, 2024

karmada-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 12, 2024

Monokaix force-pushed the rb-suspend branch from 03a2ca6 to 06f2f66 Compare December 12, 2024 08:55

karmada-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 12, 2024

RainbowMango reviewed Dec 18, 2024

View reviewed changes

Monokaix force-pushed the rb-suspend branch 6 times, most recently from 6e5f6ff to 060994b Compare December 23, 2024 08:08

kevin-wangzefeng reviewed Dec 23, 2024

View reviewed changes

Monokaix added 3 commits December 23, 2024 17:41

RB suspension: API change

dc174c2

Signed-off-by: Monokaix <[email protected]>

RB suspension: sheduler and controller change

6ef531f

Signed-off-by: Monokaix <[email protected]>

RB suspension: e2e test

553e498

Signed-off-by: Monokaix <[email protected]>

Monokaix force-pushed the rb-suspend branch from 060994b to 553e498 Compare December 23, 2024 09:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: cluster-level resource scheduling suspend and resume capabilities #5937

feat: cluster-level resource scheduling suspend and resume capabilities #5937

Monokaix commented Dec 11, 2024

karmada-bot commented Dec 11, 2024

karmada-bot commented Dec 11, 2024

Monokaix commented Dec 12, 2024

codecov-commenter commented Dec 12, 2024 •

edited

Loading

RainbowMango left a comment

RainbowMango Dec 18, 2024

Monokaix Dec 19, 2024

RainbowMango Dec 19, 2024

Monokaix Dec 19, 2024

XiShanYongYe-Chang Dec 20, 2024

Monokaix Dec 23, 2024 •

edited

Loading

RainbowMango Dec 18, 2024

Monokaix Dec 19, 2024

XiShanYongYe-Chang Dec 23, 2024

kevin-wangzefeng Dec 23, 2024

Monokaix Dec 23, 2024

		// Suspended represents the condition that the ResourceBinding or ClusterResourceBinding is suspended to schedule.
		Suspended string = "Suspended"

feat: cluster-level resource scheduling suspend and resume capabilities #5937

Are you sure you want to change the base?

feat: cluster-level resource scheduling suspend and resume capabilities #5937

Conversation

Monokaix commented Dec 11, 2024

karmada-bot commented Dec 11, 2024

karmada-bot commented Dec 11, 2024

Monokaix commented Dec 12, 2024

codecov-commenter commented Dec 12, 2024 • edited Loading

Codecov Report

RainbowMango left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Monokaix Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Dec 12, 2024 •

edited

Loading

Monokaix Dec 23, 2024 •

edited

Loading