Releases: karmada-io/karmada
karmada v1.2.3 release
Changes since v1.2.2
Bug Fixes
karmada-controller-manager
/karmada-agent
: Fixed a resource status can not be collected issue in case of Resource Interpreter returns an error. (#2429, @zhuwint)karmada-sechduler
: Fixed a panic issue whenreplicaDivisionPreference
isWeighted
andWeightPreference
is nil. (#2453, @XiShanYongYe-Chang)karmadactl
: Fixedinit
can not honor IPV6 address issue when generating kubeconfig file. (#2529, @duanmengkk)karmada-controller-manager
: use cluster secret ref namespace in unified-auth-controller when generate ClusterRoleBinding. (#2531, @XiShanYongYe-Chang)karmadactl
: Fixed options ofdeinit
can not be shown issue. (#2566, @helen-frank)
karmada v1.1.5 release
Changes since v1.1.4
Bug Fixes
Helm Chart
: Fixedkarmada-apiserver
always pending in case of upgrade issue. (#2436, @Poor12)karmadactl
: Fixedinit
can not honor IPV6 address issue when generating kubeconfig file. (#2535, @duanmengkk)karmada-controller-manager
: use cluster secret ref namespace in unified-auth-controller when generate ClusterRoleBinding.(#2538, @XiShanYongYe-Chang )
karmada v1.3.0 release
What's New
Taint-based eviction in graceful way
We introduced a new controller named taint manager
which aims to evict workloads from faulty clusters after a grace period.
Then the scheduler would select new best-fit clusters for the workloads. In addition, if the feature GracefulEviction
is enabled,
the eviction will be very smooth, that is, the removal of evicted workloads will be delayed until the workloads are available on
new clusters or reach the maximum grace period. For more details please refer to Failover Overview.
(Feature contributor: @Garrybest, @XiShanYongYe-Chang)
Global proxy for resources across multi-clusters
We introduced a new proxy feature to karmada-search
that allows users to access resources in multiple clusters in a way just like accessing resources in a single cluster. No matter whether the resources are managed by Karmada, by leveraging the proxy, users
can manipulate the resources from the Karmada control plane.
For more details please refer to Global Resource Proxy.
(Feature contributor: @ikaven1024, @XiShanYongYe-Chang)
Cluster resource modeling
To provide a more accurate scheduling basis for the scheduler, we introduced a way to model the cluster's available resources.
The cluster status controller will model the resources as per the customized resource models, which is more accurate than the general resource summary. For more details please refer to Cluster Resource Modeling.
(Feature contributor: @halfrost, @Poor12)
Bootstrap token-based cluster registration
Now for clusters in Pull
mode, we provide a way for them to register with the Karmada control plane. By leveraging the commands token
and register
in kubectl
, the registration process including deploying the karmada-agent
can be completed very easily. For more details please refer to Register cluster with Pull mode.
(Feature contributor: @lonelyCZ )
Significant improvement in system scalability
We improved the system scalability, such as:
- Enable pprof(#2008)
- Introduce
cachedRESTMapper
(#2187) - Adopt the transform function to reduce memory usage(#2383)
With these improvements, Karmada can easily manage hundreds of huge clusters. The detailed test report will be released soon.
Other Notable Changes
API changes
- The
Cluster
API is added optional fieldID
to uniquely identify the cluster. (@RainbowMango, #2180) - The
Cluster
API is added optional fieldProxyHeader
to specify the HTTP header required by the proxy server. (@mrlihanbo, #1874) - The
Cluster
API is added optional field namedResourceModels
to specify resource modeling. (@halfrost, #2386) - The
Work
andResourceBinding
/ClusterResourceBinding
APIs are added fieldhealth
to represent the state of workload. (@XiShanYongYe-Chang, #2351)
Bug Fixes
karmadactl
: Fixed issue that Kubernetes v1.24 cannot be joined. (@zgfh, #1972)karmadactl
: Fixed a panic issue when retrieving resources from an unknown cluster(karmadactl get xxx --cluster=not-exist
). (@my-git9, #2171)karmadactl
: Fixed failed promoting if a resource with another kind using the same name has been promoted before. (@wuyingjun-lucky, #1824)karmada-search
: Fixed panic when the resource annotation is nil. (@XiShanYongYe-Chang, #1921)karmada-search
: Fixed paniccomparing uncomparable type cache.ResourceEventHandlerFuncs
. (@liys87x, #1951)karmada-search
: Fixed failed query on a single namespace (@luoMonkeyKing, #2227)karmada-controller-manager
: Fixed thatJob
status might be incorrectly marked asCompleted
. (@Garrybest, #1987)karmada-controller-manager
: Fixed returning err when the interpreter webhook returns nil patch and nil patchType. (@CharlesQQ, #2161)karmada-controller-manager
: Fixed that Argo CD cannot assess Deployment health status. (@xuqianjins, #2241)karmada-controller-manager
: Fixed that Argo CD cannot assess StatefulSet/DaemonSet health status. (@RainbowMango, #2252)karmada-controller-manager
/karmada-agent
: Fixed an resource status can not be collected issue in case of Resource Interpreter returns an error. (@XiShanYongYe-Chang, #2428)karmada-sechduler
: Fixed a panic issue whenreplicaDivisionPreference
isWeighted
andWeightPreference
is nil. (@XiShanYongYe-Chang, #2451)
Features & Enhancements
karmadactl
: Added--force
flag todeinit
to skip confirmation. (@zgfh, #2016)karmadactl
: The flag-c
of sub-commandpromote
now has been changed to uppercase-C
. (@Fish-pro, #2140)- ``karmadactl
: Introduced
--cluster-zone` and `--cluster-region` flags to `join` command to specify the zone and region of joining cluster. (@chaunceyjiang, #2048) karmadactl
: Introduced--namespace
flag toexec
command to specify the workload namespace. (@carlory, #2092)karmadactl
: Allowed reading namespaces from the context field of karmada config forget
command. (@carlory, #2148)karmadactl
: Introducedapply
subcommand to apply a configuration to a resource by file name or stdin. (@carlory, #2000)karmadactl
: Introduced--namespace
flag todescribe
command to specify the namespace the workload belongs to. (@TheStylite, #2153)karmadactl
: Introduced--cluster
flag forapply
command to allow users to select one or many member clusters to propagate resources. (@carlory, #2192)karmadactl
: Introduced options subcmd to list global command-line options. (@lonelyCZ, #2283)karmadactl
: Introduced thetoken
command to manage bootstrap tokens. (@lonelyCZ, #2399)karmadactl
: Introduced theregister
command for joining PULL mode cluster. (@lonelyCZ, #2388)karmada-scheduler
: Introduced--enable-empty-workload-propagation
flag to enable propagating empty workloads. (@CharlesQQ, #1720)karmada-scheduler
: Allowed extended plugins in an out-of-tree mode. (@kerthcet, #1663)karmada-scheduler
: Introduced--disable-scheduler-estimator-in-pull-mode
flag to disable scheduler-estimator for clusters in pull mode. (@prodanlabs, #2064)karmada-scheduler
: Introduced--plugins
flag to enable or disable scheduler plugins. (@chaunceyjiang, #2135)karmada-scheduler
: Now the scheduler starts to re-schedule in case of cluster state changes. (@chaunceyjiang, #2301)karmada-search
: The search API supports searching for resources according to labels. (@XiShanYongYe-Chang, #1917)karmada-search
: The annotationcluster.karmada.io/name
which is used to represent the source of cache now has been changed toresource.karmada.io/cached-from-cluster
. (@calvin0327, #1960)karmada-search
: Fixed panic issue when dumping error info. (@AllenZMC, #2231)karmada-controller-manager/karmada-agent
: Cluster state controller now able to collect partial API list in the case of discovery failure. (@duanmengkk, #1968)karmada-controller-manager/karmada-agent
: Introduced--cluster-success-threshold
flag to specify cluster success threshold. Default to 30s. (@dddddai, #1884)karmada-controller-manager/karmada-agent
: Added CronJob support to the default resource interpreter framework. (@chaunceyjiang, #2060)karmada-controller-manager
/karmada-agent
: Introduced--leader-elect-lease-duration
,--leader-elect-renew-deadline
and--leader-elect-retry-period
flags to specify leader election behaviors. (@CharlesQQ, #2056)karmada-controller-manager
/karmada-agent
: Fixed panic issue when dumping error inf...
karmada v1.2.2 release
Changes since v1.2.1
Bug Fixes
karmadactl
: Fixed a panic issue when retrieving resources from an unknown cluster(karmadactl get xxx --cluster=not-exist
). (#2171, #2201, @my-git9)karmadactl
: Fix Kubernetes v1.24 can not be joined issue. (#1972, @zgfh)karmada-controller-manager
: Fixed Argo CD can not assess Deployment health status issue. (#2256, @xuqianjins)karmada-controller-manager
: Fixed Argo CD can not assess StatefulSet/DaemonSet health status issue. (#2264, @RainbowMango)karmada-search
: Fixed can not query a single namespace issue. (#2274, @luoMonkeyKing)karmada-search
: Fixed panic issue when dumps error info. (#2333, @AllenZMC)Helm Chart
: Fixed misconfiguredMutatingWebhookConfiguration
and added missingAPIService
configuration forkarmada-aggregated-apiserver
. (#2420, @zhixian82)
karmada v1.1.4 release
Changes since v1.1.3
Bug Fixes
karmadactl
: Fix Kubernetes v1.24 can not be joined issue. (#2199, @zgfh)Helm Chart
: Fixed misconfiguredMutatingWebhookConfiguration
and added missingAPIService
configuration forkarmada-aggregated-apiserver
. (#2421, @zhixian82)
karmada v1.0.5 release
Changes since v1.0.4
Bug Fixes
karmadactl
: Fixed Kubernetes v1.24 can not be joined issue. (#2200, @zgfh)Helm Chart
: Fixed misconfiguredMutatingWebhookConfiguration
and added missingAPIService
configuration forkarmada-aggregated-apiserver
. (#2422, @zhixian82)
karmada v1.2.1 release
Changes since v1.2.0
Bug Fixes
karmada-search
: Fixed panic when the resource annotation is nil. (#1939, @XiShanYongYe-Chang )karmada-search
: Fixed a panic issuecomparing uncomparable type cache.ResourceEventHandlerFuncs
. (#1971, @liys87x )- karmadactl: fixed promoting failed if a resource with another kind using the same name has been promoted before. (#1983, @wuyingjun-lucky )
karmadactl
: Removed--dry-run
flag fromdescribe
,exec
andlog
commands. (#2036, @wlp1153468871 )karmada-controller-manager
: FixedJob
status might be incorrectly marked asCompleted
issue. (#2007, @Garrybest )karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2127, @AllenZMC )
karmada v1.0.4 release
Changes since v1.0.3
Bug Fixes
karmadactl
: Fixed namespace can not be customized issue. (#1826, @likakuli)karmadactl
: fixed can not taint while karmada control plane config is not located on default path. (#1837, @wuyingjun-lucky)karmada-controller-manager
: FixedJob
status might be incorrectly marked asCompleted
issue. (#2011, @Garrybest)karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2133, @AllenZMC)
karmada v1.1.3 release
Changes since v1.1.2
Bug Fixes
karmadactl
: Fixed namespace can not be customized issue. (#1827, @likakuli )karmadactl
: fixed karmadactl can not taint while karmada control plane config is not located on default path. (#1838, @wuyingjun-lucky )karmada-controller-manager
: FixedJob
status might be incorrectly marked asCompleted
issue. (#2010, @Garrybest )karmada-controller-manager
/karmada-agent
: fixed panic issue when dumps error info. (#2126, @AllenZMC )
karmada v1.2.0 release
What's New
Significant improvement on scheduling capability and scalability
1. Karmada Descheduler
A new component karmada-descheduler
was introduced, for rebalancing the scheduling decisions over time.
One example use case is: it helps evict pending replicas (Pods) from resource-starved clusters so that karmada-scheduler
can "reschedule" these replicas (Pods) to a cluster with sufficient resources.
For more details please refer to Descheduler user guide.
(Feature contributor: @Garrybest)
2. Multi region HA support
By leveraging the newly added spread-by-region
constraint, users are now able to deploy workloads aross regions
, e.g. people may want their workloads always running on different regions for HA purposes.
We also introduced two plugins to karmada-scheduler
, which add to accurate scheduling.
ClusterLocality
is a scoring plugin that favors clusters already assigned.SpreadConstraint
is a filter plugin that filters clusters as per spread constraints.
(Feature contributors: @huone1, @gf457832386)
We are also in the progress of enhancing the multi-cluster failover mechanism. Part of the work has been included in this release.
For example:
- A new flag(
--cluster-failure-threshold
) has been added to bothkarmada-controller-manager
andkarmada-agent
, which specifies the cluster failure threshold (defaults to 30s). A cluster will be considerednot-ready
only when it stays unhealthy longer than supposed. - A new flag(
--failover-eviction-timeout
) has been added tokarmada-controller-manager
, which specifies the grace period of eviction (defaults to 5 minutes). If a cluster staysnot-ready
longer than supposed, the controller taints the cluster. (Note: The taint is essentially the eviction order and the implementation is planned for the next release.)
(Feature contributors: @Garrybest, @dddddai)
Fully adopted aggregated API
The Aggregated API
was initially introduced in Release 1.0, which allows users to access clusters through Karmada by a single aggregated API endpoint. By leveraging this feature, we introduced a lot of interesting features to karmadactl
and kubectl-karmada
.
1. The get
sub-command now supports clusters both in push
and pull
mode.
# karmadactl get deployment -n default
NAME CLUSTER READY UP-TO-DATE AVAILABLE AGE ADOPTION
nginx member1 2/2 2 2 33h N
nginx member2 1/1 1 1 4m38s Y
podinfo member3 2/2 2 2 27h N
2. The newly added logs
command prints the container logs in a specific cluster.
# ./karmadactl logs nginx-6799fc88d8-9mpxn -c nginx -C member1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
...
3. We also added watch
and exec
commands to karmadactl
, in addition to get
and logs
. They all use the aggregated API.
(Feature contributor: @lonelyCZ)
Distributed search and analytics engine for Kubernetes resources (alpha
)
The newly introduced karmada-search
caches resources in clusters and allows users to search for resources without directly touching real clusters.
# kubectl get --raw /apis/search.karmada.io/v1alpha1/search/cache/apis/apps/v1/deployments
{
"apiVersion": "v1",
"kind": "List",
"metadata": {},
"items": [{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"cluster.karmada.io/name": "member1",
},
}
},
]
}
The karmada-search
also supports syncing cached resources to backend stores like Elasticsearch or OpenSearch. By leveraging the search engine, you can perform full-text searches with all desired features, by field, and by indice; rank results by score, sort results by field, and aggregate results.
(Feature contributors: @huntsman-li, @liys87x)
Resource Interpreter Webhook enhancement
Introduced InterpretStatus
for the Resource Interpreter Webhook
framework, which enables customized resource status collection.
Karmada can thereby learn how to collect status for your resources, especially custom resources. For example, a custom resource may have many status fields and only Karmada can collect only those you want.
Refer to [Customizing Resource Interpreter][https://github.com/karmada-io/karmada/blob/master/docs/userguide/customizing-resource-interpreter.md] for more details.
(Feature contributor: @XiShanYongYe-Chang)
Integrating verification with the ecosystem
Benefiting from the Kubernetes native APIs, Karmada can easily integrate the Kubernetes ecosystem. The following components are verified by the Karmada community:
Kyverno
: policy engine. Refer to working with kyverno for more details.Gatekeeper
: another policy engine. Refer to working with gatekeeper for more details.fluxcd
: GitOps tooling for helm chart. Refer to working with fluxcd for more details.
(Feature contributors: @Poor12, @learner0810)
Other Notable Changes
Bug Fixes
karmadactl
: Fixed the cluster joining failures in the case of legacy secrets. (@zgfh, #1306)karmadactl
: Fixed the issue that you cannot use the '-v 6' log level. (@zgfh, #1426)karmadactl
: Fixed the issue that the--namespace
flag ofinit
command did not work. (@sayaoailun, #1416)karmadactl
: Allowed namespaces to be customized. (@sayaoailun, #1449)karmadactl
: Fixed theinit
failure due to data path not clean. (@prodanlabs, #1455)karmadactl
: Fixed theinit
failure to read the KUBECONFIG environment variable. (@lonelyCZ, #1437)karmadactl
: Fixed theinit
command failure to select the default release version. (@prodanlabs, #1456)karmadactl
: Fixed the issue that thekarmada-system
namespace already exists when deployingkarmada-agent
. (@hanweisen, #1604)karmadactl
: Fixed the issue that the karmada-controller-manager args did not honor customized namespaces.` (@prodanlabs, #1683)karmadactl
: Fixed a panic due to nil annotation whenpromoting
resources to Karmada.` (@duanmengkk, #1759)karmadactl
: Fixed thepromote
command failure to migrate cluster-scoped resources. (@duanmengkk, #1766)karmadactl
: fixed the karmadactl taint failure while the karmada control plane config is not located in the default path. (@wuyingjun-lucky, #1825)helm-chart
: Fixed the karmada-agent installation failure due to the lack of permission. (@AllenZMC, #1457)helm-chart
: Fixed the issue that version constraints skip pre-releases. (@pigletfly, #1444)karmada-controller-manager
: Fixed the issue that ResourceBinding may hinder en-queue in the case of schedule failures. (@mrlihanbo, #1499)karmada-controller-manager
: Fixed the panic when the interpreter webhook returns nil patch. (@CharlesQQ, #1584)karmada-controller-manager
: Fixed the RB/CRB controller failure to aggregate status in the case of work condition update. (@mrlihanbo, #1513)karmada-aggregate-apiserver
: Fixed timeout issue when requestingcluster/proxy
with options-w
orlogs -f
fromkarmadactl get
. (@XiShanYongYe-Chang, #1620)karmada-aggregate-apiserver
: Fixed exec failed:error: unable to upgrade connection: you must specify at least 1 of stdin, stdout, stderr
. (@pangsq, #1632)
Features & Enhancements
karmada-controller-manager
: Introduced several flags to specify controller's concurrent capacities(--rate-limiter-base-delay
,--rate-limiter-max-delay
,--rate-limiter-qps
,--rate-limiter-bucket-size
). (@pigletfly, #1399)karmada-controller-manager
: The klog flags now have been grouped for better readability. (@RainbowMango, #1468)karmada-controller-manager
: Fixed theFullyApplied
condition ofResourceBinding/ClusterResourceBinding
mislabeling issue in the case of non-scheduling. (@huone1, #1512)karmada-controller-manager
: Added defaultAggregateStatus
webhook forDaemonSet
andStatefulSet
. (@Poor12, #1586)- `karmada-controll...