add docs for new ManuallyPromote upgrade rollout strategy

Author: jubrad

doc/user/content/self-managed-deployments/upgrading/_index.md

@@ -234,9 +234,32 @@ The behavior of the new version rollout follows your `rolloutStrategy` setting:
 | `rolloutStrategy` | Description                        |
 | ----------------- | -----------------------------------|
 | `WaitUntilReady`  | *Default*. New instances are created and all dataflows are determined to be ready before cutover and terminating the old version, temporarily requiring twice the resources during the transition. |
-| `ImmediatelyPromoteCausingDowntime`| Tears down the prior version before creating and promoting the new version. This causes downtime equal to the duration it takes for dataflows to hydrate, but does not require additional resources. |
+| `ImmediatelyPromoteCausingDowntime` | Tears down the prior version before creating and promoting the new version. This causes downtime equal to the duration it takes for dataflows to hydrate, but does not require additional resources. |
+| `ManuallyPromote` | [BETA ] Creates a new generation of pods, leaving the old generation as the serving generation until the user manually promotes the new generation by updating the `forcePromote` field. |
 | `inPlaceRollout`| *Deprecated*.  The setting is ignored. |
 
+
+
+When using `ManuallyPromote`, the new generation can be promoted at any
+time, even if it has dataflows that are not fully caught up, by setting
+`forcePromote` to the same value as `requestRollout` in the Materialize spec.
+
+To minimize downtime, promotion should occur when the new generation
+has caught up to the prior generation. To determine if the new
+generation has caught up, consult the `UpToDate` condition in the
+status of the Materialize Resource. If the condition's reason is
+`ReadyToPromote` the new generation is ready to promote.
+
+{{<warning>}}
+Do not leave new generations unpromoted indefinitely.
+
+The new generation keeps open read holds which prevent compaction. Once promoted or
+cancelled, those read holds are released. If left unpromoted for an extended time, this
+data can build up, and can cause extreme deletion load on the metadata backend database
+when finally promoted or cancelled.
+{{</warning>}}
+
+
 ## Verifying the Upgrade
 
 After initiating the rollout, you can monitor the status field of the Materialize custom resource to check on the upgrade.

src/cloud-resources/src/crd/materialize.rs

@@ -42,9 +42,15 @@ pub mod v1alpha1 {
         /// Create a new generation of pods, leaving the old generation as the serving generation
         /// until the user manually promotes the new generation.
         ///
-        /// Users can promote the new generation at any time, even if the new generation pods are
-        /// not fully caught up, by setting `forcePromote` to the same value as `requestRollout` in
-        /// the Materialize spec.
+        /// When using `ManuallyPromote`, the new generation can be promoted at any
+        /// time, even if it has dataflows that are not fully caught up, by setting
+        /// `forcePromote` to the same value as `requestRollout` in the Materialize spec.
+        ///
+        /// To minimize downtime, promotion should occur when the new generation
+        /// has caught up to the prior generation. To determine if the new
+        /// generation has caught up, consult the `UpToDate` condition in the
+        /// status of the Materialize Resource. If the condition's reason is
+        /// `ReadyToPromote` the new generation is ready to promote.
         ///
         /// {{<warning>}}
         /// Do not leave new generations unpromoted indefinitely.