Remove infinite retry policy when announcing the leadership

Instead of retrying infinitely in the LeadershipState, we are now spawning a
background appender and hand the announce leadership message over. Additionally,
we are now monitoring the health of the appender task to escalate potential
failures which would then result in shutting the PP down and let the PPM handle
the situation.

This fixes restatedev#2333.

crates/bifrost/src/background_appender.rs

@@ -56,8 +56,8 @@ where
     }
 
     /// Start the background appender as a TaskCenter background task. Note that the task will not
-    /// autmatically react to TaskCenter's shutdown signal, it gives control over the shutdown
-    /// behaviour to the the owner of [`AppenderHandle`] to drain or drop when appropriate.
+    /// automatically react to TaskCenter's shutdown signal, it gives control over the shutdown
+    /// behaviour to the owner of [`AppenderHandle`] to drain or drop when appropriate.
     pub fn start(
         self,
         task_center: TaskCenter,
@@ -213,6 +213,11 @@ impl<T> AppenderHandle<T> {
     pub fn sender(&self) -> &LogSender<T> {
         self.sender.as_ref().unwrap()
     }
+
+    /// Polls the underlying appender task to check whether it has finished or not.
+    pub async fn poll_appender_task(&mut self) -> Result<()> {
+        self.inner_handle.as_mut().expect("must be present").await?
+    }
 }
 
 #[derive(Clone)]