From d5a497b09f8b9499da42a8986449a132577eb206 Mon Sep 17 00:00:00 2001
From: Michael Koepf <47541996+michaelkoepf@users.noreply.github.com>
Date: Fri, 3 Jan 2025 19:37:21 +0100
Subject: [PATCH] [observability] Add Grafana and Loki as log monitoring stack

- Group logging, metrics, and monitoring into common sidebar menu group ("observability")
- Move monitoring quickstart guide (Prometheus + Grafana) to separate page ("observability quickstart guides")
- Extend monitoring quickstart guide (log monitoring with Loki) + provide new quickstart configuration ZIP archive
- Minor changes and fixes in Flink quickstart guide
- Add version hint for logback in doc

Issue #295, #304
---
 .../assets/fluss-quickstart-observability.zip | Bin 14592 -> 15864 bytes
 website/docs/install-deploy/overview.md       |   4 +-
 .../maintenance/observability/_category_.json |   4 +
 .../{ => observability}/logging.md            |   8 +-
 .../{ => observability}/metric-reporters.md   |   2 +-
 .../{ => observability}/monitor-metrics.md    | 175 +-------------
 .../maintenance/observability/quickstart.md   | 220 ++++++++++++++++++
 website/docs/quickstart/flink.md              |   9 +-
 8 files changed, 239 insertions(+), 183 deletions(-)
 create mode 100644 website/docs/maintenance/observability/_category_.json
 rename website/docs/maintenance/{ => observability}/logging.md (95%)
 rename website/docs/maintenance/{ => observability}/metric-reporters.md (99%)
 rename website/docs/maintenance/{ => observability}/monitor-metrics.md (81%)
 create mode 100644 website/docs/maintenance/observability/quickstart.md

diff --git a/website/docs/assets/fluss-quickstart-observability.zip b/website/docs/assets/fluss-quickstart-observability.zip
index 3ebcb67a3fc7c6f4d62adca215ec2f0a73d7dae5..8a1ecda7e7b20234236c0a418a39e0b236f4b534 100644
GIT binary patch
delta 1978
zcmah~c{JN;7fvLC*w=}r1Y;>8ZNy$``KFdMr7EQM3PLTh<^(0yjv338AI9EJQMFZT
zrxa6LJG6tABDU5ts<tT|N+%=H_MG{?Kfd>z_dWOA_dfUBKc4$MaiM$B0=75^6b@pY
z=pqHDOaUy{4-6{#M+OxD!^JXPts$aPw4+c8P|^~~95;eQ4N@B<z~?zW=f&X1;5S(c
zPCz1rz!miyl{~D%h6uBgmmARvLL4Aa4g>_k0XHD#029kk=K(=jg{&Jv;{}1Ct4H7L
zC&ms>nKBXTJa9~#&$5JCPT?2;Z{q`Nd~)$Q{fr4xL;B>*#y!&BUy{;7x2$*jrt)<+
zJNFSV^E|(Xmij4Lt8mNKshB;!i>R<6;YZFh2-;|bJb@GEo|Y)@s8Nnwx*K;+obJ^_
z;Z{gYpbrdj#{v;{>NJGS!W{@-MrF_XH$Jb{V9l*u<wn?yJTn7TZ@h`2X@8<Ua%cR7
zNDk7sd~OG0A<_)4vw#;zdtouNTh$%C;E%Jx%)k!a1+UTObwqRvDX`iq#-#lTrUaE0
z9x{+|P(UUCVIHd#Mbuvfn_xP>v`AjVu$1%f68GzS_3I2D*XES0U^yyqbGMazPL{oQ
zSWfRM7=I$G8^bdn`AdRnwpo$DynlB8YKp$w>vw*;MLoDiHw!BXv>mbbO0)HHh(w)8
zY)qnRWU^ccMh?7PL%dGyr~W-z_HNm7W&^3zj<J24C{MN=w1aQp2>f&hXilt3b$gR^
zz4K~&ph2_aDIBV0v~*LEiG5kjP0AuPbH|faW{WJ=B8{rD38aH|VUPF@pYU6>ID4q?
z;<k@jt6&0$k--enRFe<1VX}~Gw&%T@ORNboZL#apmBCJ!in%O)+zOhrkaGUMV)tgx
z!ke9tsTHPP?Xm^}rn!RKe}Z(q|5|NWlTTx}u;a!(o9E0_>iw;;Pdmt;O96$ehssGJ
z{uVJ?pBRT+fCtRyD-Ykm$ZQhgxgLBWVLCgJ33rfY2XB=!c!W=7C_1re{rOQk;+}#?
zSbz`qDw<$_#6M|q8d+KVn5eAc)x)Gtp(7Rs!9XA*Rx$m!IDkU-yh8Fv1yKC`P$WuF
z7$v}4lNK28wF%~^(SMWJ9#5Ip6Y5-XJfAG(3&w>v;04_#;|ikuNuq5jGpHyvRlje;
z6f@ygeWw%NQRYUz_g4>D;~gFuD%7QvEOv#?XvHz<AWcHsn8GxXCfb$!eq=wIJ5+c(
z<uSRoN@Q8hCTVE?sj{2vXs+uIi1Hq&TK+BALsM>R*|8BL$zRXjO}7<!)+yXkDO$9V
z`ToOm39kw@)jff-dRxhF#jYU5^*M}tiB3Wyo`meSdCa+2@P~c|5qfT^e|DN*%v`eX
zt}H8arbiXQ8M3VtUFl6rc@Ng9YundbD#H8!)S13t-M6ISGVJm?_U?gN7#4<|C76x<
zF(N>-TJ)3^UMixP`pxH`i}E4hW2=*u)~|cmUu|(A@n&<Ano`uJ$0#;ASiK8K&NiqD
zvBz_%d%&ZK$nTs-N!`7+AHhGDP*{j`LHnrFiL>2Hn8QXQI#+(??*!zb03H3Sjk$^4
zPX-I-m1`$cs_y3Wl>s5iuSzqka&u3fmZ(heGs!2MF&}(kboFxd?f<xPU!qJ3Jt^aS
z`klN+3x2CFJVI-txTJW0(S-ZwwMm5Kv%PWRJyh+5I||j|myDq$!J%MXq}8R-+xK>&
z_32{`QH93q;6jJ2Z}dV!Bh+P5!h(i+<+&my`Q5*BJ3(inPh9SO<Gfkb26L?)^VFsZ
zw`}{M;PyG4I%BCLzKady9&>8ooySc)1cRGY+3La!CQVmLn~6YkKBNykeHw7hq2fd^
zQEsq8T#R9C{wy*n^6YpoE(lfYZEbhnjeZv9{^*c9acy!exHPT2`&ye*OT5hH>6?57
zkI4u1v7*%5sqQx@zRa%G7(V{zDcxZTUcBjN^7*DjT@Yf*iD2Wyte3nnW5e-5a<try
zo(ua4H6;5CF8_cIL+{v+@(xm+zv--GtK0ThfzRq6A|n|i&pRJc0qFAkkc(;Qk+uo(
z(IaroEn~^I1A>b157%V}MHLTfCN`IXQH<EeSyh3?>fk@7g$!!UfB(_AaRwrPDdNnu
zEaPR~sEX#go5F|b?9P${Sj~n>o$6XwDHC$E#!^5ETKDTJGtdO~!eUR!aX?x212|&L
zAV>ZK#B0ekv|`-BP`0l~V@WK#0OGMS{QvS4pbG1K{G%lUXll!XlK{MS666yM1k~}E
z0Af0LNHY&BwghT)Ou*W}x=tZSni8;~g9j#bPs{(0NPrNZ<w6Mb$utYif+7XJhz2KH
zG+0xxf&UVW&p_}IyfJ?J7^wXJ1msIY_V7XgLQeof(*gmTXm!9|&kVv~$A-WQJy`)k
PkRT|N3k15T!y51(8~H?5

delta 946
zcmexS-B83A;LXg!#Q*}0n`TE&<TGJolFCnyxH|cwt?<NXA0`NE@_#mtiAVgIBd!*1
z7Gv~bVhRT<kc8>|y=G1%*bpe;F!>^<<>dbsT->1{oD9rTNd@T;9Uy}QL8|A7+^BxT
z$iSe^%)sCcRV~g1RSUFcav{5LebH<Op1q$%rG8I3W3lb^OVuwfx!V(8IUS#nRwUAW
z;P#<+W#yXB-fMRlcP@{aF?AjgnD9T_a5ZhiCcDZVg4;Jl=G!@4Il;u;vOcVwO|B%^
zEA;Te>8BIUiLz^YSZXZ{R*n3i-gIKYgZrwR8|J6Xy~1U7+VoCO!%bDw-G7hpG}pgd
zaFlV%w+og@TrU?);$@u6D#;(zcwud(ik9OA@A_G{Vmu<)Y~nRa9!|V?qT=xSSGIf?
z{F|1)f3lY|eD%Kf(=ArUW<{@`CB7tD_R(9rli$kMRII8D;k9I+_OIm9r!_|zG9o4%
zXi7D^(Y&o;tLnK*TdsF&zr4B3G-q8+|FjuPr_`T|jx70iYHfI>&6jG1->qC}foz3l
z&F2Hf(z%~ibbW6BV%Wa?MMcjfn|tXK<PvAA?Y8K@+!7zJ;5F}I^qe<h4sS%aFWkIw
zah;f^yL{#y*%X&>Cau3pCnSY7O=@}`RN8Lpxc2{~(8srUZ-(tEoZa_y>X}O?!m7Rm
zKiR+Vm~f&`{iJ8V1#Vb>yj{6v-u1;=k>zEpq8X;0$aZ7z;$33opsa5$e5hSP<Kp7W
zb48yVVtBwOvQ6X=|LW*RX>+-cP5pJS;6e<0V8O)Mhn^>Pm2BgTzy6<%@BQ`fuQmUl
z&fNOAE%8H{=6%T}-tT6lRes<{B*eR=b0LXo@^rRfXi5X7n|yHUD`elu#8i46lsrLc
zX|g?6F<5*nSC}-I!Kc>8Il0N$9ifqD@+;#&u$-NVJQE8jolj0Qb7BT2^vQ-+G82=9
zHeWIGWn=**%gL%1$)Myqd7-5YFG@O{yu~6FECbXs$yQ*phb13V0^4Lq3yaAUEWMd-
zLYNO!IVa0m`7%x71j{^7<)6I8%85yk3(PgJPGZVXh03T*{$Q;%$ySarZ?d6{(qu^!
g?#Vv3PK=kJJn6}2Y~}cP8F(2MvoSEZ8-a2b0O+EM<p2Nx

diff --git a/website/docs/install-deploy/overview.md b/website/docs/install-deploy/overview.md
index 5bbf187d..6f03c63e 100644
--- a/website/docs/install-deploy/overview.md
+++ b/website/docs/install-deploy/overview.md
@@ -124,8 +124,8 @@ We have listed them in the table below the figure.
                 CoordinatorServer/TabletServer report internal metrics and Fluss client (e.g., connector in Flink jobs) can report additional, client specific metrics as well.
             </td>
             <td>
-               <li>[JMX](/docs/maintenance/metric-reporters#jmx)</li>
-               <li>[Prometheus](/docs/maintenance/metric-reporters#prometheus)</li>
+               <li>[JMX](/docs/maintenance/observability/metric-reporters#jmx)</li>
+               <li>[Prometheus](/docs/maintenance/observability/metric-reporters#prometheus)</li>
             </td>
         </tr>
     </tbody>
diff --git a/website/docs/maintenance/observability/_category_.json b/website/docs/maintenance/observability/_category_.json
new file mode 100644
index 00000000..d6a5d582
--- /dev/null
+++ b/website/docs/maintenance/observability/_category_.json
@@ -0,0 +1,4 @@
+{
+  "label": "Observability",
+  "position": 4
+}
diff --git a/website/docs/maintenance/logging.md b/website/docs/maintenance/observability/logging.md
similarity index 95%
rename from website/docs/maintenance/logging.md
rename to website/docs/maintenance/observability/logging.md
index 63607515..7676cc7d 100644
--- a/website/docs/maintenance/logging.md
+++ b/website/docs/maintenance/observability/logging.md
@@ -1,6 +1,6 @@
 ---
 sidebar_label: Logging
-sidebar_position: 6
+sidebar_position: 4
 ---
 
 # Logging
@@ -21,7 +21,7 @@ Log4j periodically scans this file for changes and adjusts the logging behavior
 
 
 ### Log4j 2 configuration
-The following [logging-related configuration options](./configuration.md) are available:
+The following [logging-related configuration options](../configuration.md) are available:
 
 | Configuration                   | Description                                                             | Default                        |
 |---------------------------------|-------------------------------------------------------------------------|--------------------------------|
@@ -56,6 +56,10 @@ For Fluss distributions this means you have to:
 * remove the `log4j-slf4j-impl` jar from the lib directory.
 * add the `logback-core`, and `logback-classic` jars to the lib directory.
 
+:::info
+Fluss currently uses SLF4J 1.7.x, which is _incompatible_ with logback 1.3.0 and higher.
+:::
+
 The Fluss distribution ships with the following logback configuration files in the conf directory, which are used automatically if logback is enabled:
 * `logback-console.xml`: used for CoordinatorServer/TabletServer if they are run in the foreground (e.g., Kubernetes).
 * `logback.xml`: used for CoordinatorServer/TabletServer by default.
diff --git a/website/docs/maintenance/metric-reporters.md b/website/docs/maintenance/observability/metric-reporters.md
similarity index 99%
rename from website/docs/maintenance/metric-reporters.md
rename to website/docs/maintenance/observability/metric-reporters.md
index f4e796cb..2e387adc 100644
--- a/website/docs/maintenance/metric-reporters.md
+++ b/website/docs/maintenance/observability/metric-reporters.md
@@ -1,6 +1,6 @@
 ---
 sidebar_label: Metric Reporters
-sidebar_position: 4
+sidebar_position: 2
 ---
 
 # Metric Reporters
diff --git a/website/docs/maintenance/monitor-metrics.md b/website/docs/maintenance/observability/monitor-metrics.md
similarity index 81%
rename from website/docs/maintenance/monitor-metrics.md
rename to website/docs/maintenance/observability/monitor-metrics.md
index 7304a7e2..4d4b5175 100644
--- a/website/docs/maintenance/monitor-metrics.md
+++ b/website/docs/maintenance/observability/monitor-metrics.md
@@ -1,6 +1,6 @@
 ---
 sidebar_label: Monitor Metrics
-sidebar_position: 5
+sidebar_position: 3
 ---
 
 # Monitor Metrics
@@ -690,175 +690,4 @@ How to use flink metrics, you can see [flink metrics](https://nightlies.apache.o
             <td>Meter</td>
         </tr>  
     </tbody>
-</table>
-
-## Observability (Prometheus + Grafana)
-
-We provide a minimal quickstart configuration for application observability with Prometheus and
-Grafana [here](../assets/fluss-quickstart-observability.zip). The quickstart configuration comes with 2 dashboards.
-
-- `Fluss – overview`: Selected metrics to observe the overall cluster status
-- `Fluss – detail`: Majority of metrics listed in [metrics list](#metrics-list)
-
-
-### Quickstart
-
-Based on the [Flink quickstart guide](/docs/quickstart/flink), you can add observability capabilities as follows.
-
-1. Download the [observability quickstart configuration](../assets/fluss-quickstart-observability.zip) and extract the ZIP archive in your working directory.
-After extracting the archive, the contents of the working directory should be as follows.
-
-```
-├── docker-compose.yml              # docker compose manifest from quickstart guide
-└── fluss-quickstart-observability  # downloaded and extracted ZIP archive
-    ├── grafana
-    │   ├── grafana.ini
-    │   └── provisioning
-    │       ├── dashboards
-    │       │   ├── default.yml
-    │       │   └── fluss
-    │       │       └── ...
-    │       └── datatsources
-    │           └── default.yml
-    └── prometheus
-        └── prometheus.yml
-```
-
-
-2. Next, you need to adapt the `docker-compose.yml` manifest and
-
-- add containers for Prometheus and Grafana and mount the corresponding configuration directories, and
-- configure Fluss to expose metrics via Prometheus
-```
-metrics.reporters: prometheus
-metrics.reporter.prometheus.port: 9250
-```
-- configure Flink to expose metrics via Prometheus
-```
-metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
-metrics.reporter.prom.port: 9250
-```
-
-You can simply copy the manifest below into your `docker-compose.yml`
-
-<!-- TODO: based on manifest in Flink quickstart guide + additions (see enumeration above) -->
-```yaml
-services:
-  #begin Flink cluster
-  coordinator-server:
-    image: fluss/fluss:0.5.0
-    command: coordinatorServer
-    depends_on:
-      - zookeeper
-    environment:
-      - |
-        FLUSS_PROPERTIES=
-        zookeeper.address: zookeeper:2181
-        coordinator.host: coordinator-server
-        remote.data.dir: /tmp/fluss/remote-data
-        lakehouse.storage: paimon
-        paimon.catalog.metastore: filesystem
-        paimon.catalog.warehouse: /tmp/paimon
-        metrics.reporters: prometheus
-        metrics.reporter.prometheus.port: 9250
-  tablet-server:
-    image: fluss/fluss:0.5.0
-    command: tabletServer
-    depends_on:
-      - coordinator-server
-    environment:
-      - |
-        FLUSS_PROPERTIES=
-        zookeeper.address: zookeeper:2181
-        tablet-server.host: tablet-server
-        data.dir: /tmp/fluss/data
-        remote.data.dir: /tmp/fluss/remote-data
-        kv.snapshot.interval: 0s
-        lakehouse.storage: paimon
-        paimon.catalog.metastore: filesystem
-        paimon.catalog.warehouse: /tmp/paimon
-        metrics.reporters: prometheus
-        metrics.reporter.prometheus.port: 9250
-  zookeeper:
-    restart: always
-    image: zookeeper:3.9.2
-  #end
-  #begin Flink cluster
-  jobmanager:
-    image: fluss/quickstart-flink:1.20-0.5
-    ports:
-      - "8083:8081"
-    command: jobmanager
-    environment:
-      - |
-        FLINK_PROPERTIES=
-        jobmanager.rpc.address: jobmanager
-        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
-        metrics.reporter.prom.port: 9250
-    volumes:
-      - shared-tmpfs:/tmp/paimon
-  taskmanager:
-    image: fluss/quickstart-flink:1.20-0.5
-    depends_on:
-      - jobmanager
-    command: taskmanager
-    environment:
-      - |
-        FLINK_PROPERTIES=
-        jobmanager.rpc.address: jobmanager
-        taskmanager.numberOfTaskSlots: 10
-        taskmanager.memory.process.size: 2048m
-        taskmanager.memory.framework.off-heap.size: 256m
-        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
-        metrics.reporter.prom.port: 9250
-    volumes:
-      - shared-tmpfs:/tmp/paimon
-  #end
-  #begin observability
-  prometheus:
-    image: bitnami/prometheus:2.55.1-debian-12-r0
-    ports:
-      - 9092:9090
-    volumes:
-      - ./fluss-quickstart-observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
-  grafana:
-    image:
-      grafana/grafana:11.4.0
-    ports:
-      - 3002:3000
-    depends_on:
-      - prometheus
-    volumes:
-      - ./fluss-quickstart-observability/grafana:/etc/grafana:ro
-  #end
-
-volumes:
-  shared-tmpfs:
-    driver: local
-    driver_opts:
-      type: "tmpfs"
-      device: "tmpfs"
-```
-
-and run
-
-```shell
-docker compose up -d
-```
-
-to apply the changes.
-
-:::warning
-This recreates `shared-tmpfs` and all data is lost (created tables, running jobs, etc.)
-:::
-
-Make sure that the Prometheus and Grafana container are up and running using
-
-```shell
-docker ps
-```
-
-3. Now you are all set! You can visit
-
-- [Grafana](http://localhost:3002/dashboards) to observe the cluster status of the Fluss and Flink cluster with the provided dashboards, or
-- the [Prometheus Web UI](http://localhost:9092) to directly query Prometheus with [PromQL](https://prometheus.io/docs/prometheus/2.55/getting_started/).
\ No newline at end of file
+</table>
\ No newline at end of file
diff --git a/website/docs/maintenance/observability/quickstart.md b/website/docs/maintenance/observability/quickstart.md
new file mode 100644
index 00000000..c3798e4f
--- /dev/null
+++ b/website/docs/maintenance/observability/quickstart.md
@@ -0,0 +1,220 @@
+---
+sidebar_label: Quickstart Guides
+sidebar_position: 1
+---
+
+# Observability Quickstart Guides
+
+On this page, you can find the following guides to set up and observability stack based on the instructions in the [Flink quickstart guide](/docs/quickstart/flink):
+
+- [Observability with Prometheus, Loki and Grafana](#observability-with-prometheus-loki-and-grafana)
+
+## Observability with Prometheus, Loki and Grafana
+
+We provide a minimal quickstart configuration for application observability with Prometheus (metric aggregation system), Loki (log aggregation sytem) and Grafana (dashboard system)
+ [here](../../assets/fluss-quickstart-observability.zip). 
+
+The quickstart configuration comes with 2 metric dashboards.
+
+- `Fluss – overview`: Selected metrics to observe the overall cluster status
+- `Fluss – detail`: Majority of metrics listed in [metrics list](./monitor-metrics.md#metrics-list)
+
+Follow the instructions below to add observability capabilities to your setup.
+
+1. Download the [observability quickstart configuration](../../assets/fluss-quickstart-observability.zip) and extract the ZIP archive in your working directory.
+After extracting the archive, the contents of the working directory should be as follows.
+
+```
+├── docker-compose.yml              # docker compose manifest from quickstart guide
+└── fluss-quickstart-observability  # downloaded and extracted ZIP archive
+    ├── grafana
+    │   ├── grafana.ini
+    │   └── provisioning
+    │       ├── dashboards
+    │       │   ├── default.yml
+    │       │   └── fluss
+    │       │       └── ...
+    │       └── datatsources
+    │           └── default.yml
+    ├── prometheus
+    │   └── prometheus.yml
+    └── slf4j
+        └── ...
+```
+
+2. Next, you need to configure Fluss to expose logs to Loki. We will use [Loki4j](https://loki4j.github.io/loki-logback-appender/) which uses Logback as logging backend.
+The container manifest below configures Fluss to use Logback and Loki4j. Save it to a file named `fluss-slf4j-logback.Dockerfile` in your working directory.
+
+```dockerfile
+# should be the same Fluss image as in the Flink quickstart guide
+FROM fluss/fluss:0.5.0
+
+ENV ENV_BASE_DIR /opt/fluss
+
+# remove default logging backend from classpath and add logback to classpath
+RUN rm -rf ${ENV_BASE_DIR}/lib/log4j-slf4j-impl-*.jar && \
+    wget https://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.2.13/logback-classic-1.2.13.jar -P ${ENV_BASE_DIR}/lib/ && \
+    wget https://repo1.maven.org/maven2/ch/qos/logback/logback-core/1.2.13/logback-core-1.2.13.jar -P ${ENV_BASE_DIR}/lib/
+
+# add loki4j logback appender to classpath
+RUN wget https://repo1.maven.org/maven2/com/github/loki4j/loki-logback-appender/1.4.2/loki-logback-appender-1.4.2.jar -P ${ENV_BASE_DIR}/lib/
+
+# logback configuration that exposes metrics to loki
+COPY fluss-quickstart-observability/slf4j/logback-loki-console.xml ${ENV_BASE_DIR}/conf/logback-console.xml
+```
+
+:::note
+Detailed configuration instructions for Fluss and Logback can be found [here](./logging.md#configuring-logback).
+:::
+
+3. Additionally, you need to adapt the `docker-compose.yml` manifest and 
+
+- add containers for Prometheus, Loki and Grafana and mount the corresponding configuration directories, and
+- build and use the new Fluss image manifest (`fluss-sfl4j-logback.Dockerfile`), configure Fluss to expose metrics via Prometheus
+```
+metrics.reporters: prometheus
+metrics.reporter.prometheus.port: 9250
+```
+and add the desired application name that should be used when displaying logs in Grafana as environment variable (`APP_NAME`).
+- configure Flink to expose metrics via Prometheus
+```
+metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
+metrics.reporter.prom.port: 9250
+```
+
+You can simply copy the manifest below into your `docker-compose.yml`
+
+<!-- TODO: based on manifest in Flink quickstart guide + additions (see enumeration above) -->
+```yaml
+services:
+  #begin Fluss cluster
+  coordinator-server:
+    image: fluss-slf4j-logback:0.5.0
+    build:
+      dockerfile: fluss-slf4j-logback.Dockerfile
+    command: coordinatorServer
+    depends_on:
+      - zookeeper
+    environment:
+      - |
+        FLUSS_PROPERTIES=
+        zookeeper.address: zookeeper:2181
+        coordinator.host: coordinator-server
+        remote.data.dir: /tmp/fluss/remote-data
+        lakehouse.storage: paimon
+        paimon.catalog.metastore: filesystem
+        paimon.catalog.warehouse: /tmp/paimon
+        metrics.reporters: prometheus
+        metrics.reporter.prometheus.port: 9250
+      - APP_NAME=coordinator-server
+  tablet-server:
+    image: fluss-slf4j-logback:0.5.0
+    build:
+      dockerfile: fluss-slf4j-logback.Dockerfile
+    command: tabletServer
+    depends_on:
+      - coordinator-server
+    environment:
+      - |
+        FLUSS_PROPERTIES=
+        zookeeper.address: zookeeper:2181
+        tablet-server.host: tablet-server
+        data.dir: /tmp/fluss/data
+        remote.data.dir: /tmp/fluss/remote-data
+        kv.snapshot.interval: 0s
+        lakehouse.storage: paimon
+        paimon.catalog.metastore: filesystem
+        paimon.catalog.warehouse: /tmp/paimon
+        metrics.reporters: prometheus
+        metrics.reporter.prometheus.port: 9250
+        logback.configurationFile=logback-loki-console.xml
+      - APP_NAME=tablet-server
+  zookeeper:
+    restart: always
+    image: zookeeper:3.9.2
+  #end
+  #begin Flink cluster
+  jobmanager:
+    image: fluss/quickstart-flink:1.20-0.5
+    ports:
+      - "8083:8081"
+    command: jobmanager
+    environment:
+      - |
+        FLINK_PROPERTIES=
+        jobmanager.rpc.address: jobmanager
+        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
+        metrics.reporter.prom.port: 9250
+    volumes:
+      - shared-tmpfs:/tmp/paimon
+  taskmanager:
+    image: fluss/quickstart-flink:1.20-0.5
+    depends_on:
+      - jobmanager
+    command: taskmanager
+    environment:
+      - |
+        FLINK_PROPERTIES=
+        jobmanager.rpc.address: jobmanager
+        taskmanager.numberOfTaskSlots: 10
+        taskmanager.memory.process.size: 2048m
+        taskmanager.memory.framework.off-heap.size: 256m
+        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
+        metrics.reporter.prom.port: 9250
+    volumes:
+      - shared-tmpfs:/tmp/paimon
+  #end
+  #begin observability
+  prometheus:
+    image: bitnami/prometheus:2.55.1-debian-12-r0
+    ports:
+      - 9092:9090
+    volumes:
+      - ./fluss-quickstart-observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+  loki:
+    image: grafana/loki:3.3.2
+    ports:
+      - 3102:3100
+  grafana:
+    image:
+      grafana/grafana:11.4.0
+    ports:
+      - 3002:3000
+    depends_on:
+      - prometheus
+      - loki
+    volumes:
+      - ./fluss-quickstart-observability/grafana:/etc/grafana:ro
+  #end
+
+volumes:
+  shared-tmpfs:
+    driver: local
+    driver_opts:
+      type: "tmpfs"
+      device: "tmpfs"
+```
+
+and run
+
+```shell
+# note the --build flag!
+docker compose up -d --build
+```
+
+to apply the changes.
+
+:::warning
+This recreates `shared-tmpfs` and all data is lost (created tables, running jobs, etc.)
+:::
+
+Make sure that the modified and added containers are up and running using
+
+```shell
+docker ps
+```
+
+4. Now you are all set! You can visit
+                     
+- Grafana to view Fluss logs with the [log explorer](http://localhost:3002/a/grafana-lokiexplore-app/) and observe metrics of the Fluss and Flink cluster with the [provided dashboards](http://localhost:3002/dashboards)
+- the [Prometheus Web UI](http://localhost:9092) to directly query Prometheus with [PromQL](https://prometheus.io/docs/prometheus/2.55/getting_started/).
\ No newline at end of file
diff --git a/website/docs/quickstart/flink.md b/website/docs/quickstart/flink.md
index 94f62d0f..711fc7be 100644
--- a/website/docs/quickstart/flink.md
+++ b/website/docs/quickstart/flink.md
@@ -37,11 +37,11 @@ cd fluss-quickstart-flink
 
 <!-- TODO: keep docker-compose manifest in sync at 
  - /docs/quickstart-flink.md 
- - /docs/maintenance/monitor-metrics.md, 
+ - /docs/maintenance/observability/quickstart.md, 
  but MIND differences in the manifests -->
 ```yaml
 services:
-  #begin Flink cluster
+  #begin Fluss cluster
   coordinator-server:
     image: fluss/fluss:0.5.0
     command: coordinatorServer
@@ -135,6 +135,7 @@ to check whether all containers are running properly.
 You can also visit http://localhost:8083/ to see if Flink is running normally.
 
 :::note 
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](../maintenance/observability/quickstart.md) and then continue with this guide.
 - If you want to run with your own Flink environment, remember to download the [fluss-connector-flink](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [paimon-flink](https://paimon.apache.org/docs/0.8/flink/quick-start/) connector jars and then put them to `FLINK_HOME/lib/`.
 - All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
 :::
@@ -493,6 +494,4 @@ docker compose down -v
 to stop all containers.
 
 ## Learn more
-Now that you're up an running with Fluss and Flink, check out 
-- the [Apache Flink Engine](engine-flink/getting-started.md) docs to learn more features with Flink
-- [this guide](/docs/maintenance/monitor-metrics/#observability-prometheus--grafana) to learn how to set up an observability stack for Fluss and Flink.
\ No newline at end of file
+Now that you're up an running with Fluss and Flink, check out the [Apache Flink Engine](engine-flink/getting-started.md) docs to learn more features with Flink.
\ No newline at end of file