feat: add support for time flags and missing components #8

zdtsw · 2024-03-26T12:50:44Z

add support for DW components
add support for DSC and DSCI CR
renaming for both kserve and modelmesh to use the same script
add COMPONENT as flag
move functions into common.sh
add GH action on shellcheck, test run https://github.com/zdtsw-forking/must-gather/actions/runs/8507027602/job/23298174673
add variable in docker image build: we should consider to release image per RHOAI release not only use stable

--
add support for oc adm 4.16 with --since and --since-time flag to limit log on disk space
ref: openshift/must-gather#400
jira: https://issues.redhat.com/browse/RHOAIENG-4981

ref: https://github.com/kubernetes/api/blob/v0.30.0/core/v1/types.go#L6313

--

Test: local build quay.io/wenzhou/must-gather:test9

export MUST_GATHER_SINCE=1h; oc adm must-gather --image=quay.io/wenzhou/must-gather:97f097bf
export MUST_GATHER_SINCE_TIME=2024-05-03T07:02:23Z; oc adm must-gather --image=quay.io/wenzhou/must-gather:97f097bf

For Component Reviews:

a basic check that all CRDs for the component have been added, no missing one.
get to know how must-gather is being used (follow README and it works like a magic) or comment back anything is not working or not clear from README. this part will be used as input for Docs or KCS to customer
in the future, must-gather image should be maintained by component team.

- add support for DW components - renaming for both kserve and modelmesh to use the same script - add COMPONENT as flag - move functions into common.sh Signed-off-by: Wen Zhou <[email protected]>

openshift-ci · 2024-03-26T12:50:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Wen Zhou <[email protected]>

jstourac · 2024-03-28T16:57:53Z

README.md

+
+```
+export COMPONENT=kserve
+oc adm must-gather --image=quay.io/modh/must-gather:stable


I don't see the restriction for the kserve only here.

correct
this COMPONET can be applied for other components too.
here is just an example of usage.
maybe i should make it clear in the README

Oh, right - I missed the export COMPONENT=kserve line yesterday somehow (too tired probably). Then sorry, I understand now. But I like the change you did above to the text - it looks clearer and more obvious to me now. Thank you!

jstourac · 2024-03-28T16:57:56Z

README.md


 The must-gather script allows a cluster admin to collect information about various key resources and namespaces
 for Red Hat OpenShift Data Science.

 ## Data Collected

-The must-gather script currently collects data from following namespaces
+The must-gather script currently collects data from following default namespaces

 - redhat-ods-operator
 - rhods-notebooks
 - redhat-ods-applications


Suggested change

- redhat-ods-applications

- redhat-ods-applications

- redhat-ods-applications-auth-provider

Shall we want it too?

i would say we only cover the namespace for rhoai.
authorino should have its must-gather just like servicemesh and knative have theirs.
so if user will need anything speicific from authorino they should run that image instead.

Hmm, okay - I don't insist because I don't know all the context here. But my understanding was that we create the namespace (per its name), so I though that there will be data that may be relevant.

collection-scripts/gather

jstourac · 2024-03-28T17:10:17Z

collection-scripts/common.sh

+		log_collection_args=--since="${MUST_GATHER_SINCE}"
+	fi
+	if [ -n "${MUST_GATHER_SINCE_TIME:-}" ]; then
+		log_collection_args=--since-time="${MUST_GATHER_SINCE_TIME}"


I'm really surprised if this really works without exporting the value at the end...

i dont follow this comment.
log_collection_args is not a local variable, after source the common.sh
why need to set to a return to get_operator_resource to get it?

Yeah, you are correct. I realized it later too when I was AFK already.

Would be nice to follow a syntax where variables that are meant to be global are upper-case - this improves the code readability from my point of view, since it clearly tells you that this variable should/can be defined/assigned anywhere. Anyway, if we follow what is in original must-gather, we don't need to amend.

jstourac · 2024-03-28T17:12:35Z

collection-scripts/common.sh

+	# an ISO formatted time. since MUST_GATHER_SINCE and MUST_GATHER_SINCE_TIME
+	# are formatted differently, we re-format them so they can be used
+	# transparently by node-logs invocations.
+	node_log_collection_args=""


I don't see this to be used anywhere, so I suppose this gonna be added later?

this function is from openshift-must-gather
they provide more functions including the node log which in our case is not used (no logic for this part)
i dont wanna chop part of the function so leave this part in the PR, but added comment "# even we do not run "oc adm node-logs" in rhoai"

Understand, then it could be probably commented out so it's not confusing; but I don't insist.

i added more comment, prefer not to change original function for now.

Co-authored-by: Jan Stourac <[email protected]>

zdtsw · 2024-03-28T18:36:36Z

README.md


 The must-gather script allows a cluster admin to collect information about various key resources and namespaces
 for Red Hat OpenShift Data Science.

 ## Data Collected

-The must-gather script currently collects data from following namespaces
+The must-gather script currently collects data from following default namespaces

 - redhat-ods-operator
 - rhods-notebooks
 - redhat-ods-applications


i would say we only cover the namespace for rhoai.
authorino should have its must-gather just like servicemesh and knative have theirs.
so if user will need anything speicific from authorino they should run that image instead.

zdtsw · 2024-03-28T18:37:43Z

README.md

+
+```
+export COMPONENT=kserve
+oc adm must-gather --image=quay.io/modh/must-gather:stable


correct
this COMPONET can be applied for other components too.
here is just an example of usage.
maybe i should make it clear in the README

README.md

zdtsw · 2024-03-28T18:53:24Z

collection-scripts/common.sh

+		log_collection_args=--since="${MUST_GATHER_SINCE}"
+	fi
+	if [ -n "${MUST_GATHER_SINCE_TIME:-}" ]; then
+		log_collection_args=--since-time="${MUST_GATHER_SINCE_TIME}"


i dont follow this comment.
log_collection_args is not a local variable, after source the common.sh
why need to set to a return to get_operator_resource to get it?

zdtsw · 2024-03-28T18:54:52Z

collection-scripts/common.sh

+	# an ISO formatted time. since MUST_GATHER_SINCE and MUST_GATHER_SINCE_TIME
+	# are formatted differently, we re-format them so they can be used
+	# transparently by node-logs invocations.
+	node_log_collection_args=""


this function is from openshift-must-gather
they provide more functions including the node log which in our case is not used (no logic for this part)
i dont wanna chop part of the function so leave this part in the PR, but added comment "# even we do not run "oc adm node-logs" in rhoai"

collection-scripts/gather

danielezonca · 2024-03-29T09:52:43Z

collection-scripts/gather_serving

+source "common.sh"
+resources=("inferenceservice" "inferencegraphs")
+
+nslist=$(get_all_namespace "$resources")
+run_mustgather "$nslist"


@Jooho @terrytangyuan
FYI

Feel free to comment if you think it should contain something more (this replace the previous "model_mesh" conf)

collection-scripts/gather_serving

danielezonca · 2024-03-29T10:21:41Z

collection-scripts/gather

+case "$component" in
+    "dsp")
+        /usr/bin/gather_data_science_pipelines
+        ;;
+    "modelmesh")
+        /usr/bin/gather_serving
+        ;;
+    "kserve")
+        /usr/bin/gather_serving
+        ;;
+    "dashboard")
+        echo "It has been included in the default namespace."
+        ;;
+    "workbench")
+        /usr/bin/gather_notebooks
+        ;;
+    "dw")
+        /usr/bin/gather_codeflare
+        /usr/bin/gather_kuberay
+        /usr/bin/gather_kueue
+        ;;
+    "mr"
+        echo "TBD."
+        ;;
+    *) # for all 
+        /usr/bin/gather_data_science_pipelines
+        /usr/bin/gather_serving
+        /usr/bin/gather_notebooks
+        /usr/bin/gather_codeflare
+        /usr/bin/gather_kuberay
+        /usr/bin/gather_kueue
+        ;;


Minor comment, this is a bit error prone: every time we add a new component/command we need to add it under the component and the "all" branch

yep.
i was thinking about this when i added the "COMPONENT" part: maybe we should add "update must-gather as one step in the new component onboarding list"?
or we need a new design to struct these functions for new components in a better way.

README.md

jstourac

Okay, I realized now, that this PR is probably still a WIP so I don't gonna spam you more on this. Sorry if this was too early to comment.

Note: this tool may be usable for you to use https://github.com/koalaman/shellcheck

jstourac · 2024-03-29T08:51:46Z

README.md


 The must-gather script allows a cluster admin to collect information about various key resources and namespaces
 for Red Hat OpenShift Data Science.

 ## Data Collected

-The must-gather script currently collects data from following namespaces
+The must-gather script currently collects data from following default namespaces

 - redhat-ods-operator
 - rhods-notebooks
 - redhat-ods-applications


Hmm, okay - I don't insist because I don't know all the context here. But my understanding was that we create the namespace (per its name), so I though that there will be data that may be relevant.

README.md

jstourac · 2024-03-29T08:52:34Z

README.md

+
+```
+export COMPONENT=kserve
+oc adm must-gather --image=quay.io/modh/must-gather:stable


Oh, right - I missed the export COMPONENT=kserve line yesterday somehow (too tired probably). Then sorry, I understand now. But I like the change you did above to the text - it looks clearer and more obvious to me now. Thank you!

jstourac · 2024-03-29T08:55:10Z

collection-scripts/common.sh

+	# an ISO formatted time. since MUST_GATHER_SINCE and MUST_GATHER_SINCE_TIME
+	# are formatted differently, we re-format them so they can be used
+	# transparently by node-logs invocations.
+	node_log_collection_args=""


Understand, then it could be probably commented out so it's not confusing; but I don't insist.

jstourac · 2024-03-29T09:44:45Z

Dockerfile

@@ -1,6 +1,10 @@
-FROM quay.io/openshift/origin-cli:latest
+# current latest is point to 4.16.0
+FROM quay.io/openshift/origin-must-gather:latest


optional: technically it would be safer to pin to a particular tag instead of the latest.

i thought about using tag, but from what i understand, this new feature with time is just introduced in 4.16
and they wanna make it GA in 4.17
then we will need to update this when 4.17 is out which might be in a couple of month.
meanwhile we are having 2 more new component in the pipeline reaching GA in summer, so we will update in this repo in the new future as well.
would it be better to use tag at that time?

No issues, Wen - I proposed it as an optional, since even before your changes there was latest used. Thus I'm completely fine with what you propose 🙂

jstourac · 2024-03-29T10:22:36Z

collection-scripts/common.sh

+    for kind in "$1"; do
+        nslist+=$(oc get "$kind" --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}')
+    done
+    echo $(uniq_list "$nslist")


Suggested change

echo $(uniq_list "$nslist")

uniq_list "$nslist"

? 🤔

updated in new commit

jstourac · 2024-03-29T10:23:38Z

collection-scripts/common.sh

+
+# run must-gather in the namespaces one by one
+function run_mustgather() {
+    for ns in $1; do


Suggested change

for ns in $1; do

for ns in "$@"; do

jstourac · 2024-03-29T10:23:56Z

collection-scripts/common.sh

+# get the list of namespaces where defined resources exist
+function get_all_namespace() {
+    local nslist
+    for kind in "$1"; do


Suggested change

for kind in "$1"; do

for kind in "$@"; do

updated in new commit

jstourac · 2024-03-29T10:27:22Z

collection-scripts/gather

+        /usr/bin/gather_kuberay
+        /usr/bin/gather_kueue
+        ;;
+    "mr"


Suggested change

"mr"

"mr")

removed "mr" for now

jstourac · 2024-03-29T10:34:04Z

collection-scripts/gather_data_science_pipelines

+nslist=$(get_all_namespace "$resources")
+run_mustgather "$nslist"


same as elsewhere (unless I am missing some global context here):

Suggested change

run_mustgather "$nslist"

run_mustgather "${nslist[@]}"

updated in new commit

Signed-off-by: Wen Zhou <[email protected]>

- super-linter run for PR if open to main or master or rhoai-* branches - disable linter SC1091 for source - disable linter SC2034 2154, we keep log_colleciton_args unused - disable linter SC2001, do not replace sed for variable bash replace - disable linter SC2086, do not force double quote on complicated regex Signed-off-by: Wen Zhou <[email protected]>

Signed-off-by: Wen Zhou <[email protected]>

ykaliuta · 2024-04-03T20:45:06Z

collection-scripts/common.sh

+  version=''
+
+  # get version from rhoai csv
+  csv_name=$(oc get csv|grep rhods|awk '{print $1}')


probably you do not need grep here, awk can address strings with regexp, awk '/rhods/{print $1}'

thanks, updated with suggestion

ykaliuta · 2024-04-03T20:47:05Z

collection-scripts/common.sh

+
+  # if version not found, get version from rhods operator pod's container
+  operator_name=$(oc get pod -n redhat-ods-operator --ignore-not-found |grep redhat-ods-operator |awk '{print $1}')
+  [[ z != z"${operator_name}" ]] && version=$(oc exec ${operator_name} -n redhat-ods-operator -c manager -- env|grep OPERATOR_CONDITION_NAME|awk -F'=' '{print $2}'| sed -n -r -e 's/.*([[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+(:?\.[[:digit:]])?(:?-[^@]+)?).*/\1/p')


Will that sed -n -r -e '/OPERATOR_CONDITION_NAME/s/.*=.*([[:digit:]]+\.[[:digit:]]+\.[[:digit:]]+(:?\.[[:digit:]])?(:?-[^@]+)?).*/\1/p' work the same as the sequence of grep, awk and sed calls?

i changed the check.
the current one requires executing in the pod, new check is based on deployment env settings.

some code in the "common.sh" are copied from old implementation, so i kept them as-was when moving around into new files.

ykaliuta · 2024-04-03T20:53:26Z

collection-scripts/common.sh

+
+  # get version from rhoai csv
+  csv_name=$(oc get csv|grep rhods|awk '{print $1}')
+  [[ z == z"${version}" && z != z"${csv_name}" ]] && version=$(oc get csv "$(oc get csv|grep rhods|awk '{print $1}')" -o json|jq .spec.version |xargs)


Wondering why that old shell programming trick with x$VAR used here? In modern bash -z/-n and even comparation with empty string should work inside of [[ operator.

thanks, updated in new commit

same reason for why not update them at first.

- change to use -z / -n for check variable - get version from deployment env variable, instead of start pod to check - update comments to use RHOAI - use awk for grep+awk Signed-off-by: Wen Zhou <[email protected]>

This reverts commit 3530a1f.

VaishnaviHire · 2024-04-04T16:49:13Z

@dchourasia Can we migrate the new image of must-gather to CPaaS ?

Signed-off-by: Wen Zhou <[email protected]>

zdtsw · 2024-04-10T13:42:56Z

/test odh-manifests-must-gather-pr-image

zdtsw · 2024-04-10T15:57:48Z

/test odh-manifests-must-gather-pr-image

zdtsw · 2024-04-11T06:10:53Z

/test odh-manifests-must-gather-pr-image

- also use fixed tag on base image Signed-off-by: Wen Zhou <[email protected]>

Signed-off-by: Wen Zhou <[email protected]>

- fix log_collection_args which is causing error - fix wrong resource name - fix wrong timestamp format in README - add more into README - fix wrong env variable and use label instead of env variable Signed-off-by: Wen Zhou <[email protected]>

Signed-off-by: Wen Zhou <[email protected]>

zdtsw · 2024-06-18T10:45:04Z

@Jooho / @anishasthana / @HumairAK / @harshad16 add you as reviewers, as for sanity check since the change also updated existing components.

if no objection we will get this change out by Thu in order to catch 2.11 code freeze

gregsheremeta · 2024-06-18T11:28:25Z

DSP stuff here lgtm

but Humair can look too

harshad16

/lgtm
from the workbench/IDE side.

astefanutti

LGTM

README.md

HumairAK · 2024-06-18T21:05:50Z

/lgtm

Signed-off-by: Wen Zhou <[email protected]>

zdtsw · 2024-06-19T06:53:16Z

/approve

sutaakar · 2024-06-19T07:20:20Z

@astefanutti Should AppWrapper be added too?

astefanutti · 2024-06-19T07:25:11Z

@astefanutti Should AppWrapper be added too?

That's a good point. It's a private API, but I guess it can be useful to have it there too.

danielezonca

A couple of minor comments

danielezonca · 2024-04-02T12:48:25Z

.github/workflows/linter.yml

+    - main
+    - master
+    - 'rhoai-[0-9].[0-9]'


Are there branches that should be excluded? This list seems to me that covers ~everything

collection-scripts/gather

README.md

zdtsw · 2024-06-19T08:02:15Z

@astefanutti Should AppWrapper be added too?

That's a good point. It's a private API, but I guess it can be useful to have it there too.

in which component you want AppWrapper to be added? or it does not matter if kuberay or kueue?

astefanutti · 2024-06-19T08:04:30Z

@astefanutti Should AppWrapper be added too?

That's a good point. It's a private API, but I guess it can be useful to have it there too.

in which component you want AppWrapper to be added? or it does not matter if kuberay or kueue?

Technically, it's in the codeflare component, which I realise may be missing here.

zdtsw · 2024-06-19T08:08:55Z

@astefanutti Should AppWrapper be added too?

That's a good point. It's a private API, but I guess it can be useful to have it there too.

in which component you want AppWrapper to be added? or it does not matter if kuberay or kueue?

Technically, it's in the codeflare component, which I realise may be missing here.

correct.
i did not create one for CFO, but i can add that one and only AppWrapper in it, if no others are missing

zdtsw · 2024-06-19T08:15:22Z

@astefanutti / @sutaakar can you review 8ec0bd5 ?

astefanutti · 2024-06-19T08:17:13Z

@astefanutti / @sutaakar can you review 8ec0bd5 ?

@zdtsw LGTM

sutaakar · 2024-06-19T08:18:21Z

LGTM

Signed-off-by: Wen Zhou <[email protected]>

ykaliuta · 2024-06-19T08:53:51Z

collection-scripts/gather


+# set component or default to all
+component=${COMPONENT:-all}


I have a proposal (bash):

declare -A COMPONENTS=( [dsp]=gather_data_science_pipelines [modelmesh]=gather_serving [kserve]=gather_serving [dashboard]=gather_dashboard [workbench]=gather_notebooks [kuberay]=gather_kuberay [kueue]=gather_kueue [kfto]=gather_kfto ) BIN_DIR="$(dirname $0)" component=${COMPONENT:-all} binary=${COMPONENTS[$component]} if [[ -n $binary ]]; then "$BIN_DIR/$binary" else for binary in "${COMPONENTS[@]}"; do "$BIN_DIR/$binary" done fi

but you will need to implement gather_dashboard stub.

i added your suggestion into https://issues.redhat.com/browse/RHOAIENG-8809

zdtsw · 2024-06-19T11:12:00Z

/approve

update:

6ea985a

- add support for DW components - renaming for both kserve and modelmesh to use the same script - add COMPONENT as flag - move functions into common.sh Signed-off-by: Wen Zhou <[email protected]>

zdtsw added 2 commits March 26, 2024 14:49

update: add DSC and DSCI CR into mustgather

e71801a

Signed-off-by: Wen Zhou <[email protected]>

feat(since): add support to use --since and --since-time

37bf89a

Signed-off-by: Wen Zhou <[email protected]>

zdtsw requested a review from VaishnaviHire March 28, 2024 16:03

jstourac reviewed Mar 28, 2024

View reviewed changes

Apply suggestions from code review

29bf882

Co-authored-by: Jan Stourac <[email protected]>

zdtsw commented Mar 28, 2024

View reviewed changes

Update README.md

321995c

danielezonca reviewed Mar 29, 2024

View reviewed changes

jstourac reviewed Mar 29, 2024

View reviewed changes

udpate: code review + shellcheck

f3d4420

Signed-off-by: Wen Zhou <[email protected]>

zdtsw force-pushed the chore branch from dc2505d to f3d4420 Compare March 29, 2024 12:11

zdtsw added 2 commits April 1, 2024 12:43

test(GH)

3530a1f

Signed-off-by: Wen Zhou <[email protected]>

zdtsw changed the title ~~update:~~ feat: add support for time flags and missing components Apr 3, 2024

ykaliuta reviewed Apr 3, 2024

View reviewed changes

zdtsw added 2 commits April 4, 2024 11:43

update: code review

a1a557c

- change to use -z / -n for check variable - get version from deployment env variable, instead of start pod to check - update comments to use RHOAI - use awk for grep+awk Signed-off-by: Wen Zhou <[email protected]>

Revert "test(GH)"

0afc635

This reverts commit 3530a1f.

update(cfo): remove CFO due to 3 CRD have been removed

1464470

Signed-off-by: Wen Zhou <[email protected]>

zdtsw added 2 commits May 2, 2024 12:59

update(serving+dsp): add more resource into list

3e7d4b5

- also use fixed tag on base image Signed-off-by: Wen Zhou <[email protected]>

update: add KFTO

0b869cb

Signed-off-by: Wen Zhou <[email protected]>

zdtsw force-pushed the chore branch 2 times, most recently from cffa842 to 9e18d59 Compare May 2, 2024 16:35

update+fix:

97f097b

- fix log_collection_args which is causing error - fix wrong resource name - fix wrong timestamp format in README - add more into README - fix wrong env variable and use label instead of env variable Signed-off-by: Wen Zhou <[email protected]>

zdtsw force-pushed the chore branch from 9e18d59 to 97f097b Compare May 3, 2024 09:24

update: follow up review comments

21ee930

Signed-off-by: Wen Zhou <[email protected]>

ykaliuta approved these changes Jun 18, 2024

View reviewed changes

zdtsw requested review from Jooho, anishasthana, HumairAK and harshad16 June 18, 2024 08:57

harshad16 reviewed Jun 18, 2024

View reviewed changes

zdtsw requested review from astefanutti and sutaakar June 18, 2024 16:08

astefanutti approved these changes Jun 18, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

update: code review

da6e55c

Signed-off-by: Wen Zhou <[email protected]>

danielezonca reviewed Jun 19, 2024

View reviewed changes

update: add CFO with appwrappers

55538e8

Signed-off-by: Wen Zhou <[email protected]>

zdtsw force-pushed the chore branch from 8ec0bd5 to 55538e8 Compare June 19, 2024 08:31

ykaliuta reviewed Jun 19, 2024

View reviewed changes

danielezonca approved these changes Jun 19, 2024

View reviewed changes

zdtsw merged commit 5ccf43a into red-hat-data-services:main Jun 19, 2024
1 check passed

	- redhat-ods-applications
	- redhat-ods-applications
	- redhat-ods-applications-auth-provider

		nslist=$(get_all_namespace "$resources")
		run_mustgather "$nslist"


		# set component or default to all
		component=${COMPONENT:-all}

feat: add support for time flags and missing components #8

feat: add support for time flags and missing components #8

Conversation

zdtsw commented Mar 26, 2024 • edited Loading

openshift-ci bot commented Mar 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jstourac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VaishnaviHire commented Apr 4, 2024

zdtsw commented Apr 10, 2024

zdtsw commented Apr 10, 2024

zdtsw commented Apr 11, 2024

zdtsw commented Jun 18, 2024

gregsheremeta commented Jun 18, 2024 • edited Loading

harshad16 left a comment

Choose a reason for hiding this comment

astefanutti left a comment

Choose a reason for hiding this comment

HumairAK commented Jun 18, 2024

zdtsw commented Jun 19, 2024

sutaakar commented Jun 19, 2024

astefanutti commented Jun 19, 2024

danielezonca left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zdtsw commented Jun 19, 2024

astefanutti commented Jun 19, 2024

zdtsw commented Jun 19, 2024

zdtsw commented Jun 19, 2024

astefanutti commented Jun 19, 2024

sutaakar commented Jun 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zdtsw commented Jun 19, 2024

zdtsw commented Mar 26, 2024 •

edited

Loading

gregsheremeta commented Jun 18, 2024 •

edited

Loading