Add serial_diff pipeline aggregation. #9673

dwelsch-esi · 2025-04-22T00:13:19Z

Description

Add serial_diff pipeline aggregation.

Issues Resolved

Version

Frontend features

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ve aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]>

Signed-off-by: Dave Welsch <[email protected]>

github-actions · 2025-04-22T00:13:31Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

kolchfa-aws · 2025-04-22T13:47:05Z

@bowenlan-amzn Could you please review this PR? Thanks!

_aggregations/pipeline/serial-diff.md

bowenlan-amzn · 2025-05-07T03:58:23Z

_aggregations/pipeline/serial-diff.md

+```
+{% include copy-curl.html %}
+
+## Example response


I think for this aggregation, it's better to use a visualization to show the response, not the json output

Added the visualization with the explanation.

bowenlan-amzn · 2025-05-07T03:59:14Z

_aggregations/pipeline/serial-diff.md

+
+# Serial diff aggregations
+
+The `serial_diff` aggregation is a parent pipeline aggregation that computes a sequence of differences between values from current and previous aggregations. 


Suggested change

The `serial_diff` aggregation is a parent pipeline aggregation that computes a sequence of differences between values from current and previous aggregations.

The `serial_diff` aggregation is a parent pipeline aggregation that computes the difference between metric values from current and previous bucket, and put the difference in the current bucket.

For parent pipeline aggregation, I suppose the main thing is it check each bucket of a multi-bucket agg and store the results back to each bucket.
Compared to sibling pipeline aggregation, it sees the whole array of buckets and only return one bucket or metric.

With this context in mind, I hope the suggestion makes sense.

Thanks! Reworded and added more context.

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws · 2025-05-16T21:15:20Z

@bowenlan-amzn Thank you for the review! I addressed your comments.

natebower

@kolchfa-aws Please see my comment and changes and let me know if you have any questions. Thanks!

_aggregations/pipeline/serial-diff.md

natebower · 2025-05-20T11:55:59Z

_aggregations/pipeline/serial-diff.md

+## Example: Multi-period differences
+
+Use a larger `lag` value to compare with buckets further back in time. The following example computes differences on weekly byte data with a lag of 4 (comparing each bucket with the bucket 4 weeks back). This has the effect of removing any variation with a period of four weeks:
+


Line 130: "compare with buckets further back in time" needs a little revision for clarity, as does "the bucket 4 weeks back".

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

bowenlan-amzn · 2025-05-20T16:49:39Z

_aggregations/pipeline/serial-diff.md

+
+The `serial_diff` aggregation is a parent pipeline aggregation that calculates the difference between metric values in the current bucket and a previous bucket. It stores the result in the current bucket.
+
+Unlike sibling pipeline aggregations, which operate across all buckets and produce a single output, parent pipeline aggregations process each bucket individually and write the result back into each bucket.


It's better to move this to the main page for pipeline-agg

Signed-off-by: Fanit Kolchina <[email protected]>

…nsearch-doc-website into aggs-pipeline-ser-diff-1

* Refactor pipeline aggregations to match other aggregation types. Remove aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]> * Add serial_diff pipeline aggregation. Signed-off-by: Dave Welsch <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Tech review comment Signed-off-by: Fanit Kolchina <[email protected]> * Remove invalid parameter Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit b81635d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Refactor pipeline aggregations to match other aggregation types. Remove aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]> * Add serial_diff pipeline aggregation. Signed-off-by: Dave Welsch <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Tech review comment Signed-off-by: Fanit Kolchina <[email protected]> * Remove invalid parameter Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

* Refactor pipeline aggregations to match other aggregation types. Remove aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]> * Add serial_diff pipeline aggregation. Signed-off-by: Dave Welsch <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Tech review comment Signed-off-by: Fanit Kolchina <[email protected]> * Remove invalid parameter Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Eric Pugh <[email protected]>

dwelsch-esi and others added 5 commits April 10, 2025 16:00

Refactor pipeline aggregations to match other aggregation types. Remo…

8bb98c1

…ve aggregations/pipeline-agg.md; Add aggregations/pipeline/index.md. Individual aggregation files added in other PRs. Signed-off-by: Dave Welsch <[email protected]>

Merge branch 'opensearch-project:main' into main

2d0fafa

Merge branch 'main' of github.com:dwelsch-esi/opensearch-doc-website

6116325

Merge branch 'main' of github.com:dwelsch-esi/opensearch-doc-website

1de3636

Add serial_diff pipeline aggregation.

e79930f

Signed-off-by: Dave Welsch <[email protected]>

dwelsch-esi requested review from AMoo-Miki, Naarcha-AWS, dlvenable, epugh, kolchfa-aws and natebower as code owners April 22, 2025 00:13

github-actions bot assigned kolchfa-aws Apr 22, 2025

kolchfa-aws added Tech review PR: Tech review in progress Content gap backport 2.19 labels Apr 22, 2025

bowenlan-amzn added this to Performance Roadmap Apr 22, 2025

github-project-automation bot moved this to Todo in Performance Roadmap Apr 22, 2025

bowenlan-amzn self-assigned this Apr 22, 2025

dwelsch-esi commented Apr 28, 2025

View reviewed changes

_aggregations/pipeline/serial-diff.md Outdated Show resolved Hide resolved

kolchfa-aws added backport 3.0 and removed backport 2.19 labels May 6, 2025

bowenlan-amzn reviewed May 7, 2025

View reviewed changes

bowenlan-amzn moved this from Todo to In-Review in Performance Roadmap May 7, 2025

kolchfa-aws added 2 commits May 16, 2025 15:44

Merge branch 'main' into aggs-pipeline-ser-diff-1

b30948d

Doc review

e241547

Signed-off-by: Fanit Kolchina <[email protected]>

natebower reviewed May 20, 2025

View reviewed changes

Apply suggestions from code review

b3b33fa

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws added Done but waiting to merge PR: The work is done and ready to merge and removed Tech review PR: Tech review in progress labels May 20, 2025

bowenlan-amzn approved these changes May 20, 2025

View reviewed changes

kolchfa-aws and others added 2 commits May 20, 2025 14:39

Tech review comment

90d29fd

Signed-off-by: Fanit Kolchina <[email protected]>

Merge branch 'main' into aggs-pipeline-ser-diff-1

4538431

kolchfa-aws approved these changes May 20, 2025

View reviewed changes

github-project-automation bot moved this from In-Review to In Progress in Performance Roadmap May 20, 2025

kolchfa-aws added 2 commits May 20, 2025 14:50

Remove invalid parameter

f1b6f42

Signed-off-by: Fanit Kolchina <[email protected]>

Merge branch 'aggs-pipeline-ser-diff-1' of github.com:dwelsch-esi/ope…

c999623

…nsearch-doc-website into aggs-pipeline-ser-diff-1

kolchfa-aws merged commit b81635d into opensearch-project:main May 20, 2025
5 checks passed

github-project-automation bot moved this from In Progress to Done in Performance Roadmap May 20, 2025

opensearch-trigger-bot bot mentioned this pull request May 20, 2025

[Backport 3.0] Add serial_diff pipeline aggregation. #9945

Merged

github-actions bot pushed a commit that referenced this pull request May 20, 2025

Add serial_diff pipeline aggregation. (#9673) (#9945)

e482af1


		# Serial diff aggregations

		The `serial_diff` aggregation is a parent pipeline aggregation that computes a sequence of differences between values from current and previous aggregations.

		## Example: Multi-period differences

		Use a larger `lag` value to compare with buckets further back in time. The following example computes differences on weekly byte data with a lag of 4 (comparing each bucket with the bucket 4 weeks back). This has the effect of removing any variation with a period of four weeks:


		The `serial_diff` aggregation is a parent pipeline aggregation that calculates the difference between metric values in the current bucket and a previous bucket. It stores the result in the current bucket.

		Unlike sibling pipeline aggregations, which operate across all buckets and produce a single output, parent pipeline aggregations process each bucket individually and write the result back into each bucket.

Add serial_diff pipeline aggregation. #9673

Add serial_diff pipeline aggregation. #9673

Uh oh!

Conversation

dwelsch-esi commented Apr 22, 2025

Description

Issues Resolved

Version

Frontend features

Checklist

Uh oh!

github-actions bot commented Apr 22, 2025

Uh oh!

kolchfa-aws commented Apr 22, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws commented May 16, 2025

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants