EmfMetricPublisher class added #5752

Fred1155 · 2024-12-13T23:01:41Z

Motivation and Context

The existing CloudWatchMetricPublisher has limitations when used with AWS Lambda functions. When current implementation be applied to lambda functions, it can results in metrics being lost when the lambda function terminates before the scheduled batch upload. EMF allows metrics to be published through CloudWatch Logs using a structured JSON format, which CloudWatch automatically processes to extract and publish metrics. The proposed EMFMetricPublisher will convert SDK metric collections into EMF-formatted log entries, leveraging Lambda's as well as other execution environment that has built-in integration with cloudwatch logs such as ECS.

Modifications

Add a new metric publisher that transforms an metricCollection to a list of emf format Strings and publish it to CouldWatch using cloudwatch agents

Testing

Unit test covering different cases of metricCollection in convertMetricCollectionToEMF method
End to End test (In process)

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

I have read the CONTRIBUTING document
Local run of mvn install succeeds
My code follows the code style of this project
My change requires a change to the Javadoc documentation
I have updated the Javadoc documentation accordingly
I have added tests to cover my changes
All new and existing tests passed
I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

I confirm that this pull request can be released under the Apache 2 license

…cPublisher

zoewangg

Left some high level comments, still reviewing the implementation

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisher.java

...sher/src/test/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisherTest.java

zoewangg · 2024-12-13T23:47:47Z

...sher/src/test/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisherTest.java

+    }
+
+    @Test
+    void testConvertMetricCollectionToEMF_EmptyCollection(){


Consider using parameterized tests if we are just testing with different input

…-v2 into bole/EMFMetricPublisher

metric-publishers/emf-metric-publisher/pom.xml

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisher.java

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/MetricEmfConverter.java

zoewangg · 2024-12-18T00:25:36Z

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/MetricEmfConverter.java

+        // Write CloudWatchMetrics array
+        jsonWriter.writeFieldName("CloudWatchMetrics");
+        jsonWriter.writeStartArray();
+        jsonWriter.writeStartObject();


Is this correct? It seems this CloudWatchMetrics only has one object

I think so. In the example https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html, CloudWatchMetrics should have one object, because we only have one namespace and one set of dimensions

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/MetricEmfConverter.java

zoewangg · 2024-12-18T00:52:57Z

metric-publishers/emf-metric-publisher/pom.xml

+        <version>2.29.35-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>emf-metric-publisher</artifactId>


We don't have to change it now, but we should discuss it in API surface area review. Should we rename it to emf-metric-logger, since technically it's not publishing metrics?

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisher.java

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/MetricEmfConverter.java

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisher.java

zoewangg · 2024-12-23T22:20:59Z

...sher/src/test/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisherTest.java

+import software.amazon.awssdk.testutils.LogCaptor;
+
+
+public class EmfMetricPublisherTest extends LogCaptor.LogCaptorTestBase{


Can we add a test case where metricCollector throws an exception?

...test/java/software/amazon/awssdk/metrics/publishers/emf/internal/MetricEmfConverterTest.java

zoewangg

It's probably easier if you see the code, so I pushed suggested changes for MetricEmfConverter here :) Feel free to change as needed. bole/EMFMetricPublisher...zoewang/reviewPR5752

metric-publishers/emf-metric-publisher/pom.xml

...ublisher/src/main/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisher.java

...main/java/software/amazon/awssdk/metrics/publishers/emf/internal/EmfMetricConfiguration.java

...sher/src/test/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisherTest.java

...src/main/java/software/amazon/awssdk/metrics/publishers/emf/internal/MetricEmfConverter.java

...test/java/software/amazon/awssdk/metrics/publishers/emf/internal/MetricEmfConverterTest.java

metric-publishers/emf-metric-publisher/src/main/resources/log4j2.properties

...sher/src/test/java/software/amazon/awssdk/metrics/publishers/emf/EmfMetricPublisherTest.java

...src/main/java/software/amazon/awssdk/metrics/publishers/emf/internal/MetricEmfConverter.java

...test/java/software/amazon/awssdk/metrics/publishers/emf/internal/MetricEmfConverterTest.java

zoewangg · 2025-01-03T22:03:35Z

...test/java/software/amazon/awssdk/metrics/publishers/emf/internal/MetricEmfConverterTest.java

+
+
+                ObjectMapper mapper = new ObjectMapper();
+                JsonNode jsonNode = mapper.readTree(emfLogs.get(0));
+                Set<ValidationMessage> errors = schema.validate(jsonNode);


formatting is a bit weird

...test/java/software/amazon/awssdk/metrics/publishers/emf/internal/MetricEmfConverterTest.java

metric-publishers/emf-metric-publisher/pom.xml

zoewangg

Good job!

sonarqubecloud · 2025-01-03T23:29:21Z

Quality Gate passed

Issues
27 New issues
0 Accepted issues

Measures
0 Security Hotspots
89.5% Coverage on New Code
2.3% Duplication on New Code

See analysis details on SonarQube Cloud

Fred1155 added 5 commits December 11, 2024 09:27

EmfMetricPublisher class with basic unit test added

06e8f84

Edge cases handled

1e08463

javadoc for class added

d073d86

merge master branch

cf0af99

Java doc for methods added

5e34796

Fred1155 requested a review from a team as a code owner December 13, 2024 23:01

Merge branch 'feature/master/emf-metric-publisher' into bole/EMFMetri…

9f221ff

…cPublisher

zoewangg reviewed Dec 13, 2024

View reviewed changes

Fred1155 added 8 commits December 13, 2024 16:51

minor unit test changes

c8ea4aa

Merge branch 'bole/EMFMetricPublisher' of github.com:aws/aws-sdk-java…

79cf7d4

…-v2 into bole/EMFMetricPublisher

Change from using jacksonCore directly to using jsonWriter

04c837f

EmfMetricConfiguration class added

2920596

MetricEmfConverter class added

c79e15f

new module checklist done

e9320b0

checkStyle fixed

94e70a8

minor build problem fixed

9254b13

zoewangg reviewed Dec 18, 2024

View reviewed changes

Fred1155 added 4 commits December 19, 2024 15:07

internal package added

980fdd6

fixed the issue passing raw objects

6b36d4e

added java clock to unit test

02a1036

added unit test for publish method

bb7f690

zoewangg reviewed Dec 23, 2024

View reviewed changes

Fred1155 added 7 commits December 23, 2024 16:01

minor changes

f3084f3

config class to builder pattern

34b161e

config class to builder pattern

87a931a

minor change

4e43df3

End to end test passed

9e32782

added valid case in WARN_LOG allowlist

964be82

unit tests adjusted

dcb5589

Fred1155 added 2 commits January 2, 2025 14:40

Change logGroupName to required field

4ed8298

move MetricValueNormalizer class to utils package

7ee4714

zoewangg reviewed Jan 2, 2025

View reviewed changes

converter implementation changed

a6047a0

zoewangg reviewed Jan 3, 2025

View reviewed changes

schemaConformTest added

ee67d0e

zoewangg reviewed Jan 3, 2025

View reviewed changes

Fred1155 added 2 commits January 3, 2025 14:45

SchemaConformTest adjusted

e61b17e

minor change

eae5c76

zoewangg approved these changes Jan 3, 2025

View reviewed changes

Fred1155 merged commit e8b9bbd into feature/master/emf-metric-publisher Jan 3, 2025
17 checks passed

Fred1155 deleted the bole/EMFMetricPublisher branch January 6, 2025 17:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EmfMetricPublisher class added #5752

EmfMetricPublisher class added #5752

Fred1155 commented Dec 13, 2024 •

edited

Loading

zoewangg left a comment

zoewangg Dec 13, 2024

zoewangg Dec 18, 2024

Fred1155 Dec 18, 2024

zoewangg Dec 18, 2024

zoewangg Dec 23, 2024

zoewangg left a comment •

edited

Loading

zoewangg Jan 3, 2025

zoewangg left a comment

sonarqubecloud bot commented Jan 3, 2025

		import software.amazon.awssdk.testutils.LogCaptor;


		public class EmfMetricPublisherTest extends LogCaptor.LogCaptorTestBase{

EmfMetricPublisher class added #5752

EmfMetricPublisher class added #5752

Conversation

Fred1155 commented Dec 13, 2024 • edited Loading

Motivation and Context

Modifications

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

zoewangg left a comment

Choose a reason for hiding this comment

zoewangg Dec 13, 2024

Choose a reason for hiding this comment

zoewangg Dec 18, 2024

Choose a reason for hiding this comment

Fred1155 Dec 18, 2024

Choose a reason for hiding this comment

zoewangg Dec 18, 2024

Choose a reason for hiding this comment

zoewangg Dec 23, 2024

Choose a reason for hiding this comment

zoewangg left a comment • edited Loading

Choose a reason for hiding this comment

zoewangg Jan 3, 2025

Choose a reason for hiding this comment

zoewangg left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 3, 2025

Quality Gate passed

Fred1155 commented Dec 13, 2024 •

edited

Loading

zoewangg left a comment •

edited

Loading