Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v2] Benchmark harness improvements & add more service benchmarks #9264

Open
wants to merge 16 commits into
base: benchmarks
Choose a base branch
from

Conversation

aemous
Copy link
Contributor

@aemous aemous commented Jan 30, 2025

Issue #, if available:
N/A

Description of changes:

  • Adds support for specifying file literals in the benchmark definitions JSON.
  • Updated README with documentation on specifying file literals.
  • Added dynamodb benchmarks for get-item, batch-get-item, put-item commands, and a benchmark for the customization ddb put.
  • Aded ec2, ecs, cloudwatch logs, and cloudwatch benchmarks.
  • Added STS benchmarks for AssumeRole, GetCallerIdentity, AssumeRoleWithWebIdentity.
  • Removed S3 benchmarks using the CRT client because they are currently non-functional with a non-trivial fix. Will possibly revisit getting them working later.
  • Fix bug in benchmark summarizer with computing averages.
  • Add support to the benchmark harness for debugging the child process.
  • Redirect child process output to files so that the only output of run-benchmarks script on success is the benchmark results.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@aemous aemous changed the title Benchmarks more services [v2] Benchmark harness improvements & DynamoDB benchmarks Jan 30, 2025
@aemous aemous requested review from aws-sdk-python-automation and a team January 30, 2025 16:14
@aemous aemous marked this pull request as ready for review January 30, 2025 16:14
@aemous aemous changed the title [v2] Benchmark harness improvements & DynamoDB benchmarks [v2] Benchmark harness improvements & add more service benchmarks Jan 30, 2025
"name": "s3.sync.upload",
"command": ["s3", "sync", "./test_dir", "s3://bucket/test_dir", "--quiet"],
"name": "dynamodb.getitem",
"command": ["dynamodb", "get-item", "--table-name", "MyTable", "--key", "file://key.json", "--return-consumed-capacity", "TOTAL"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much does the file:// usage impact this output as opposed to specifying the key inline? I see we use it in examples, but curious if it's worth separating DDB's performance from that of file://? But if it's relatively small compared to the full operation, okay with this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One argument for file usage is so that we can have performance testing coverage of that feature of the CLI.

I can see the value of keeping it out if possible. We can also add it as a dimension, "KeyLoadedFromFile": "True".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants