Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[load] locust tests support #8409

Merged
merged 64 commits into from
Feb 27, 2025
Merged
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
4230ce1
[load] support multi-region load test configuration
mbhardwaj-msft Nov 28, 2024
e2f9857
merge latest main
mbhardwaj-msft Nov 28, 2024
6dbc06e
refactor
mbhardwaj-msft Nov 28, 2024
fc572c0
update recordings
mbhardwaj-msft Nov 28, 2024
67b0ea0
azdev style load
mbhardwaj-msft Nov 28, 2024
2036fc4
refactor yaml keys to constants file
mbhardwaj-msft Nov 28, 2024
5141df3
refactor test load test
mbhardwaj-msft Nov 28, 2024
7fdecd5
update recordings
mbhardwaj-msft Nov 28, 2024
441b5ec
azdev style load
mbhardwaj-msft Nov 28, 2024
293de07
update help text
mbhardwaj-msft Nov 29, 2024
6864806
update help text
mbhardwaj-msft Nov 29, 2024
d7f576b
action on comments
mbhardwaj-msft Nov 29, 2024
889b1a9
refactor test regional-load autostop
mbhardwaj-msft Nov 29, 2024
46cdd85
check for valid region names client-side
mbhardwaj-msft Dec 2, 2024
68e8f40
[load] support advanced url tests
mbhardwaj-msft Dec 2, 2024
25283f6
action on comments
mbhardwaj-msft Dec 3, 2024
8029f70
action on comments
mbhardwaj-msft Dec 3, 2024
bc649e1
action on comments
mbhardwaj-msft Dec 3, 2024
3045740
update recordings
mbhardwaj-msft Dec 3, 2024
6766bd3
action on comments
mbhardwaj-msft Dec 4, 2024
88a3091
update test and recordings
mbhardwaj-msft Dec 4, 2024
862727f
refactor test exception scenarios
mbhardwaj-msft Dec 4, 2024
df7e056
azdev style load
mbhardwaj-msft Dec 4, 2024
8a1e1ec
Merge pull request #1 from mbhardwaj-msft/mbhardwaj/azload-20241202
mbhardwaj-msft Dec 5, 2024
63f020a
[load] dashboard reports | debug mode | copy artifacts sas url
mbhardwaj-msft Dec 5, 2024
ac1aaab
action on comments
mbhardwaj-msft Dec 6, 2024
06b47c3
update recordings
mbhardwaj-msft Dec 6, 2024
f725e14
set copy artifacts ut live-only
mbhardwaj-msft Dec 6, 2024
d63c746
Merge pull request #2 from mbhardwaj-msft/mbhardwaj/azload-20241205
mbhardwaj-msft Dec 9, 2024
e9d4744
change copy-artifacts to get-artifacts
mbhardwaj-msft Dec 9, 2024
ad7444a
update recordings
mbhardwaj-msft Dec 9, 2024
05085ac
Merge branch 'main' of https://github.com/mbhardwaj-msft/azure-cli-ex…
mbhardwaj-msft Dec 9, 2024
e36a2ba
[load] download-files high scale test update console message
mbhardwaj-msft Dec 9, 2024
5e015cb
[load] download-files high scale test update console message
mbhardwaj-msft Dec 10, 2024
6ca56c1
remove unused import
mbhardwaj-msft Dec 10, 2024
e7c049a
update texts
mbhardwaj-msft Dec 10, 2024
3db9e03
update texts
mbhardwaj-msft Dec 10, 2024
59ca975
Merge pull request #3 from mbhardwaj-msft/mbhardwaj/azload-20241209
mbhardwaj-msft Dec 10, 2024
213725e
[load] convert test to jmx
mbhardwaj-msft Dec 13, 2024
d5e2a61
update comment
mbhardwaj-msft Dec 13, 2024
7e4ad6d
update prompt text
mbhardwaj-msft Dec 16, 2024
7d9d523
Merge pull request #4 from mbhardwaj-msft/mbhardwaj/azload-20241212
mbhardwaj-msft Dec 16, 2024
d8d7f33
[load] baseline trends | sas download
mbhardwaj-msft Dec 19, 2024
e8f8089
unit tests and recordings
mbhardwaj-msft Dec 19, 2024
8f5c4d8
trends aggregate param,unit tests, recordings
mbhardwaj-msft Dec 20, 2024
a86901f
update HISTORY.rst
mbhardwaj-msft Dec 20, 2024
7acd847
remove faulty recording
mbhardwaj-msft Dec 20, 2024
ed91f7c
action copilot review
mbhardwaj-msft Dec 20, 2024
79fbf70
undo faulty commit by copilot
mbhardwaj-msft Dec 20, 2024
9d59cac
test instruction
mbhardwaj-msft Dec 20, 2024
e22fc55
action on comments
mbhardwaj-msft Dec 24, 2024
7c99b99
Merge pull request #6 from mbhardwaj-msft/mbhardwaj/azload-20241216
mbhardwaj-msft Dec 24, 2024
8b0651f
[load] locust test type support
mbhardwaj-msft Jan 2, 2025
7789ab6
[load] locust test type support
mbhardwaj-msft Jan 2, 2025
cc7cc5a
rebase to main
mbhardwaj-msft Jan 9, 2025
e4ee00e
azdev mask recordings
mbhardwaj-msft Jan 9, 2025
2353d7e
update HISTORY.rst
mbhardwaj-msft Jan 9, 2025
7438b73
copyright in locust script
mbhardwaj-msft Jan 9, 2025
81eb3cc
azdev scan fix for locust file
mbhardwaj-msft Jan 9, 2025
f132554
pull latest main
mbhardwaj-msft Feb 26, 2025
0725b7e
fix history order
mbhardwaj-msft Feb 26, 2025
4afd475
Merge branch 'main' of https://github.com/mbhardwaj-msft/azure-cli-ex…
mbhardwaj-msft Feb 26, 2025
fe34268
rerun locust test
mbhardwaj-msft Feb 26, 2025
d9909fa
run azdev scan
mbhardwaj-msft Feb 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor test regional-load autostop
mbhardwaj-msft committed Nov 29, 2024
commit 889b1a9a4bc9fc5a8f20fed1aaf27cb74e0784a2
318 changes: 94 additions & 224 deletions src/load/azext_load/tests/latest/test_load_test_autostop.py
Original file line number Diff line number Diff line change
@@ -70,301 +70,171 @@ def test_load_test_autostop(self):
"--engine-instance {engine_instance} ",
checks=checks,
)

# Update load test with autostop criteria through command line arguments
self.kwargs.update(
{
"autostop_error_rate": LoadTestConstants.AUTOSTOP_ERROR_RATE,
"autostop_error_rate_time_window": LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", False),
JMESPathCheck("autoStopCriteria.errorRate", LoadTestConstants.AUTOSTOP_ERROR_RATE),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop-error-rate {autostop_error_rate} '
'--autostop-time-window {autostop_error_rate_time_window} '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, autostop_error_rate=LoadTestConstants.AUTOSTOP_ERROR_RATE, autostop_error_rate_time_window=LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW)

# Update load test with autostop criteria when error rate is integer
# Order of this test case is important as response payload is checked in next test case
self.kwargs.update(
{
"autostop_error_rate": LoadTestConstants.AUTOSTOP_ERROR_RATE_INTEGER,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", False),
JMESPathCheck("autoStopCriteria.errorRate", float(LoadTestConstants.AUTOSTOP_ERROR_RATE_INTEGER)),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop-error-rate {autostop_error_rate} '
'--autostop-time-window {autostop_error_rate_time_window} '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, autostop_error_rate=LoadTestConstants.AUTOSTOP_ERROR_RATE_INTEGER)

# Update load test with autostop disabled through command line arguments
# Order of this test case is important as response payload is checked from previous test case
self.kwargs.update(
{
"autostop": LoadTestConstants.AUTOSTOP_DISABLED,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", True),
JMESPathCheck("autoStopCriteria.errorRate", float(LoadTestConstants.AUTOSTOP_ERROR_RATE_INTEGER)),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop {autostop} '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, autostop=LoadTestConstants.AUTOSTOP_DISABLED)

# Update load test with autostop criteria through config file
# Order of this test case is important as response payload for time-window is checked in next test case
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", False),
JMESPathCheck("autoStopCriteria.errorRate", 85.0),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", 120),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP)

# Update load test with autostop criteria through config file: only error rate
# Order of this test case is important as response payload for time-window is checked from previous test case
# Order of this test case is important as response payload for error-rate is checked in next test case
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP_ERROR_RATE,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", False),
JMESPathCheck("autoStopCriteria.errorRate", 98.5),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", 120),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP_ERROR_RATE)

# Update load test with autostop criteria through config file: only time window
# Order of this test case is important as response payload for error-rate is checked from previous test case
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP_TIME_WINDOW,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", False),
JMESPathCheck("autoStopCriteria.errorRate", 98.5),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", 250),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP_TIME_WINDOW)

# Update load test with CLI autostop criteria when both config file and CLI arguments are provided
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP,
"autostop_error_rate": LoadTestConstants.AUTOSTOP_ERROR_RATE,
"autostop_error_rate_time_window": LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", False),
JMESPathCheck("autoStopCriteria.errorRate", LoadTestConstants.AUTOSTOP_ERROR_RATE),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
'--autostop-error-rate {autostop_error_rate} '
'--autostop-time-window {autostop_error_rate_time_window} '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, autostop_error_rate=LoadTestConstants.AUTOSTOP_ERROR_RATE, autostop_error_rate_time_window=LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW, load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP)

# Update load test with CLI autostop criteria disabled true when
# config file has autostop criteria and CLI argument is --autostop disable
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP,
"autostop": LoadTestConstants.AUTOSTOP_DISABLED,
}
)
checks = [
JMESPathCheck("autoStopCriteria.autoStopDisabled", True),
JMESPathCheck("autoStopCriteria.errorRate", LoadTestConstants.AUTOSTOP_ERROR_RATE),
JMESPathCheck("autoStopCriteria.errorRateTimeWindowInSeconds", LoadTestConstants.AUTOSTOP_ERROR_RATE_TIME_WINDOW),
]
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
'--autostop {autostop} '
"--resource-group {resource_group} ",
checks=checks,
)
_configure_command_jmes_checks(self, checks, autostop=LoadTestConstants.AUTOSTOP_DISABLED, load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_AUTOSTOP)

# Invalid autostop test case: autostop not of string type
self.kwargs.update({
"autostop": 1,
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop {autostop} '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Invalid autostop type" in str(e)
_configure_command_assert_exception(self, "Invalid autostop type", autostop=1)

# Invalid autostop test case: autostop not in allowed values
self.kwargs.update({
"autostop": "random",
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop {autostop} '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Allowed values: enable, disable" in str(e)
_configure_command_assert_exception(self, "Allowed values: enable, disable", autostop="random")

# Invalid autostop test case: autostop error rate > 100.0
self.kwargs.update({
"autostop_error_rate": 110.5,
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop-error-rate {autostop_error_rate} '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Autostop error rate should be in range of [0.0,100.0]" in str(e)
_configure_command_assert_exception(self, "Autostop error rate should be in range of [0.0,100.0]", autostop_error_rate=110.5)

# Invalid autostop test case: autostop error rate < 0.0
self.kwargs.update({
"autostop_error_rate": -2.5,
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop-error-rate {autostop_error_rate} '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Autostop error rate should be in range of [0.0,100.0]" in str(e)
_configure_command_assert_exception(self, "Autostop error rate should be in range of [0.0,100.0]", autostop_error_rate=-2.5)

# Invalid autostop test case: autostop error rate not of float type
# This is not needed as the argument is type checked
# argument --autostop-error-rate: invalid float value: 'rate'

# Invalid autostop test case: autostop error rate time window < 0
self.kwargs.update({
"autostop_error_rate_time_window": -1,
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--autostop-time-window {autostop_error_rate_time_window} '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Autostop error rate time window should be greater than or equal to 0" in str(e)
_configure_command_assert_exception(self, "Autostop error rate time window should be greater than or equal to 0", autostop_error_rate_time_window=-1)

# Invalid autostop test case: autostop error rate time window not of integer type
# This is not needed as the argument is type checked
# argument --autostop-time-window: invalid int value: '90.4'
# argument --autostop-time-window: invalid int value: 'window'

# Invalid autostop from config test case: autostop random string
self.kwargs.update({
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_INVALID_AUTOSTOP,
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Invalid value for autoStop. Valid values are 'disable' or an object with errorPercentage and timeWindow" in str(e)
_configure_command_assert_exception(self, "Invalid value for autoStop. Valid values are 'disable' or an object with errorPercentage and timeWindow", load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_INVALID_AUTOSTOP)

# Invalid autostop from config test case: autostop error rate > 100.0
self.kwargs.update({
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_INVALID_AUTOSTOP_ERROR_RATE,
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Invalid value for errorPercentage. Value should be a number between 0.0 and 100.0" in str(e)
_configure_command_assert_exception(self, "Invalid value for errorPercentage. Value should be a number between 0.0 and 100.0", load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_INVALID_AUTOSTOP_ERROR_RATE)

# Invalid autostop from config test case: autostop time window < 0
_configure_command_assert_exception(self, "Invalid value for timeWindow. Value should be an integer greater than or equal to 0", load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_INVALID_AUTOSTOP_TIME_WINDOW)


def _configure_command_jmes_checks(self, checks, autostop=None, autostop_error_rate=None, autostop_error_rate_time_window=None, load_test_config_file=None):
command = "az load test update " \
"--test-id {test_id} " \
"--load-test-resource {load_test_resource} " \
"--resource-group {resource_group} "
if autostop is not None:
self.kwargs.update({
"autostop": autostop,
})
command += '--autostop {autostop} '
if autostop_error_rate is not None:
self.kwargs.update({
"autostop_error_rate": autostop_error_rate,
})
command += '--autostop-error-rate {autostop_error_rate} '
if autostop_error_rate_time_window is not None:
self.kwargs.update({
"autostop_error_rate_time_window": autostop_error_rate_time_window,
})
command += '--autostop-time-window {autostop_error_rate_time_window} '
if load_test_config_file is not None:
self.kwargs.update({
"load_test_config_file": load_test_config_file,
})
command += '--load-test-config-file "{load_test_config_file}" '
self.cmd(
command,
checks=checks,
)


def _configure_command_assert_exception(self, message, autostop=None, autostop_error_rate=None, autostop_error_rate_time_window=None, load_test_config_file=None):
command = "az load test update " \
"--test-id {test_id} " \
"--load-test-resource {load_test_resource} " \
"--resource-group {resource_group} "
if autostop is not None:
self.kwargs.update({
"autostop": autostop,
})
command += '--autostop {autostop} '
if autostop_error_rate is not None:
self.kwargs.update({
"autostop_error_rate": autostop_error_rate,
})
command += '--autostop-error-rate {autostop_error_rate} '
if autostop_error_rate_time_window is not None:
self.kwargs.update({
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE_WITH_INVALID_AUTOSTOP_TIME_WINDOW,
})
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
'--load-test-config-file "{load_test_config_file}" '
"--resource-group {resource_group} ",
checks=checks,
)
except Exception as e:
assert "Invalid value for timeWindow. Value should be an integer greater than or equal to 0" in str(e)
"autostop_error_rate_time_window": autostop_error_rate_time_window,
})
command += '--autostop-time-window {autostop_error_rate_time_window} '
if load_test_config_file is not None:
self.kwargs.update({
"load_test_config_file": load_test_config_file,
})
command += '--load-test-config-file "{load_test_config_file}" '
try:
self.cmd(
command,
)
except Exception as e:
assert message in str(e)
314 changes: 49 additions & 265 deletions src/load/azext_load/tests/latest/test_load_test_regional_load.py
Original file line number Diff line number Diff line change
@@ -74,21 +74,7 @@ def test_load_test_regional_load_config(self):

# INVALID: Update load test with regionewise engines
# which do not match the total engine instances
self.kwargs.update(
{
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Sum of engine instances in regionwise load test configuration (5) should be equal to total engine instances (4)" in str(e)
_configure_command_assert_exception(self, "Sum of engine instances in regionwise load test configuration (5) should be equal to total engine instances (4)", regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES)

# VALID: Update load test with regionewise engines
# and total engine instances from CLI
@@ -141,300 +127,98 @@ def test_load_test_regional_load_config(self):
# INVALID: Update load test with regional load configuration from YAML
# which does not contain the updated total engine instances
# and the count does not match with existing configuration
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_NO_TOTAL,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
'--load-test-config-file "{load_test_config_file}" ',
)
except Exception as e:
assert "Sum of engine instances in regionwise load test configuration (4) should be equal to total engine instances (5)" in str(e)
_configure_command_assert_exception(self, "Sum of engine instances in regionwise load test configuration (4) should be equal to total engine instances (5)", load_test_config_file=LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_NO_TOTAL)

# INVALID: Update load test with regional load configuration from YAML
# where engine instances do not match count of regional load configuration
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_COUNT_MISMATCH,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
'--load-test-config-file "{load_test_config_file}" ',
)
except Exception as e:
assert "Sum of engine instances in regionwise load test configuration (2) should be equal to total engine instances (3)" in str(e)
_configure_command_assert_exception(self, "Sum of engine instances in regionwise load test configuration (2) should be equal to total engine instances (3)", load_test_config_file=LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_COUNT_MISMATCH)

# INVALID: Update load test with regionwise engines from CLI containing invalid region
self.kwargs.update(
{
"engine_instances": LoadTestConstants.ENGINE_INSTANCES,
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_INVALID_REGION,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--engine-instances {engine_instances} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Unsupported region invalidregion in the multi-region load test configuration" in str(e)
_configure_command_assert_exception(self, "Unsupported region invalidregion in the multi-region load test configuration", engine_instances=LoadTestConstants.ENGINE_INSTANCES, regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_INVALID_REGION)

# INVALID: Update load test with regionwise engines from CLI containing invalid type for engine count
self.kwargs.update(
{
"engine_instances": LoadTestConstants.ENGINE_INSTANCES,
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_INVALID_TYPE_FLOAT,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--engine-instances {engine_instances} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Expected integer" in str(e)
_configure_command_assert_exception(self, "Expected integer", engine_instances=LoadTestConstants.ENGINE_INSTANCES, regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_INVALID_TYPE_FLOAT)

# INVALID: Update load test with regionwise engines from CLI containing invalid type for engine count
self.kwargs.update(
{
"engine_instances": LoadTestConstants.ENGINE_INSTANCES,
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_INVALID_TYPE_STRING,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--engine-instances {engine_instances} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Expected integer" in str(e)
_configure_command_assert_exception(self, "Expected integer", engine_instances=LoadTestConstants.ENGINE_INSTANCES, regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_INVALID_TYPE_STRING)

# INVALID: Update load test with regionwise engines from CLI containing no parent region
self.kwargs.update(
{
"engine_instances": LoadTestConstants.ENGINE_INSTANCES,
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_NO_PARENT_REGION,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--engine-instances {engine_instances} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Multi-region load test should contain at-least one engine in the parent region East US from where the test is created" in str(e)
_configure_command_assert_exception(self, "Multi-region load test should contain at-least one engine in the parent region East US from where the test is created", engine_instances=LoadTestConstants.ENGINE_INSTANCES, regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_NO_PARENT_REGION)

# INVALID: Update load test with regionwise engines from CLI where argument value has invalid format
self.kwargs.update(
{
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_INVALID_FORMAT_1,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Expected region=engineCount" in str(e)
_configure_command_assert_exception(self, "Expected region=engineCount", regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_INVALID_FORMAT_1)

# INVALID: Update load test with regionwise engines from CLI where argument value has invalid format
self.kwargs.update(
{
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_INVALID_FORMAT_2,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Expected region=engineCount" in str(e)
_configure_command_assert_exception(self, "Expected region=engineCount", regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_INVALID_FORMAT_2)

# INVALID: Update load test with regionwise engines from CLI where argument value has invalid format
self.kwargs.update(
{
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_INVALID_FORMAT_3,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Region or engine count cannot be empty" in str(e)
_configure_command_assert_exception(self, "Region or engine count cannot be empty", regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_INVALID_FORMAT_3)

# INVALID: Update load test with regional load configuration from YAML
# which contains invalid region
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_INVALID_REGION,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
'--load-test-config-file "{load_test_config_file}" ',
)
except Exception as e:
assert "Unsupported region randomregion in the multi-region load test configuration" in str(e)
_configure_command_assert_exception(self, "Unsupported region randomregion in the multi-region load test configuration", load_test_config_file=LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_INVALID_REGION)

# INVALID: Update load test with regional load configuration from YAML
# which contains invalid type for engine count
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_INVALID_TYPE_FLOAT,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
'--load-test-config-file "{load_test_config_file}" ',
)
except Exception as e:
assert "Engine instances is required of type integer" in str(e)
_configure_command_assert_exception(self, "Engine instances is required of type integer", load_test_config_file=LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_INVALID_TYPE_FLOAT)

# INVALID: Update load test with regional load configuration from YAML
# which contains invalid type for engine count
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_INVALID_TYPE_STRING,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
'--load-test-config-file "{load_test_config_file}" ',
)
except Exception as e:
assert "Engine instances is required of type integer" in str(e)
_configure_command_assert_exception(self, "Engine instances is required of type integer", load_test_config_file=LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_INVALID_TYPE_STRING)

# INVALID: Update load test with regional load configuration from YAML
# where parent region is not present
_configure_command_assert_exception(self, "Multi-region load test should contain at-least one engine in the parent region East US from where the test is created", load_test_config_file=LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_NO_PARENT_REGION)

# INVALID: Update load test with regionwise engines containing only 1 region
_configure_command_assert_exception(self, "Multi-region load tests should contain a minimum of 2 geographic regions in the configuration", regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_1, load_test_config_file=LoadTestConstants.LOAD_TEST_CONFIG_FILE)

# INVALID: Update load test with regionwise engines containing duplicate regions
_configure_command_assert_exception(self, "Multi-region load test configuration should not contain duplicate region values", engine_instances=LoadTestConstants.ENGINE_INSTANCES, regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES_3)

# INVALID: Multi-region load test does not support private traffic mode
result = self.cmd(
"az network vnet subnet list --resource-group {resource_group} --vnet-name {virtual_network}"
).get_output_in_json()
subnet_id = result[0]["id"]
_configure_command_assert_exception(self, "You can run multi-region load tests only against public endpoints. Select public test traffic mode to proceed", engine_instances=LoadTestConstants.ENGINE_INSTANCES, regionwise_engines=LoadTestConstants.REGIONWISE_ENGINES, subnet_id=subnet_id)


def _configure_command_assert_exception(self, message, regionwise_engines=None, engine_instances=None, load_test_config_file=None, subnet_id=None):
command = "az load test update " \
"--test-id {test_id} " \
"--load-test-resource {load_test_resource} " \
"--resource-group {resource_group} "
if regionwise_engines:
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.REGIONAL_LOAD_CONFIG_FILE_NO_PARENT_REGION,
"regionwise_engines": regionwise_engines,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
'--load-test-config-file "{load_test_config_file}" ',
)
except Exception as e:
assert "Multi-region load test should contain at-least one engine in the parent region East US from where the test is created" in str(e)

# INVALID: Update load test with regionwise engines containing only 1 region
command += "--regionwise-engines {regionwise_engines} "
if engine_instances:
self.kwargs.update(
{
"load_test_config_file": LoadTestConstants.LOAD_TEST_CONFIG_FILE,
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_1,
"engine_instances": engine_instances,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
'--load-test-config-file "{load_test_config_file}" '
"--regionwise-engines {regionwise_engines} ",
checks=checks,
)
except Exception as e:
assert "Multi-region load tests should contain a minimum of 2 geographic regions in the configuration" in str(e)

# INVALID: Update load test with regionwise engines containing duplicate regions
command += "--engine-instances {engine_instances} "
if load_test_config_file:
self.kwargs.update(
{
"engine_instances": LoadTestConstants.ENGINE_INSTANCES,
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES_3,
"load_test_config_file": load_test_config_file,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--engine-instances {engine_instances} "
"--regionwise-engines {regionwise_engines} ",
)
except Exception as e:
assert "Multi-region load test configuration should not contain duplicate region values" in str(e)

# INVALID: Multi-region load test does not support private traffic mode
result = self.cmd(
"az network vnet subnet list --resource-group {resource_group} --vnet-name {virtual_network}"
).get_output_in_json()
subnet_id = result[0]["id"]
command += '--load-test-config-file "{load_test_config_file}" '
if subnet_id:
self.kwargs.update(
{
"engine_instances": LoadTestConstants.ENGINE_INSTANCES,
"regionwise_engines": LoadTestConstants.REGIONWISE_ENGINES,
"subnet_id": subnet_id,
}
)
try:
self.cmd(
"az load test update "
"--test-id {test_id} "
"--load-test-resource {load_test_resource} "
"--resource-group {resource_group} "
"--engine-instances {engine_instances} "
"--regionwise-engines {regionwise_engines} "
"--subnet-id {subnet_id} ",
)
except Exception as e:
assert "You can run multi-region load tests only against public endpoints. Select public test traffic mode to proceed" in str(e)
command += "--subnet-id {subnet_id} "
try:
self.cmd(command)
except Exception as e:
assert message in str(e)