Adding Integration Tests#21
Open
shekhar316 wants to merge 2 commits into
Open
Conversation
Signed-off-by: SHEKHAR SAXENA <shekhar.saxena@ibm.com>
Signed-off-by: SHEKHAR SAXENA <shekhar.saxena@ibm.com>
Reviewer's GuideIntroduce a self-contained Python-based E2E testing framework for Kruize Optimizer, including an orchestrated deployment runner, cluster and log utilities, configuration, and new pytest suites for full workflow and webhook validation. File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- The
test_webhook_multiple_payloads_one_invalidtest intest_04_webhook.pyappears truncated and never asserts on the response or logs a result, so it should be completed to validate the expected 400 and avoid a silently passing/no-op test. - In
run_e2e_tests.py, configuration keys are used inconsistently (e.g.,self.config.get('kruize_port', ...)vsconfig['api']['kruize_port'], andkind_cluster_namein cleanup vscluster.namein setup); aligning these to a single canonical structure undercluster/apiwill prevent mismatched ports or orphaned clusters during teardown.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The `test_webhook_multiple_payloads_one_invalid` test in `test_04_webhook.py` appears truncated and never asserts on the response or logs a result, so it should be completed to validate the expected 400 and avoid a silently passing/no-op test.
- In `run_e2e_tests.py`, configuration keys are used inconsistently (e.g., `self.config.get('kruize_port', ...)` vs `config['api']['kruize_port']`, and `kind_cluster_name` in cleanup vs `cluster.name` in setup); aligning these to a single canonical structure under `cluster`/`api` will prevent mismatched ports or orphaned clusters during teardown.
## Individual Comments
### Comment 1
<location path="tests/e2e/tests/test_01_complete_workflow.py" line_range="356-365" />
<code_context>
+ def test_08_webhook_callback_received(self, optimizer_client, config):
</code_context>
<issue_to_address>
**issue (testing):** Webhook callback test does not fail when the expected callback is never observed, which makes the test misleading relative to its docstring.
The test’s docstring expects experiment counters to be updated, but if `totalExperimentsProcessed` never increases within the timeout, the test only logs warnings and still passes. This can mask webhook regressions. Please either fail when `webhook_received` stays `False` (or otherwise unmet preconditions are detected), or clearly mark this as a non-strict check and update the expectations in the test’s documentation accordingly.
</issue_to_address>
### Comment 2
<location path="tests/e2e/tests/test_04_webhook.py" line_range="261-50" />
<code_context>
+ def test_webhook_multiple_payloads_one_invalid(self, optimizer_client):
</code_context>
<issue_to_address>
**issue (testing):** The `test_webhook_multiple_payloads_one_invalid` test does not assert on the response, so it never actually verifies behavior.
This test currently sends one valid and one invalid payload but never checks the result, so it will pass no matter what the server does. Please add an assertion for the expected status code (e.g. `assert response.status_code == 400` as suggested by the docstring) and, if relevant, key fields in the response body to verify the mixed-payload behavior.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+356
to
+365
| def test_08_webhook_callback_received(self, optimizer_client, config): | ||
| """ | ||
| Test: Verify webhook callback is received | ||
| Expected: Experiment counters are updated | ||
| """ | ||
| logger.info("Test: Verify webhook callback") | ||
|
|
||
| # Get current state | ||
| jobs_overview = optimizer_client.get_jobs_overview() | ||
| total_experiments = jobs_overview.get('totalExperiments', 0) |
There was a problem hiding this comment.
issue (testing): Webhook callback test does not fail when the expected callback is never observed, which makes the test misleading relative to its docstring.
The test’s docstring expects experiment counters to be updated, but if totalExperimentsProcessed never increases within the timeout, the test only logs warnings and still passes. This can mask webhook regressions. Please either fail when webhook_received stays False (or otherwise unmet preconditions are detected), or clearly mark this as a non-strict check and update the expectations in the test’s documentation accordingly.
| """ | ||
| logger.info("Test: Webhook with invalid JSON") | ||
|
|
||
| response = requests.post( |
There was a problem hiding this comment.
issue (testing): The test_webhook_multiple_payloads_one_invalid test does not assert on the response, so it never actually verifies behavior.
This test currently sends one valid and one invalid payload but never checks the result, so it will pass no matter what the server does. Please add an assertion for the expected status code (e.g. assert response.status_code == 400 as suggested by the docstring) and, if relevant, key fields in the response body to verify the mixed-payload behavior.
Contributor
|
@shreyabiradar07 Can you please review this PR |
shreyabiradar07
requested changes
May 14, 2026
| │ └── log_utils.py # Log parsing | ||
| └── tests/ | ||
| ├── test_01_complete_workflow.py # 10 tests | ||
| ├── test_02_profiles.py # 10 tests |
There was a problem hiding this comment.
README.md documents test_02_profiles.py (10 tests), test_03_bulk_jobs.py (11 tests) but these files don't exist under tests, please verify and update.
shreyabiradar07
requested changes
May 14, 2026
| self.deployment_mgr.create_namespace(app_namespace) | ||
|
|
||
| # Deploy Prometheus first | ||
| self.deployment_mgr.deploy_prometheus() |
There was a problem hiding this comment.
Prometheus deployment can be skipped for OpenShift cluster
| action='store_true', | ||
| help='Skip cleanup after tests' | ||
| ) | ||
|
|
There was a problem hiding this comment.
Adding namespace argument would help to override default namespace in case of OpenShift deployment, otherwise users need to edit test_config.yaml file every time
|
@shekhar316 Please share the test results for both the cluster types |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Add a self-contained end-to-end test framework for Kruize Optimizer, including deployment orchestration and comprehensive workflow and webhook tests.
New Features:
Enhancements:
Documentation:
Tests:
Chores: