Add FIT persistency scenarios: atomic store, reset-resistant snapshots, and recovery-from-reset behavior by Subramanian-K812 · Pull Request #13 · qorix-group/reference_integration

Subramanian-K812 · 2026-05-14T06:03:44Z

Implements feature integration tests (FIT) verifying three core persistency requirements: KVS atomic store semantics, reset-resistant snapshot preservation, and recovery-from-reset behavior. Both Rust and C++ scenario implementations are provided with full parity across all test cases.

New Test Scenarios (Rust + C++)

Atomic Store (Atomicity of flush() Operations)

persistency.atomic_store — Verify that a single flush() call atomically persists all pending in-memory writes. No partial-write state is observable: either all keys are present in the snapshot, or none are.

persistency.atomic_store_no_partial_write — Verify the "or nothing" side of atomic store semantics by confirming that un-flushed writes never reach persistent storage when KVS is dropped without flushing.

persistency.atomic_store_multi_instance — Verify that atomic store semantics are maintained independently for each KVS instance when multiple instances operate in the same working directory.

Reset Resistant (Snapshot Preservation During Rotation)

persistency.reset_resistant — Verify that KVS preserves the previous snapshot after a flush-rotation cycle, so that a snapshot representing the last-known-good state is always available after a reset. The current snapshot (snapshot_0) holds the updated value while the previous snapshot (snapshot_1) is preserved.

persistency.reset_resistant_multi_instance — Verify that snapshot rotation for two KVS instances in the same directory is completely isolated — one instance's snapshot files never contaminate the other's rotation sequence.

Recovery From Reset (Post-Reset State Consistency)

persistency.recovery_from_reset — Verify that after a simulated reset (un-flushed in-memory write followed by process termination), the on-disk KVS snapshot still holds the last successfully flushed value. A post-reset boot therefore automatically recovers to a consistent, known-good state.

persistency.recovery_from_reset_multi_instance — Verify that two KVS instances in the same directory each independently recover to their own last-flushed state after a simulated reset, with no cross-instance snapshot contamination.

Test Scenarios Matrix

Rust & C++ (Parity Implementations)

Scenario Name	Test Class	Test File
persistency.atomic_store	TestAtomicStore	test_atomic_store.py
persistency.atomic_store_no_partial_write	TestAtomicStoreNoPartialWrite	test_atomic_store.py
persistency.atomic_store_multi_instance	TestAtomicStoreMultiInstance	test_atomic_store.py
persistency.reset_resistant	TestResetResistant	test_reset_resistant.py
persistency.reset_resistant_multi_instance	TestResetResistantMultiInstance	test_reset_resistant.py
persistency.recovery_from_reset	TestRecoveryFromReset	test_recovery_from_reset.py
persistency.recovery_from_reset_multi_instance	TestRecoveryFromResetMultiInstance	test_recovery_from_reset.py

Requirements Traceability

Test Class	Requirement IDs
TestAtomicStore	feat_req__persistency__atomic_store
TestAtomicStoreNoPartialWrite	feat_req__persistency__atomic_store
TestAtomicStoreMultiInstance	feat_req__persistency__atomic_store feat_req__persistency__multiple_kvs
TestResetResistant	feat_req__persistency__reset_resistant
TestResetResistantMultiInstance	feat_req__persistency__reset_resistant feat_req__persistency__multiple_kvs
TestRecoveryFromReset	feat_req__persistency__recovery_from_reset
TestRecoveryFromResetMultiInstance	feat_req__persistency__recovery_from_reset feat_req__persistency__multiple_kvs

Implementation Summary

Files Added

Python Test Cases (3 new test files):

feature_integration_tests/test_cases/tests/persistency/test_atomic_store.py — Atomic store verification tests
feature_integration_tests/test_cases/tests/persistency/test_reset_resistant.py — Reset-resistant snapshot preservation tests
feature_integration_tests/test_cases/tests/persistency/test_recovery_from_reset.py — Recovery-from-reset tests

Rust Scenario Implementations (3 new scenario files):

feature_integration_tests/test_scenarios/rust/src/scenarios/persistency/atomic_store.rs
feature_integration_tests/test_scenarios/rust/src/scenarios/persistency/reset_resistant.rs
feature_integration_tests/test_scenarios/rust/src/scenarios/persistency/recovery_from_reset.rs

C++ Scenario Implementations (3 new scenario files):

feature_integration_tests/test_scenarios/cpp/src/scenarios/persistency/atomic_store.cpp
feature_integration_tests/test_scenarios/cpp/src/scenarios/persistency/reset_resistant.cpp
feature_integration_tests/test_scenarios/cpp/src/scenarios/persistency/recovery_from_reset.cpp

New Base Classes:

feature_integration_tests/test_cases/persistency_scenario.py — Base class for persistency test scenarios with snapshot reading utilities

Files Modified

feature_integration_tests/test_cases/BUILD — Added test targets for new FIT test files
feature_integration_tests/test_cases/fit_scenario.py — Extended base scenario infrastructure
feature_integration_tests/test_scenarios/rust/BUILD — Updated Rust build configuration
feature_integration_tests/test_scenarios/rust/src/scenarios/persistency/mod.rs — Registered new Rust scenarios
feature_integration_tests/test_scenarios/cpp/src/scenarios/mod.cpp — Registered new C++ scenarios
feature_integration_tests/test_scenarios/cpp/src/internals/persistency/kvs_instance.h — Extended KVS instance helpers
feature_integration_tests/test_scenarios/cpp/src/internals/persistency/kvs_instance.cpp — Implemented KVS instance helpers

Requirements-Based Traceability

Each test is decorated with @add_test_properties() linking to specific feature requirements from the persistency specification:

feat_req__persistency__atomic_store — Atomicity of flush()
feat_req__persistency__reset_resistant — Snapshot preservation
feat_req__persistency__recovery_from_reset — Recovery semantics

Verification Scope

This implementation verifies the following persistency feature requirements:

Requirement	Coverage
Atomicity (feat_req__persistency__atomic_store)	All writes in a flush() are atomic; no partial-write states are observable
Reset Resistance (feat_req__persistency__reset_resistant)	Prior consistent snapshots are preserved during rotation cycles
Recovery (feat_req__persistency__recovery_from_reset)	Un-flushed writes never reach persistent storage; post-reset boots recover to last-known-good state

…,recovery_from_reset

PiotrKorkus · 2026-05-14T06:56:53Z

its not fully checking atomicity. to check all or nothing you would need to e.g. change permissions of dir to read-only in the middle of scenario.

Added AtomicStoreFlushFailure scenario (Rust + C++) for atomicity.

PiotrKorkus · 2026-05-14T07:06:57Z

+/// Negative-path: un-flushed writes must NOT persist on KVS reload.
+pub struct AtomicStoreNoPartialWrite;
+
+impl Scenario for AtomicStoreNoPartialWrite {


I would say we need a scenario where flush is called but it fails

Covered the scenario in AtomicStoreFlushFailure

PiotrKorkus · 2026-05-14T07:08:45Z

simmilar situation here, req says about being reset-resistant but no interuption is simulated. Test should check that after interuption previous snapshot can be loaded by kvs.

PiotrKorkus · 2026-05-14T07:12:06Z

+            "The un-flushed write (100.0) appears to have been persisted incorrectly."
+        )
+
+    def test_instance2_recovers_to_last_flushed_value(self, results: Any, temp_dir: Path) -> None:


missing crucial functionality - we should make sure that kvs can load it on second try. I assume we need another binary call or at least full drop in scenario and reload

Fixed. Scenario added in test_recovery_from_reset.py

PiotrKorkus · 2026-05-14T07:14:03Z

this is not reset API

PiotrKorkus · 2026-05-14T08:01:22Z

rust asserts get_value fails and cpp seems to not load defaults at all

Fixed reset_key is being used now.

PiotrKorkus · 2026-05-14T08:02:51Z

it doesnt clear all previously written keys, reset clears memory state.
snapshots after rotation still have keys

PiotrKorkus · 2026-05-14T08:03:32Z

check values

Added value assertions to both snapshot isolation test

…ry_from_reset reload phases

Subramanian-K812 added 4 commits May 6, 2026 06:17

Add persistency FIT scenarios for reset_resistant,recovery_from_reset…

a133a89

…,recovery_from_reset

Added multi-instance scenario for reset,recovery and atomicity

efee2c1

Merge main: resolve conflicts for atomic/reset/recovery scenarios

ffbb796

Merge origin/main: resolve kvs_instance, mod.cpp and mod.rs conflicts

cd35711

PiotrKorkus reviewed May 14, 2026

View reviewed changes

Added scenario for atomic_store flush-failure, reset_resistant/recove…

aeb411c

…ry_from_reset reload phases

Subramanian-K812 force-pushed the Subramanian-K812_add_fit_persistency_reset_resistant_recovery_atomic branch from 961950f to aeb411c Compare May 14, 2026 11:57

Subramanian-K812 requested a review from PiotrKorkus May 15, 2026 05:13

Conversation

Subramanian-K812 commented May 14, 2026

New Test Scenarios (Rust + C++)

Atomic Store (Atomicity of flush() Operations)

Reset Resistant (Snapshot Preservation During Rotation)

Recovery From Reset (Post-Reset State Consistency)

Test Scenarios Matrix

Rust & C++ (Parity Implementations)

Requirements Traceability

Implementation Summary

Files Added

Files Modified

Requirements-Based Traceability

Verification Scope

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants