Enable Cassandra speculative executions for reads in Java services #2986

Description

Follow-up to #2918 (relates to #2893).

#2918 lowers Cassandra's range_request_timeout to 5s and retries timed-out reads once on another node, which fixes the ~10s of 500s when a Cassandra node crashes. Two gaps remain:

• A few requests that are in flight on the crashed node's connection still fail (we saw ~8-10 per crash at 300 reads/s).
• If a node hangs instead of crashing, every read sent to it still waits the full 5s before the retry kicks in, until the driver marks the node down (heartbeat is 30s).

Speculative execution covers both. When a read is slow, the driver sends the same read to another node and uses whichever answer comes back first. The driver only does this for idempotent statements, and Spring Data leaves idempotence unset on derived queries, so today it never fires.

Plan:

• Add CassandraIdempotentSelectsConfiguration to nv-boot-starter-cassandra. It marks SELECT statements as idempotent (blocking and reactive templates) when nothing else has set idempotence. Writes are left alone, so lightweight transactions (IF NOT EXISTS, IF conditions) are never sent twice. It's on by default and can be turned off with spring.cassandra.idempotent-selects.enabled=false.
• Set speculative max-executions: 2 (the original request plus one extra) in icms-service, api-keys, nvcf-service, ess-service and nvct-service, keeping their current delays (500ms for nvcf, 1500ms for the others).

Definition of Done

• SELECTs run through Spring Data are marked idempotent; writes and LWTs are not.
• Speculative executions are enabled for reads in all five Java services.
• Unit tests cover the statement handling, the post-processors and the on/off switch.
• Crash and hang tests on a 3-node cluster show no failed reads while one node is down or hung.

Resources (optional)

• fix: stop Cassandra index reads stalling while a replica is down #2918
• api-keys reports 500 for keys it created when one Cassandra node crashes #2893
• https://docs.datastax.com/en/developer/java-driver/4.17/manual/core/speculative_execution/

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.