Future usage of `cratedb-toolkit` for support case analysis

[hammerhead]

Motivation

When customers raise support issues with CrateDB, troubleshooting often requires a holistic view of the cluster state.

Customers usually include the information they consider relevant, but this is naturally only a subset of the information available. During analysis, support often needs to request additional data, such as table schemas, execution plans, shard allocations, disk utilization, data volume, logs, monitoring data, or job statistics.

This can lead to several rounds of written exchange before enough information is available to diagnose and close the case. These feedback cycles increase time to resolution and can negatively impact customer satisfaction.

Aim

We want to have a one-stop command for customers to execute that collects as much raw data as possible.
Interpretation of the data shall only later happen on the CrateDB support side. Keeping captured data as raw as possible allows CrateDB support to investigate even low-level details. Interpretations shall be derived later.

Status Quo

The toolkit currently provides several commands to collect or analyze diagnostic information:

• ctk info jobs: Interprets job data (runtime percentiles, per-user aggregates, individual query statistics)
• ctk cfr jobstats collect|view: Collects job statistics on an ongoing basis. Also interprets job data, unclear differentiation from ctk info jobs.
• ctk info cluster: Returns raw system table data to some extend (sys.nodes, sys.allocations), but mainly returns hand-picked metrics and health indicators (shards not started, number of ongoing rebalances, ...)
• ctk cfr sys-export: Returns a true raw system table export as-is, no filtering or interpretation
• ctk cfr info record: A combination of ctk info cluster and ctk info jobs

Missing Pieces

Information that is currently not captured:

• Any JMX/monitoring data (out of scope, but long-term interesting to standardise collection too)
• CrateDB logs (out of scope for cratedb-toolkit as a client application, at least without additional CrateDB adjustments to make logs collectable?)
• CREATE TABLE statements (should be fully reproducible from system tables?)

Assessment

From my perspective, the current implementation mixes two different concerns:

1. collecting raw diagnostic data
2. interpreting diagnostic data

Going forward from a support-perspective, I suggest we limit information collection to raw data, as currently implemented in ctk cfr sys-export. Such commands can define the schema for data exchange between customers and support. Other (internal) tools can build upon that schema and provide derived insights. In the long-run, such tools may become deliverable products in the hands of customers for self-service.

As a result of reducing cratedb-toolkit scope, user orientation improves and customer-facing tooling becomes less dependent on CrateDB versions. Currently, it is very difficult to understand which command does what from the documentation. Partly due to three letter acronyms (ctk, cfr), combined with generic words such as "info" (ctk cfr info) and documentation being very sparse.

---

@seut @ThomasBestfleischCrate @karynzv This issue describes the current state as discussed on Monday

[amotl]

Hi. What you can find in CrateDB Toolkit about relevant topics is mostly some stray Python code and a few ambitions that never took off here but spawned other tools ¹. On the remaining recipes, there haven't been many iterations, mostly due to lack of felt use and interest. While we didn't work on improving usability and documentation yet, we can see one major request here:

Currently, it is very difficult to understand which command does what from the documentation. Partly due to three letter acronyms (ctk, cfr), combined with generic words such as "info" (ctk cfr info) and documentation being very sparse.

I am also struggling with the same symptoms each time I am trying to use those commands. Do you think adding a few explanations to the documentation can improve the situation as suggested?

• https://cratedb-toolkit.readthedocs.io/info/
• https://cratedb-toolkit.readthedocs.io/cfr/

Footnotes

1. For other types of CrateDB diagnostics, let me also refer you to the excellent cratedb-xlens. 🌻 ↩