What we found
- On the same incidents, claude-fable-5-1 used 446,252.5 fewer provider-visible input tokens on the median case through the GreptimeDB all-in-one interface. [confirmed] claude-fable-5-1: Raw used fewer provider-visible input tokens than Split in 13 of 13 eligible cases, a case median of -446,252.5. Exact sign p 0.00024414062; it passes Holm correction within its family at adjusted p 0.001953125. It is the only one of 8 pre-specified endpoints to pass correction. Another 6 have a case median pointing the same way toward GreptimeDB but did not pass, so they are recorded as a direction and nothing more. On the same incidents the models diagnosed 80 correctly through GreptimeDB and 60 through the three-backend bundle, out of 112 runs each; diagnosis accuracy is descriptive, not a pre-specified endpoint.
- No evidence that the GreptimeDB Semantic Graph systematically reduced investigation work. [not_confirmed] 0 of 8 pre-specified endpoints pass Holm correction, and the case medians do not agree either: 3 point to the Semantic Graph and 5 point the other way. Correct diagnoses came out at 77 with the layer and 80 without. This does not show the layer has no effect, nor that the two interfaces are equivalent; this cohort is too small to support either conclusion.
- The Graph-to-Raw diagnosis gap runs opposite ways on the two cohorts. [descriptive] Split by the level the fault sat at. Graph is ahead on component faults (7 cases): Graph 49/56, Raw 47/56; dependency edge faults (3 cases): Graph 23/24, Raw 20/24. Graph is behind on infrastructure node faults (4 cases): Graph 5/32, Raw 13/32. This split was made after the measurement and was not a pre-specified comparison. Every node-level case in this cohort comes from a single source, so level and source are fully confounded and neither can be credited with the reversal. This is a description, not an explanation.
The three interfaces compared
- Split: telemetry, prometheus-native-http-api, loki-native-http-api, tempo-native-http-api
- Raw: telemetry, ordinary-schema-metadata, read-only-sql, promql-without-metric-metadata
- Graph: telemetry, ordinary-schema-metadata, read-only-sql, promql, table-semantics, semantic-entities, semantic-relationships, usage-guidance, runtime-recovery-guidance, coverage-snapshot
Correct diagnoses by interface
- gpt-5.6-sol: Split 11/28, Raw 22/28, Graph 20/28
- deepseek-v4-pro: Split 9/28, Raw 18/28, Graph 18/28
- claude-fable-5-1: Split 25/28, Raw 26/28, Graph 23/28
- glm-5.3: Split 15/28, Raw 14/28, Graph 16/28
Registered questions
- semantic layer: not supported
- storage shape: partially supported
- model ranking: descriptive
The GreptimeDB Semantic Graph was actually used: 112 of 112 runs that had it made at least one successful query_semantic_graph call, 230 calls in all. The two GreptimeDB interfaces issued 127 successful SQL JOIN calls across 64 runs, of which only 2 joined across signal kinds, in 2 runs. PromQL evaluation shows up in 19 of 224 GreptimeDB runs and 109 of 112 three-backend runs.
464 completed runs across 4 models, 14 end-to-end cases and 8 micro cases. The full narrative report is in REPORT.md.