Storage IO
Storage throughput, queueing and response time per device.
Topic id: storage_io.
Conditions
A condition is a state that holds for a while. The agent opens it when the host enters it, keeps it open while it lasts, and closes it when the host comes back out, so one episode answers for the whole stretch instead of one alert per sample.
| Condition | Severity | How an episode ends |
|---|---|---|
disk latency degraded (disk_latency_degraded) | Warning to Error | It closes on a recovery rule written for this condition, which reads more than one measurement together. |
disk saturated (disk_saturated) | Notice | It closes when the measurement falls back past its recovery point. |
Example
Inventory (every 5 minutes)
1 volume; "C" 0.8 ms p90, 240 IOPS.
sparklogs.data.storage_io.volume: volume:3b1a9c1e-0000-0000-0000-100000000001
sparklogs.data.storage_io.busy_pct_avg: 22.0
sparklogs.data.storage_io.queue_depth_avg: 1.0
sparklogs.data.storage_io.latency_ms_p90_10s: 0.8
SparkLogs: CONTEXT, Info, storage_io: INVENTORY: 1 volume; "C" 0.8 ms p90, 240 IOPS.
Selected conditions
disk_latency_degraded
Storage latency is severe while the disk is busy.
Impact: Workloads above the storage stack may stall or time out.
Example
started; volume "C" latency p90 320
sparklogs.instance: volume:3b1a9c1e-0000-0000-0000-100000000001
sparklogs.data.storage_io.volume: volume:3b1a9c1e-0000-0000-0000-100000000001
sparklogs.data.storage_io.disk_latency_degraded_age_h: 0.0
SparkLogs: disk_latency_degraded, Warning, storage_io: disk_latency_degraded: NOTABLE: started; volume "C" latency p90 320
| Case | Severity | Ticket class |
|---|---|---|
onset | Trace to Fatal | storage |
held | Trace to Fatal | storage |
recovered | Trace to Fatal | storage |
disk_saturated
The disk is busy, queueing and slow to respond.
Impact: Workloads above the storage stack may wait on IO.
Example
started; volume "C" busy 94% (threshold 90%)
sparklogs.instance: volume:3b1a9c1e-0000-0000-0000-100000000001
sparklogs.data.storage_io.volume: volume:3b1a9c1e-0000-0000-0000-100000000001
sparklogs.data.storage_io.busy_pct_avg: 94.0
sparklogs.data.storage_io.queue_depth_avg: 6.0
sparklogs.data.storage_io.latency_ms_p90_10s: 48.0
sparklogs.data.storage_io.disk_saturated_age_h: 0.0
SparkLogs: disk_saturated, Notice, storage_io: disk_saturated: NOTABLE: started; volume "C" busy 94% (threshold 90%)
| Case | Severity | Ticket class |
|---|---|---|
onset | Trace to Fatal | storage |
held | Trace to Fatal | storage |
recovered | Trace to Fatal | storage |