Read O1 KPIs and Alarms

The O1 collector (O1 and the SMO Functions) stores what the O1 adapters send in the monitoring ClickHouse, database o1. This page is how you read it.

The Dashboards

Grafana (Read Logs, Traces, and Metrics) carries two O1 dashboards in the Racora folder:

• O1 KPIs: which elements are streaming and how fresh their last sample is, the O-RU and PTP state per cell, and the per-cell series: UE throughput, scheduler latency, late HARQs, Msg3 outcome and PUCCH usage, packet success over the air, PDCP delay at the CU-UP, MAC and L1 latency, CPU and memory. The element variable narrows the per-cell panels.
• O1 Alarms: the active alarms with their severity, counts by severity, the raises per element over time, the alarm history, and the events the collector could not attribute to an element.

The Tables

Table	One row per	Keyed by
o1.events	VES event, verbatim in body	element, received_at
o1.alarms	fault notification (raise, change, clear)	element, alarm_id, event_time
o1.current_alarms (view)	alarm whose last notification is a raise	element, alarm_id
o1.pm	metric sample of the PM stream	element, name, ts

element is the cell's name for a DU and its O-RU, cu-cp or cu-up for the CU planes, and unknown when nothing identified the sender. The alarm catalogue is the adapter's: 1003 and 1004 are the session to the O-RU lost or degraded, 3001 to 3010 the timing, resource and fronthaul alarms, 4000 plus the O-RU's fault id its own faults.

kubectl -n monitoring exec deploy/clickhouse -- clickhouse-client -q \
  "SELECT element, alarm_id, alarm_name, perceived_severity, raised_at FROM o1.current_alarms ORDER BY raised_at DESC"

kubectl -n monitoring exec deploy/clickhouse -- clickhouse-client -q \
  "SELECT element, name, argMax(value, ts) AS last, max(ts) AS at FROM o1.pm WHERE name = 'DlUeThroughput_Cell' GROUP BY element, name"

kubectl -n monitoring exec deploy/clickhouse -- clickhouse-client -q \
  "SELECT toStartOfMinute(ts) AS m, avg(value) FROM o1.pm WHERE element = 'cell-1' AND name = 'cells.cell_metrics.average_latency' GROUP BY m ORDER BY m"

Retention is per table (racora-smo.collector.store.ttl: one day of events, thirty of alarms, seven of performance samples) and the store is the monitoring ClickHouse, on emptyDir unless racora-monitoring.clickhouse.storage.persistent is set (chart values).

Is the Collector Up

kubectl -n racora-smo rollout status deploy/o1-collector
kubectl -n racora-smo exec deploy/o1-collector -- python -c \
  "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/status').read().decode())"

accepted counts what the adapters delivered, stored what reached ClickHouse, dropped what the bounded queue let go while the store was unreachable, and store.ok with store.last_error say whether ClickHouse answers.

An Element Is Missing

1. Not in o1.pm at all. The element has no O1 adapter streaming yet (in this release only Open Fronthaul cells with an M-plane session run one), or its adapter cannot reach the collector: kubectl -n distributed-unit logs deploy/du-<cell> -c mplane shows every failed delivery.
2. Listed as unknown. The adapter was started without a username: the umbrella's global.o1.ves.url was empty when the pod was created. Re-apply the cell.
3. Stale. The O1 KPIs dashboard marks an element whose last sample is older than three periods; the adapter is up but its source is not. For an Open Fronthaul cell the RadioReady condition says why (NRCell Status and Conditions).
4. Alarms but no performance data. Expected for the M-plane sidecar in this release: it reports faults; its performance stream needs a job it does not have yet.