Span Schema

Every component that emits spans (the gNB planes, the RAN controller, the RANN-P forwarder, CU-IP) follows the conventions on this page, so that one operation, an operator editing spec.mobility or an Intelligence Function emitting a decision, reads as one trace across process, language and repository boundaries. This is the reference for reading those traces; how to get at them is Read Logs and Traces.

Data Topology — What Lands Where

gNB planes (C++)      OTLP/HTTP :4318 ─┐
controller, cu-ip,                     ├─► otel-collector ──► traces ──► Tempo
forwarder (Python)    OTLP/gRPC :4317 ─┘        │
                                                └──────────► logs ────► ClickHouse otel.otel_logs

• Traces → Tempo (monitoring namespace, 24 h retention, emptyDir — a pod restart clears history). Grafana's Tempo datasource is the query surface.
• Logs → ClickHouse otel.otel_logs (24 h TTL). Every log record emitted inside an active span carries that span's identity in the TraceId/SpanId columns. The trace-id join is the bridge between the two stores, and it is manual: Grafana's span-to-logs link does not support the ClickHouse datasource. The query is on Read Logs and Traces.
• There is no metrics pipeline. The CU-CP's periodic counter lines (RRC/NGAP handover counters, enabled by metrics.periodicity.cu_cp_report_period in the controller-generated overlay) arrive as log lines; dashboards count log markers. The only metrics that exist are the collector's own on :8888 and the series Tempo's metrics-generator (service-graphs, span-metrics) writes to its local WAL — neither is scraped, remote-written, or stored anywhere.
• ClickHouse holds only otel_logs: the collector's ClickHouse exporter sits on the logs pipeline alone, so no otel_traces / otel_metrics tables are created — traces live in Tempo only.
• Not every component is a producer. The core controller (racora-core-controller) and the device plugins have no OpenTelemetry; their output is kubectl logs only and never reaches ClickHouse. Read the core controller with kubectl -n racora-system logs deploy/racora-core-controller. CU-IP is a producer: its library defaults to the in-cluster collector.

Resource Attributes (per Producer, on Every Span and Log)

key	value
service.name	cu-cp, cu-up, du (one per gNB plane Racora deploys), racora-controller, rann-p-forwarder, cuip
cluster	deployment identity. Canonical default: racora. The collector's resource processor (clusterLabel chart value) is the authority and upserts it on everything passing through; producer-side values are aligned defaults.
pod.name, pod.namespace	from the downward API (POD_NAME, POD_NAMESPACE)

Naming

• Span names: lowercase dotted <layer>.<workflow>[.<phase>] — cucp.neighbor_add.execute, dispatch.rollout_wait, rannd.emit.
• WS command spans are always cucp.<cmd>.execute, tracer scope cucp.remote_command — "is a remote command" is queryable as a class (scope/layer), "which command" is the span name.
• Tracer scopes: the gNB uses <layer>.<module> — cucp.remote_command, cucp.f1ap, du.cell_lifecycle; the Python producers use their service name, with a module suffix where one module dominates — racora-controller, racora-controller.mobility-sync, racora-controller.precondition, rann-p-forwarder, cuip.
• Attributes are namespaced by domain (mobility.*, dispatch.*, rollout.*, f1ap.*, infer.*, cmd.*, ws.*) except the bare identity attributes below.

The Layer Taxonomy

Every span carries the required attribute rann.layer, closed set:

value	emitted by
CTRL	controller reconcile/sync/WS-client spans
DISPATCH	controller decision loop
CUCP-WS	gNB CU-CP remote-command spans
CUCP-F1AP	gNB CU-CP F1AP procedure spans
DU-MNG	gNB DU lifecycle markers
RANN-P	measurement export (forwarder rannp.batch_sent, cuip rannp.batch_received)
RANN-D	cuip decision emit/serve
INFER	cuip intelligence-function evaluation
GRAPH	cuip graph substrate

Identity Attributes (Fixed Names and Types)

attr	type	meaning
nci	int64	NR Cell Identity (decimal)
neighbor_nci	int64	neighbor's NCI
pci, target_pci	int64	physical cell id
rnti	int64	UE RNTI
tac	int64	tracking area code
plmn	string	e.g. "90170"
cell	string	NRCell resource name (controller/cuip side)
cell_index	int64	DU-internal cell index (DU side)
report_cfg_id	int64	measurement report config id
decision_id	string	<function>:<cell>:<field> (RANN-D contract)
function	string	Intelligence Function: pci, anr, …
ws.cmd	string	WS command name, on controller-side spans that wrap one

Outcome and Error Convention

Every span carries two dimensions where they apply:

1. Classification: a per-domain string attribute recording what happened: dispatch.outcome, cmd.outcome (executed / tolerated / failed / unavailable), mobility.sync_outcome (applied / retry / unavailable), mobility.trigger_outcome (triggered / unknown_ue / unknown_target_cell / dispatched), f1ap.outcome, rollout.outcome.
2. OTel span status: ERROR (with a short message) when the operation failed from its caller's perspective; tolerated, converged and skipped outcomes leave the status UNSET. Tempo's error-rate views read only the span status, so the attribute says why and the status says whether it counts as an error.

Propagation Contract

leg	mechanism
controller → gNB (WS)	top-level traceparent JSON key (W3C) on every command payload, injected by the WS client from the active span; the gNB extracts only — absent/malformed ⇒ the command span is a fresh root, never an error
cuip → controller (decisions)	Decision.trace_context column (W3C), captured per decision inside its rannd.emit span; the controller uses it as the parent context for dispatch.consider
in-process gNB (cross-thread)	the WS command span follows the command onto the CU-CP executor, so a command's whole execution is one span

Not propagated. Arrow Flight do_put carries no context: the forwarder's rannp.batch_sent and cuip's rannp.batch_received are disjoint roots, correlated by time and record count. The controller's decision fetch (rannd.serve) is a fresh root; the decision's own trace_context is the stitch. WS replies carry no trace id. The gNB's handover procedures beyond the trigger command, and the DU-side remote commands (ssb_set, pci_set, sib_update, rrm_policy_ratio_set), emit no spans.

Span Catalog

gNB CU-CP Remote Commands (Scope cucp.remote_command, rann.layer=CUCP-WS)

All fourteen WS commands emit cucp.<cmd>.execute with their post-parse identity attributes and — on payloads that fail validation — a parse_error attribute (the five cell-lifecycle commands keep their original cgi.parse_error key). The twelve config/state commands record dispatch.outcome ∈ {accepted, rejected}; the two triggers are fire-and-forget and record mobility.trigger_outcome instead.

span	identity attributes
cucp.cell_lock/.cell_unlock/.cell_bar/.cell_unbar/.cell_status .execute	plmn, nci
cucp.mobility_cell_set.execute	nci, gnb_id_bit_length [+ plmn, pci, tac, band if present]
cucp.mobility_cell_remove.execute	nci
cucp.neighbor_add.execute	nci, neighbor_nci, report_configs_count
cucp.neighbor_remove.execute	nci, neighbor_nci
cucp.report_config_set.execute	report_cfg_id, report_type
cucp.report_config_remove.execute	report_cfg_id
cucp.periodic_report_set.execute	nci, [report_cfg_id], periodic_report.action ∈ {set, clear}
cucp.trigger_handover.execute	pci, rnti, target_pci, plmn, tac, mobility.trigger_outcome
cucp.trigger_conditional_handover.execute	pci, rnti, target_pcis_count, timeout_ms, mobility.trigger_outcome=dispatched

Notes: the trigger commands are fire-and-forget at the WS layer (the response is always success), so mobility.trigger_outcome is the only place the two silent failures — unknown UE, unknown target cell — are visible. Config commands additionally record dispatch.failure ∈ {queue_full, timeout} when the CU-CP executor could not take the command (otherwise indistinguishable from a rejection).

gNB Internals

• cucp.f1ap.tx.gnb_cu_configuration_update (scope cucp.f1ap, rann.layer=CUCP-F1AP) — attrs f1ap.procedure, f1ap.cells_to_be_*_count, f1ap.outcome, f1ap.success.
• du.cell_stop_started / du.cell_stop_completed (scope du.cell_lifecycle, rann.layer=DU-MNG) — zero-duration markers; _started carries cell_index, removal_mode, _completed carries cell_index, outcome=success.

Controller (rann.layer=CTRL Unless Noted)

span	attributes
cucp.mobility_sync	mobility.trigger ∈ {reconcile, resync-timer}, mobility.plan_size, mobility.sync_outcome, mobility.executed, mobility.tolerated, mobility.failed; ERROR on retry
cucp.mobility_sync.cmd (child, one per WS op)	ws.cmd (the wire command name), identity attrs from the op's kwargs (nci, neighbor_nci, report_cfg_id, plmn, report_type, report_configs_count), cmd.outcome; ERROR + exception on failed, ERROR on an unreachable surface (an old image's unavailable is tolerated, not ERROR). The WS client injects this span's traceparent, so the gNB's cucp.<cmd>.execute parents to the specific command
cucp.cell_lock / cucp.cell_unlock	ws.cmd, cell, plmn, nci (+ rollout_confirmed on unlock)
cucp.drain	cell, wait_s
dispatch.consider (rann.layer=DISPATCH)	cell, decision_id, function, dispatch.outcome (applied, a skipped_* gate, error, or deferred when the live object could not be re-read before acting); parented by the decision's trace_context
dispatch.apply (DISPATCH)	function, dispatch.field, dispatch.from, dispatch.to
dispatch.rollout_wait (DISPATCH)	function, cell, du_deployment, rollout.* — emitted only for functions whose post-apply policy waits on a DU rollout (pci; ANR applies emit none)

cucp.cell_lock/cucp.cell_unlock also carry function when driven by the decision loop.

RANN-P / CU-IP

span	attributes
rannp.batch_sent (forwarder, RANN-P)	rann.records, rann.total
rannp.batch_received (cuip, RANN-P, root)	rann.records
graph.update (GRAPH)	— (the streaming edge update; no attributes beyond the layer)
graph.swap (GRAPH)	graph.nodes, graph.edges
infer.eval (INFER, one per engine invocation)	function, infer.decisions, infer.decision_cells (capped at 32, + infer.decision_cells_truncated); the ANR engine adds anr.unknown_neighbor_count when UEs measure cells that are not declared NRCells
rannd.emit (RANN-D, one per decision)	cell, nci, graph_node_id, decision_id, function, reason, confidence, decision.current, decision.recommended, per-function fields via emit hooks (pci_* for PCI; anr.adds, anr.removes, anr.neighbor_count for ANR); captures the per-decision trace_context
rannd.serve (RANN-D, root)	decisions.count — the /decisions/active fetch

The Decision-Loop Trace

An autonomous PCI change reads as one trace:

rannd.emit (cuip: the decision)
└── dispatch.consider (controller, via Decision.trace_context)
    ├── cucp.cell_lock → cucp.cell_lock.execute (gNB)
    ├── cucp.drain
    ├── dispatch.apply (function=pci)
    ├── dispatch.rollout_wait
    └── cucp.cell_unlock → cucp.cell_unlock.execute (gNB)

and a declarative mobility change:

cucp.mobility_sync (controller, trigger=reconcile|resync-timer)
└── cucp.mobility_sync.cmd (ws.cmd=neighbor_add, nci=…, neighbor_nci=…)
    └── cucp.neighbor_add.execute (gNB, via WS traceparent)

An ANR decision combines the two: its dispatch subtree is minimal — no lock (ANR decisions carry no preconditions) and, per its post-apply policy, no rollout wait and no unlock:

rannd.emit (cuip, function=anr, anr.adds/anr.removes)
└── dispatch.consider (controller)
    └── dispatch.apply (function=anr, dispatch.field=spec.neighbors)

— and the actuation then flows through the reconcile's mobility sync exactly like an operator edit (the cucp.mobility_sync tree above, with neighbor_add/neighbor_remove command spans). The K8s spec patch is the seam between the two traces; status.lastApplied.anr.decisionId links the applied spec state back to the decision.

Verifying by Hand

A hand-sent runtime command with a traceparent lands under that trace id in Tempo, and the controller's log lines around a sync carry the same trace id in ClickHouse; the recipes are on Read Logs and Traces.